Debugging Techniques

Finding bug is a process of confirming the many things you believe are true, until you find the one, which is not true.
Below are some of the things, which developers believe they are true

When the function is called, it receives its parameters correctly.

At a certain line or function in your source file, a certain variable has a certain value.

In a given if-then-else statement, the “else” part is the one that is executed.

Divide and Conquer

The simple technique for debugging is monitoring, write printf statements in the source code at which you know the data is good and where it becomes corrupted.
As we do not know where the data is good and bad, usually ad hoc addition of lots of printf statements are added to track the control flow and data values during the execution of code. After adding printf statements, run the program and keep adding printf statements until you isolate the area in the program that contains the wrong data. Then concentrate on why and how this data gets modified.
In this way, you can start zeroing in on the section of code that contains the error.

Debug Macros

The preprocessor conditional compilation directives can be used as switch to include or not to include the debug messages.

For example:

#ifdef DEBUG
    printf(“sum %d\n”, sum);

The program can be compiled with DEBUG undefined for normal use;

you can define it when debugging is needed as follows

#define DEBUG
The following program prints the debug messages

#define DEBUG
int main()
    char *iter = (char*) malloc (100);
    strcpy(iter, “This is a simple program”);
    int sum;
    for (int i=0; i <100; i++) {
        #ifdef DEBUG
            printf(“char is %c and sum is %d\n”, *iter, sum);
        sum = sum + *iter;
    return 0;

You shall redirect the output to a file what would normally go to the screen. For example:

$test >tmp.out

runs the program test with debugging and sends the output to the file tmp.out.

You can use your text editor’s search capabilities to look for the information you want to find.

For large programs which has millions of code, the above mechanism will dump many millions of debug messages which makes the bug finding process difficult. We can use multiple debug level to tackle this issue. Based on the code size of the software, you shall define many debugging levels. Lower debugging levels produce less messages and higher the level, more messages it produces. When debugging an issue, start with lower level and then if you cannot figure out the cause, move to the next level and so on.

The following set of macros defines three debugging levels, DLEVEL3, DLEVEL2 and DLEVEL1.

#ifdef DLEVEL3
   #define DLEVEL2
#ifdef DLEVEL2
    #define DLEVEL1

If you software constitutes many modules, then module specific debug messages can be enabled to see only messages related to that module.

Run Time Tracing

The above techniques require source code access to identify the problem. Run time tracing allows the debugger to identify the problem on executable. This technique shall also be used to understand the code flow of the program. There are tools which traces all system calls or library routines.

strace utility provides the capability to trace the execution of an application from the perspective of system calls. Along with the system call routine, it also displays their arguments, return value, etc

Linux permits to access 4 GB virtual address space for a process and that 4 GB virtual space is divided into two parts such as user space ( 3 GB) and kernel space (1 GB). When we run any application program along with library , they allocates memory from user space . Here in the following example printf is a library function and internally uses a standard library The function printf writes data “Hello” into monitor , but how it happens.


void main()




Operating system uses system call to copy the data from user space to kernel space and write into device file which is associated with a device ( example stdout for monitor) . The driver known as modules works in kernel space which reads the data from a device file and write into the port .

To list out the system call maintained by o/s there is a development tool called strace .Ittrace the system calls and signals . It intercepts and records the system calls made by a running process. strace can print a record of each system call, its arguments, and its return value. strace is a system call tracer, i.e. a debugging tool which prints out a trace of all the system calls .

$strace ./singo


open(“/etc/”, O_RDONLY) = 3

fstat64(3, {st_mode=S_IFREG|0644, st_size=111023, …}) = 0

mmap2(NULL, 111023, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f91000

close(3) = 0

open(“/lib/”, O_RDONLY) = 3

read(3, “\177ELF\1\1\1\3\3\1\360\3247004″…, 512) = 512

write(1, “Hello”, 5Hello) = 5

Each line starts with a system call name, is followed by its arguments in the parenthesis & then has return value at the end of the line. The important system call made in the above output are open,close,read,write. The last line write(1, “Hello”, 5Hello) = 5 line corresponding to the system call that cause Hello to be printed on the screen. The first argument -1 is the file descriptor of the file i.e. stdout to write to. So it writes the message on the terminal instead of writing in some other file on disk. The second argument tells the data is a string & third argument display the total number of character in the string.

$ strace -c ./singo

Hello% time seconds usecs/call calls errors syscall

—— ———– ———– ——— ——— —————-

nan 0.000000 0 1 read

nan 0.000000 0 1 write

nan 0.000000 0 2 open

nan 0.000000 0 2 close

nan 0.000000 0 1 execve

nan 0.000000 0 1 1 access

nan 0.000000 0 1 brk

nan 0.000000 0 1 munmap

nan 0.000000 0 2 mprotect

nan 0.000000 0 7 mmap2

nan 0.000000 0 3 fstat64

nan 0.000000 0 1 set_thread_area

—— ———– ———– ——— ——— —————-

100.00 0.000000 23 1 total

From the above output, it is evident that write system call is used 1time. This is initiated by printf() statement on stdout file descriptor.

One another important option of strace is -tt, it causes strace to print out the time at which each call finished.

ltrace can be used to monitor user library calls. Most of the option present in strace is also present in ltrace. For the complete set of options run `man ltrace`.

Interactive Debugging

Interactive debuggers give you the ability to stop the program at any point, examine and change variables, and continue the program to identify the code which changes the behavior and makes the program buggy. The GNU Debugger (also known as gdb) is an interactive debugging tool which allows you to see what is going on inside the running program while it executes.


Linux provides lot of information and tools to investigate the running process (your executable). It stores information about each of the process running in the system at /proc file system.
This directory has lot of subdirectories each denoting the process ID of the running process in the system.

$ls /proc
1     204   2481  2927   30479  399   9            kallsyms    swaps
10    2040  2518  2996   30535  4     acpi         kcore       sys
11    2047  2526  3      3094   400   asound       keys        sysrq-trigger
1477  205   2538  3001   3096   404   buddyinfo    key-users   sysvipc
1478  2092  2582  3002   3102   411   bus          kmsg        timer_list
1502  2173  2654  3003   3103   418   cmdline      loadavg     timer_stats
1653  2207  2690  3004   3114   429   cpuinfo      locks       tty
1654  2226  2711  3005   3115   457   crypto       mdstat      uptime
167   2237  2733  3115  3137   494   devices      meminfo     version
168   2249  2766  5218  3163   5     diskstats    misc        vmcore
169   2264  2778  30146  3165   5051  dma          modules     vmstat
172   2302  2779  30283  3523   5052  driver       mounts      zaptel
174   2319  2796  30285  3525   51    execdomains  mtrr        zoneinfo
1990  2335  2797  30367  3526   52    fb           net
1992  2354  2808  30398  355    53    filesystems  partitions
2     2359  2809  30421  388    5555  fs           schedstat
201   2381  2838  30442  389    5557  interrupts   scsi
202   2395  2839  30444  390    6     iomem        self
2020  2408  2847  30445  397    7     ioports      slabinfo
203   2431  2859  30477  398    8     irq          stat

Find out the process ID of you program using the ps command, lets say your executable name is csqlserver, then

$ps -ef |grep csqlserver
csql      5218  3115  0 13:35 pts/1    00:00:00 csqlserver
csql      5370  3526  0 13:42 pts/3    00:00:00 grep csqlserver

For the above output, pid is 5218.
Under this directory (/proc/5218) you will find lots of useful information which helps in debugging your application
Some of the important ones are
cmdline – contains the command line options with the executable name with which you started the process
environ – contains the environment variables like PATH, HOME,etc this process uses
maps – contains the memory map of the process(similar to output of pmap command)
fd – directory which contains all the file descriptors(you will see minimum 0,1 and 2 file descriptors which refers to stdin, stdout, stderr respectively)
cwd – link to current working directory
status – process status

Linux Tools
ps , pmap, top, vmstat
tools displays information about running process in the system which helps developers to identify issues such as CPU consumption, high disk I/O, high memory usage by the programs.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s