GDB

The GNU Debugger GDB is the most common debug tool for Linux. It features most used features one can think of, including server-client debug architecture (so you run the heavy debug part on your host/pc machine), but lack some bits as checkpoint-restart during execution.

Documentation

GDB ships with extensive documentation at http://www.gnu.org/software/gdb/documentation/, but there are some good quick reference cards as well.

Basic Usage

Documentation is so large that sometimes its hard to get started, so most simple tasks can be done with the following commands, but please read GDB docs as soon as possible!

Debug version

Debug packages

First note that in order to use GDB in an efficient way,you need the debug packages,that is to say the -dbg version of your package. Under the opkg package management if you installed a binary like that:

opkg install binary

you need to install the debug part of it:

opkg install binary-dbg

Without debug packages you will be limited to assembly debugging(no source level debugging)

Note that if you have restricted space on the target,you could use NFS root if you have an ethernet connection.

No packages

As I don't have packages which segfault at hand,I'll create a binary which segfault First start exporting the path of the toolchain if it's not done yet:

$ cd arm-2010q1/bin #codesourcey
$ export PATH=$(pwd):$PATH

Then create a main.c file with the following content:

#include <stdlib.h> 
struct test{ 
	struct test *crash;
};
int main(){
  struct test *test = NULL;
  test = test->crash;
}

Then compile it:

$ arm-none-linux-gnueabi-gcc -ggdb3 -static -o binary main.c 

• ggdb3 is for including as much debug informations as possible
• -static is only for our test,it's for avoiding runtime issues in case the libc is different or incompatible between codesourcey and our target(in case of ucilbc for instance)

Cross or not crosss

A common error while using gdb,is using the wrong gdb. In order to debug a program that runs on a target from your host you need a cross gdb,that is to say a GDB that runs on your computer but can debug the target architecture. Such version is normally included in your toolchain/SDK or buildable if you use a build system

If you want to debug use GDB on target you can too,it's easier but has a huge drawback(amongs other) : ram usage (you can easily use too much memory and have an OOM(out of memory) )

Cross GDB

Code sourcey Linux

Let's say that you downloaded arm-2010q1-202-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2

You will need to unpack it,and to run the following command on your host computer (instead of just running gdb):

cd arm-2010q1/bin
./arm-none-linux-gnueabi-gdb

Openembedded

If your build system is also your debug workstation do:

bitbake gdb-cross gdbserver

And it will build a cross gdb for your host and gdbserver for your target The resulting binaries will be found in your TMPDIR here:

/home/embedded/tmpdir/cross/armv6/bin/

replace "/home/embedded/tmpdir" by your tmpdir and armv6 by your target architecture The binary name changes according to your distro settings,for me it was:

arm-angstrom-linux-gnueabi-gdb

So in order to run it do: ./arm-angstrom-linux-gnueabi-gdb

Howto

If you have networking,it's advised to choose it because it will be faster than serial,but if you only have serial choose serial

Core dump

Sometimes you can get a coredump. A man page describing the core dump,and in which condition it can occur can be found here In order to make possible core dumps(that unlimit the size limit on coredumps)

target/device$ ulimit -c unlimited

Then run your binary:

target/device$ ./binary

Then reproduce the conditions under which the coredump happen.The core file will appear in your current working directory and will be named core

Then copy the coredump to your workstation,and run gdb on your workstation(core is assumed to be in the current directory)

./arm-angstrom-linux-gnueabi-gdb 
(gdb) set sysroot /mnt/target
(gdb) file /mnt/moko/usr/bin/binary
(gdb) target core core

Networking

Fist start gdbserver on the target:

target/device$ gdbserver :2345 ./binary arg1 arg2

Note that will expose a gdb session to all networking interfaces,for instance if you have wifi on,someone could connect to it. So if you don't want that,make it listen only on your the ip of the target interface you want to use for debugging:

target/device$ gdbserver 192.168.0.202:2345 ./binary arg1 arg2

Then start GDB on your workstation,but do not connect to the target yet(else you'll have some undefined symbols):

host/pc$ ./arm-angstrom-linux-gnueabi-gdb
GNU gdb (Sourcery G++ Lite 2010q1-202) 7.0.50.20100218-cvs
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi".
For bug reporting instructions, please see:
<https://support.codesourcery.com/GNUToolchain/>.
(gdb) 

Then you need access to the target filesystem,knowing that we have networking we could use the following options:

• sshfs:

In order to use sshfs you need sftp-server installed on the target,and sshfs installed on the host computer. To mount the filesystem do:

mkdir /mnt/target/
sudo sshfs root@192.168.0.202:/ /mnt/target/ -o allow_other

• NFS(maybe you already had nfsroot on the target)
• any other system that export the filesystem,and that is mountable

Then you need to set a variable called sysroot in order to make the cross gdb find the libraries to load

(gdb) set sysroot /mnt/target

Then load the binary:

(gdb) file /mnt/target/usr/bin/binary
Reading symbols from /mnt/target/usr/bin/binary...Reading symbols from /mnt/moko/usr/bin/.debug/binary...done. 

And finally connect to the target:

(gdb) target remote 192.168.0.202:2345

Serial

First set the serial port speed on the target(before you issue the gdbserver command):

 stty speed 115200 < /dev/ttyS1

Then start gdbserver on the target:

target/device$ gdbserver /dev/ttyS1 ./binary arg1 arg2

That will freeze the program until you connect with the cross gdb version and that you type "c" for continuing the execution of the program. Note that GDB will slow the program a lot. if you have a program that uses a lot of resources,it's advised to attach to it just before the crash instead like that:

target/device$ gdbserver /dev/ttyS1 --attach PID

Then do that on your workstation

host/pc$ ./arm-angstrom-linux-gnueabi-gdb
(gdb) target remote /dev/ttyS1

If you are using the serial method you need to make sure you have setup the serial speed correctly, On the host:

 (gdb) set remotebaud 115200

Non-cross GDB

Howto

To start a new application for debug, use:

 $ gdb ./binary
 $ gdb ./binary core.dump
 $ gdb --args ./binary arg1 arg2
 $ gdb --command=my-gdb-commands.txt --args ./binary arg1 arg2

and then run it with (args just required if no --args were used):

 (gdb) run arg1 arg2

If application is already running, find out its pid (ps, top, pidof, ...) and:

$ gdb --pid $PID

Tips

if you need to execute a series of commands every time, consider writing them on a file and use --command=file (or -x file).

Breakpoints

If you control-C (^C), it will break at that point, but you can also schedule a breakpoint with:

(gdb) break function
(gdb) break line
(gdb) break file:function
(gdb) break file:line

conditional breaks are in the form:

(gdb) break where if condition

where condition is some C expression that evaluates to 1 or 0, like *ptr == NULL

One can disable or remove breakpoints with:

(gdb) enable breakpoint-number
(gdb) disable breakpoint-number
(gdb) delete breakpoint-number
(gdb) clear       # removes all breakpoints

Examining

To list source code nearby position or specific places:

(gdb) list
(gdb) list line
(gdb) list function
(gdb) list file:line
(gdb) list file:function
(gdb) list *address

To list execution backtrace (or bt for short):

(gdb) backtrace

To change frame to operate on:

(gdb) frame frame-number

To change thread to operate on:

(gdb) thread thread-number

To print some value or expression:

(gdb) print $register
(gdb) print variable
(gdb) print *address
(gdb) print *(int *)address
(gdb) print *(char **)address
(gdb) print myfunc(p1, p2) # will actually execute it and return result!
(gdb) print *a = 123 # will actually change *a value and return 123!
(gdb) print file::variable
(gdb) print function::variable

To disassembly:

(gdb) disassembly
(gdb) disassembly file:line

Print function arguments:

(gdb) info args

Print locals:

(gdb) info locals

Print breakpoints:

(gdb) info breakpoints

Print threads:

(gdb) info threads

Stepping

To go to next instruction, possible entering a function (or s for short):

(gdb) step

To go to next instruction, but avoid entering new functions (or n for short):

(gdb) next

To continue until the function returns:

(gdb) finish

To continue execution (or c for short):

(gdb) continue

Manipulating Program

To set variable to some value:

(gdb) set var name=value

To force function to return:

(gdb) return value
(gdb) return expression

Changing Signal Handlers

(gdb) handle signal action

debugging applications with old libC, those pre-nptl, can be really annoying due SIG32 and SIG33, one can ignore those with:

(gdb) handle SIG32 nostop
(gdb) handle SIG32 noprint
(gdb) handle SIG33 nostop
(gdb) handle SIG33 noprint

Shared Object Paths

Often your cross compile root is not /, so you might have to add new paths to the search list.

Unset absolute prefix:

 (gdb) set solib-absolute-prefix null

Add paths to search paths:

 (gdb) set solib-search-path /path1:/path2

Alternatively you can choose to set the prefix to the root of your target file system. Specially if you are doing embedded development and already exporting your root file system from you host machine to your target machine it can be very rewarding so simply use that as root:

 (gdb) set solib-absolute-prefix /rootfs