EBC Exercise 28 Remote gdb and more

There are many handy tools to know how to use.

gdb
gdb, the GNU Project debugger, allows you to see what is going on inside another program while it executes -- or what another program was doing at the moment it crashed.

gdb can do four main kinds of things to help you catch bugs in the act:


 * Start your program, specifying anything that might affect its behavior.
 * Make your program stop on specified conditions.
 * Examine what has happened, when your program has stopped.
 * Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

The program being debugged can be written in Ada, C, C++, Objective-C, Pascal (and many other languages). Those programs might be executing on the same machine as GDB (native) or on another machine (remote). GDB can run on most popular UNIX and Microsoft Windows variants.

For our lab we'll be using a C program and do both local execution on the BeagleBoard and remote execution with gdb running on your host and debugging code on your Beagle.

Installing gdb
On your host computer run:

host$ apt-get install gdb

On your Beagle run:

beagle$ opkg install gdb gdbserver

The Sample Program
You can get the sample program via git on both your host and Beagle.

host$ git clone git@github.com:MarkAYoder/BeagleBoard-exercises.git exercises beagle$ git clone git@github.com:MarkAYoder/BeagleBoard-exercises.git exercises host$ cd exerciese/gdbExample host$ gedit Makefile

Edit the Makefile and correct ARM_TOOLCHAIN_PATH for your machine. Also search for install: and fix it for your beagle.

host$ make all install

This will compile the code and scp it to your Beagle. You've just compiled a program that performs a complex fft on random data. It's main purpose is to see how fast it runs on the Beagle. (In case you are interested, I added to rule so you can compile it for your host computer. Try make x86 and compare the times on your host to those on the Beagle.)

The program takes several seconds to run on the Beagle, so you may want to edit the code so it doesn't run so many iterations.

Running gdb on the Beagle
On your Beagle try:

beagle$ gdb cfft_arm GNU gdb (GDB) 7.2 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-angstrom-linux-gnueabi". For bug reporting instructions, please see: ... Reading symbols from /home/root/exercises/gdbExample/cfft_arm...done. (gdb) b main Breakpoint 1 at 0x8554 (gdb) r Starting program: /home/root/gdbExample/cfft_arm Breakpoint 1, 0x00008554 in main

When first starting gdb notice the line This GDB was configured as "arm-angstrom-linux-gnueabi. This is a good sign.  The first command b main sets a breakpoint at main.  The next command runs to that break point. Now lets look at our code a this point.  Try the list command.

(gdb) l 42	} 43	 44	static complex *new_complex_vector(int size); 45	 46	int main 47	{ 48	 int i; 49	  int N, n; 50	  int nTimes; 51	 float secs; (gdb) n 56	   complex *in = new_complex_vector(N); (gdb) n 57	   complex *out = new_complex_vector(N); (gdb) n 59	   fft_init (N); (gdb) p N $1 = 16

Now l shows the code around the breakpoint. If you aren't seeing code, be sure to scp your .c and .h files to the Beagle.

The n steps to the next line in the program, the p command prints the variable that is listed. Use the s command to step into a function.

(gdb) s fft_init (N=16) at cfft.c:33 33	 tableW = malloc ((N / 2) * sizeof (complex)); (gdb) l 28	 29	void fft_init (int N) 30	{ 31	 int i, j; 32 33	 tableW = malloc ((N / 2) * sizeof (complex)); 34	 bndx = malloc (N * sizeof (int)); 35	 ndx = malloc ((N / 2) * sizeof (int)); 36	 37	 ndx[0] = 0;

That's enough to get you around a bit. help will get you information about more commands. c will continue from where you stopped.

gdb and core files
Sometimes you have a program that stops running expectedly. gdb can help find where it quit and why. Let's find the bug in the cfft program. On the Beagle:

beagle $ ./cfft_arm Floating point exception

Let the Beagle write a core file and see how gdb can use it

beagle$ ulimit -c unlimited beagle$ ./cfft_arm Floating point exception (core dumped) beagle$ gdb ./cfft_arm core GNU gdb (GDB) 7.2 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-angstrom-linux-gnueabi". For bug reporting instructions, please see: ... Reading symbols from /home/root/exercises/gdbExample/cfft_arm...done. [New Thread 30777] Reading symbols from /lib/libm.so.6...Reading symbols from /lib/.debug/libm-2.9.so...done. done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...Reading symbols from /lib/.debug/libc-2.9.so...done. done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.3...Reading symbols from /lib/.debug/ld-2.9.so...done. done. Loaded symbols for /lib/ld-linux.so.3 Core was generated by `./cfft_arm'. Program terminated with signal 8, Arithmetic exception. (gdb) bt at /home/yoder/BeagleBoard/oe/build/tmp-angstrom_2008_1/work/armv7a-angstrom-linux-gnueabi/gcc-cross-4.3.3-r16.1/gcc-4.3.3/libgcc/../gcc/config/arm/lib1funcs.asm:1079 at /home/yoder/BeagleBoard/oe/build/tmp-angstrom_2008_1/work/armv7a-angstrom-linux-gnueabi/gcc-cross-4.3.3-r16.1/gcc-4.3.3/libgcc/../gcc/config/arm/lib1funcs.asm:834
 * 1) 0 0x400c35fc in raise  from /lib/libc.so.6
 * 1) 0 0x400c35fc in raise  from /lib/libc.so.6
 * 2) 1 0x00009abc in __div0
 * 1) 2 0x00008c80 in __udivsi3
 * 1) 3 0x00008888 in fft_exec (N=16, in=0x12090) at cfft.c:87
 * 2) 4 0x0000868c in main  at main_cfft.c:62

gdb tells you right where the error occurred. You can even ask it to list the file.

(gdb) list fft_exec 67	 free (bndx); 68	 free (tableW); 69	} 70	 71	void fft_exec (int N, complex * in) 72	{ 73	 unsigned int n = N; 74	  unsigned int a, b, i, j, k, r, s; 75	  complex w, p; 76 (gdb) l 87 82	     w = tableW[k]; 83	 84	     r = 2 * n * k; 85	      s = n * (1 + 2 * k); 86	 87	     for (j = 0; j < n; j++) 88	     { 89	        a = j + r/0;		// An error 90	       b = j + s; 91	        cmult (p, w, in[b]);      //6 flop

Remote gdb
Sometimes you can't run gdb on an embedded target since it doesn't have enough resources for all of gdb. You can run gdb on your host and debug on the Beagle. You must install a version of gdb on your host that is compiled for the target.

host$ source ~/.oe/environment-oecore host$ bitbake gdb-cross

The bitbake took about 2.5 minutes.

On your Beagle run:

beagle$ gdbserver localhost:2001 ./cfft_arm

On your host run:

host$ source ~/.oe/crossCompileEnv.sh host$ ${CROSS_COMPILE}gdb ./cfft_arm GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-linux --target=arm-angstrom-linux-gnueabi". For bug reporting instructions, please see: ... Reading symbols from /home/yoder/BeagleBoard/gdbExample/cfft_arm...done.

Notice the line that says, this tells us we have the right version of gdb.

Now remotely connect to your Beagle:

(gdb) target remote beagle:2001 Remote debugging using beagle:2001 warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. 0x400007f0 in ?? (gdb) b main Cannot access memory at address 0x0 Breakpoint 1 at 0x8590: file main_cfft.c, line 47. (gdb) c Continuing. warning: `/lib/libm.so.6': Shared library architecture unknown is not compatible with target architecture arm. warning: .dynamic section for "/lib/libm.so.6" is not at the expected address (wrong library or version mismatch?) warning: `/lib/libc.so.6': Shared library architecture unknown is not compatible with target architecture arm. warning: .dynamic section for "/lib/libc.so.6" is not at the expected address (wrong library or version mismatch?) Error while mapping shared library sections: /lib/ld-linux.so.3: No such file or directory. Breakpoint 1, main at main_cfft.c:47 47	{ (gdb) l 42	} 43	 44	static complex *new_complex_vector(int size); 45	 46	int main 47	{ 48	 int i; 49	  int N, n; 50	  int nTimes; 51	 float secs; (gdb) n 56	   complex *in = new_complex_vector(N); (gdb) b fft_exec Breakpoint 2 at 0x87d4: file cfft.c, line 77. (gdb) c Continuing. Breakpoint 2, fft_exec (N=16, in=0x12090) at cfft.c:77 77	 for (i = 1; i < N; i = i * 2) (gdb) p n $1 = 1 (gdb) n 72	{ (gdb) n 3 80	   for (k = 0; k < i; k++) (gdb) p n $2 = 16

And so on. The nice thing is the remote binary, cfft_arm, doesn't have to have the debugging information in it. On your Beagle try:

beagle$ ls -l cfft_arm beagle$ file cfft_arm beagle$ strip cfft_arm beagle$ ls -l cfft_arm beagle$ file cfft_arm beagle$ gdbserver locatl:2001 ./cfft_arm

Mine went from about 26k to 8k when the symbols are removed, but it debugs just like before.

cbrowser/cscope
When you changed the value of MAXPOW2 how did you find where it was defined?

host$ grep MAXPOW *.c *.h

works, but what if your files are spread over several directories? (Like the kernel). cbrowser and cscope can help. On the host computer try:

host$ sudo apt-get install cbrowser host$ sudo apt-get install cscope host$ cscope -b host$ cbrowser

You should see something like: Try searching for other symbols.

strace
strace is a nice debugging tool that shows all the system calls a program is making. The book gives some nice examples of using strace. One place I used it was to learn where httpd looked for configuration files.

beagle$ strace httpd ... open("/etc/httpd.conf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("httpd.conf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) rt_sigaction(SIGHUP, {0x1a39c, [], SA_RESTART|0x4000000}, NULL, 8) = 0 open("/dev/null", O_RDWR|O_LARGEFILE)  = 4 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x4001df98) = 3595 io_submit(0, 0, 0xfbad2088  Process 3594 detached