EBC Exercise 28 Remote gdb and more

There are many handy tools to know how to use.

gdb
gdb, the GNU Project debugger, allows you to see what is going on inside another program while it executes -- or what another program was doing at the moment it crashed.

gdb can do four main kinds of things to help you catch bugs in the act:


 * Start your program, specifying anything that might affect its behavior.
 * Make your program stop on specified conditions.
 * Examine what has happened, when your program has stopped.
 * Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

The program being debugged can be written in Ada, C, C++, Objective-C, Pascal (and many other languages). Those programs might be executing on the same machine as GDB (native) or on another machine (remote). GDB can run on most popular UNIX and Microsoft Windows variants.

For our lab we'll be using a C program and do both local execution on the BeagleBoard and remote execution with gdb running on your host and debugging code on your Beagle.

Installing gdb
On your host computer run: $ apt-get install gdb On your Beagle run: $ opkg install gdb gdbserver

The Sample Program
You can get the sample program from dfs.

$ sftp dfs.rose-hulman.edu Connecting to dfs.rose-hulman.edu... yoder@dfs.rose-hulman.edu's password: sftp> cd users/y/yoder/shared/beagleboard sftp> get gdbExample.tar.gz sftp> exit $ tar xvf gdbExample.tar.gz gdbExample/ gdbExample/cfft.c gdbExample/main_bench.c gdbExample/changelog gdbExample/distance.c gdbExample/cfft.h gdbExample/README gdbExample/distance.h gdbExample/common.h gdbExample/Makefile gdbExample/main_cfft.c $ cd gdbExample $ gedit Makefile Edit the Makefile and correct ARM_TOOLCHAIN_PATH for your machine. Also search for install: and fix it for your beagle. $ make all install This will compile the code and scp to your Beagle. You've just compiled a program that performs a complex fft on random data. It's main purpose is to see how fast it runs on the Beagle. (In case you are interested, I added to rule so you can compile it for your host computer. Try make x86 and compare the times on your host to those on the Beagle.)

The program takes several seconds to run on the Beagle, so you may want to edit the code so it doesn't run so many iterations.

Running gdb on the Beagle
On your Beagle try:

$ gdb cfft_arm gdb cfft_arm GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-angstrom-linux-gnueabi". For bug reporting instructions, please see: ... Reading symbols from /home/root/gdbExample/cfft_arm...done. (gdb) b main Breakpoint 1 at 0x8554 (gdb) r Starting program: /home/root/gdbExample/cfft_arm

Breakpoint 1, 0x00008554 in main When first starting gdb notice the line This GDB was configured as "arm-angstrom-linux-gnueabi. This is a good sign.  The first command b main sets a breakpoint at main.  The next command runs to that break point. Now lets look at our code a this point.  Try the list command. (gdb) l 1	/home/yoder/BeagleBoard/oe/build/tmp-angstrom_2008_1/work/armv7a-angstrom-linux-gnueabi/glibc-2.9-r36.3/build-arm-angstrom-linux-gnueabi/csu/crtn.S: No such file or directory.	in /home/yoder/BeagleBoard/oe/build/tmp-angstrom_2008_1/work/armv7a-angstrom-linux-gnueabi/glibc-2.9-r36.3/build-arm-angstrom-linux-gnueabi/csu/crtn.S (gdb) quit Hmmm....  We have a problem.  For gdb to be most useful we need to tell gcc to include debugging information.  Back to your host computer and do an ls on cfft_arm and note it's size.  Mine is about 18k.  Next edit the Makefile and find ARM_CFLAGS and add -g. The line should look like: ARM_CFLAGS = $(CFLAGS) -g Now clean everything and remake. $ make clean all install $ ls -l cfft_arm How has the size of cfft_arm changed?

Back to the Beagle. $ gdp cfft_arm (gdb) b main (gdb) r (gdb) l 42	} 43	44	static complex *new_complex_vector(int size); 45	46	int main 47	{ 48	 int i; 49	  int N, n; 50	  int nTimes; 51	 float secs; (gdb) n 56	   complex *in = new_complex_vector(N); (gdb) n 57	   complex *out = new_complex_vector(N); (gdb) n 59	   fft_init (N); (gdb) p N $1 = 16 Now l shows the code around the breakpoint. If you aren't seeing code, be sure to scp your .c and .h files to the Beagle.

The n steps to the next line in the program, the p command prints the variable that is listed. Use the s command to step into a function.

(gdb) s fft_init (N=16) at cfft.c:33 warning: Source file is more recent than executable. 33	 tableW = malloc ((N / 2) * sizeof (complex)); (gdb) l 28 29	void fft_init (int N) 30	{ 31	 int i, j; 32 33	 tableW = malloc ((N / 2) * sizeof (complex)); 34	 bndx = malloc (N * sizeof (int)); 35	 ndx = malloc ((N / 2) * sizeof (int)); 36	37	 ndx[0] = 0; That's enough to get you around a bit. help will get you information about more commands. c will continue from where you stopped.

Try this: gdb cfft_arm gdb) b fft_exec (gdb) r (gdb) n 4 (gdb) l (gdb) p n We're in the fft_exec routine and have next'ed a in a way and then printed the value of n. The main routine also has an n, how do we see it's value?  It's on another stack frame, so do a backtrace to see what's on the stack and then select the frame we want. (gdb) bt (gdb) f 1 62	    fft_exec (N, out); (gdb) p n $7 = (gdb) f 0 80	    for (k = 0; k < i; k++) (gdb) p n $8 = 16 The f command selects the frame we want to inspect.  Notice in this case the value of n in main has been optimized out, that means the compiler found a way to compile the code that didn't need to put n on the stack.  In the Makefile you will find the -O3 option will tells the compiler to optimize a lot.  Try removed the -O3 and remaking everything and seeing if n appears in main.
 * 1) 0 fft_exec (N=16, in=0x12090) at cfft.c:80
 * 2) 1 0x0000865c in main  at main_cfft.c:62
 * 1) 1 0x0000865c in main  at main_cfft.c:62
 * 1) 0 fft_exec (N=16, in=0x12090) at cfft.c:80

gdb and core files
Sometimes you have a program that stops running expectedly. gdb can help find where it quit and why. Let's insert a bug in the cfft program and see how gdb finds it. Edit cfft and replace this line near the bottom a = j + r;		// An error with a = j + r/0;		// An error

so we will have a divide by zero error. Remake and install $ make all install On the Beagle you should see: $ ./cfft_arm Floating point exception Let the Beagle write a core file and see how gdb can use it $ ulimit -c unlimited $ ./cfft_arm Floating point exception (core dumped) $ gdb ./cfft_arm core GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-angstrom-linux-gnueabi". For bug reporting instructions, please see: ... Reading symbols from /home/root/gdbExample/cfft_arm...done. [New Thread 3525] Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.3 Core was generated by `./cfft_arm'. Program terminated with signal 8, Arithmetic exception. (gdb) bt   at /home/yoder/BeagleBoard/oe/build/tmp-angstrom_2008_1/work/armv7a-angstrom-linux-gnueabi/gcc-cross-4.3.3-r16.1/gcc-4.3.3/libgcc/../gcc/config/arm/lib1funcs.asm:1079 at /home/yoder/BeagleBoard/oe/build/tmp-angstrom_2008_1/work/armv7a-angstrom-linux-gnueabi/gcc-cross-4.3.3-r16.1/gcc-4.3.3/libgcc/../gcc/config/arm/lib1funcs.asm:834 gdb tells you right where the error occurred. You can even ask it to list the file. (gdb) list fft_exec 67	 free (bndx); 68	 free (tableW); 69	} 70	71	void fft_exec (int N, complex * in) 72	{ 73	 unsigned int n = N; 74	  unsigned int a, b, i, j, k, r, s; 75	  complex w, p; 76 (gdb) l 87 82	     w = tableW[k]; 83	84	     r = 2 * n * k; 85	      s = n * (1 + 2 * k); 86	87	     for (j = 0; j < n; j++) 88	     { 89	        a = j + r/0;		// An error 90	       b = j + s; 91	        cmult (p, w, in[b]);      //6 flop
 * 1) 0 0x400c35fc in raise  from /lib/libc.so.6
 * 1) 0 0x400c35fc in raise  from /lib/libc.so.6
 * 2) 1 0x00009abc in __div0
 * 1) 2 0x00008c80 in __udivsi3
 * 1) 3 0x00008888 in fft_exec (N=16, in=0x12090) at cfft.c:87
 * 2) 4 0x0000868c in main  at main_cfft.c:62

Remote gdb
Sometimes you can't run gdb on an embedded target since it doesn't have enough resources for all of gdb. You can run gdb on your host and debug on the Beagle. You must install a version of gdb on your host that is compiled for the target. $ source ~/.oe/environment-2008 $ bitbake gdb-cross The bitbake took about 4.5 minutes on a slow dsl line.

On your Beagle run: $ gdbserver localhost:2001 ./cfft_arm On your host run: $ source ~/BeagleBoard/oe/crossCompileEnv.sh $ ${CROSS_COMPILE}gdb ./cfft_arm GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-linux --target=arm-angstrom-linux-gnueabi". For bug reporting instructions, please see: ... Reading symbols from /home/yoder/BeagleBoard/gdbExample/cfft_arm...done. Notice the line that says, this tells us we have the right version of gdb.

Now remotely connect to your Beagle: (gdb) target remote beagle2:2001 Remote debugging using beagle2:2001 warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. 0x400007f0 in ?? (gdb) b main Cannot access memory at address 0x0 Breakpoint 1 at 0x8590: file main_cfft.c, line 47. (gdb) c Continuing. warning: `/lib/libm.so.6': Shared library architecture unknown is not compatible with target architecture arm. warning: .dynamic section for "/lib/libm.so.6" is not at the expected address (wrong library or version mismatch?) warning: `/lib/libc.so.6': Shared library architecture unknown is not compatible with target architecture arm. warning: .dynamic section for "/lib/libc.so.6" is not at the expected address (wrong library or version mismatch?) Error while mapping shared library sections: /lib/ld-linux.so.3: No such file or directory.

Breakpoint 1, main at main_cfft.c:47 47	{ (gdb) l 42	} 43	44	static complex *new_complex_vector(int size); 45	46	int main 47	{ 48	 int i; 49	  int N, n; 50	  int nTimes; 51	 float secs; (gdb) n 56	   complex *in = new_complex_vector(N); (gdb) b fft_exec Breakpoint 2 at 0x87d4: file cfft.c, line 77. (gdb) c Continuing.

Breakpoint 2, fft_exec (N=16, in=0x12090) at cfft.c:77 77	 for (i = 1; i < N; i = i * 2) (gdb) p n $1 = 1 (gdb) n 72	{ (gdb) n 3 80	   for (k = 0; k < i; k++) (gdb) p n $2 = 16 And so on. The nice thing is the remote binary, cfft_arm, doesn't have to have the debugging information in it. On your Beagle try: $ ls -l cfft_arm $ file cfft_arm $ strip cfft_arm $ ls -l cfft_arm $ file cfft_arm $ gdbserver locatl:2001 ./cfft_arm Mine went from about 26k to 8k when the symbols are removed, but it debugs just like before.

cbrowser/cscope
When you changed the value of MAXPOW2 how did you find where it was defined? $ grep MAXPOW *.c *.h works, but what if your files are spread over several directories? (Like the kernel). cbrowser and cscope can help. On the host computer try: $ sudo apt-get install cbrowser $ sudo apt-get install cscope $ cscope -b $ cbrowser You should see something like: Try searching for other symbols.

strace
strace is a nice debugging tool that shows all the system calls a program is making. The book gives some nice examples of using strace. One place I used it was to learn where httpd looked for configuration files. $ strace httpd ... open("/etc/httpd.conf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("httpd.conf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) rt_sigaction(SIGHUP, {0x1a39c, [], SA_RESTART|0x4000000}, NULL, 8) = 0 open("/dev/null", O_RDWR|O_LARGEFILE)  = 4 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x4001df98) = 3595 io_submit(0, 0, 0xfbad2088  Process 3594 detached