Introduction and basic resources
Here is some information about KDB - the in-kernel debugger for the Linux kernel.
The KDB and KGDB official wiki: https://kgdb.wiki.kernel.org/ (this only has 2 pages?)
Jason Wessel is the current KDB maintainer. Here is a presentation from him at LinuxCon 2010 (August 2010): http://kernel.org/pub/linux/kernel/people/jwessel/dbg_webinar/State_Of_kernel_debugging_LinuxCon2010.pdf
Here are some videos showing use of KDB and KGDB:
- video 1 of 6: http://www.youtube.com/watch?v=V6Qc8ppJ_jc - example of a call to panic from a test module (without a debugger)
- video 2 of 6: http://www.youtube.com/watch?v=LqAhY8K3XzI - example of catching the panic with KDB, and looking up the source line with gdb
- video 3 of 6: http://www.youtube.com/watch?v=bBEh_UduX04 - example of a bad access request, and looking up the source line with gdb
- video 4 of 6: http://www.youtube.com/watch?v=MfJU2E0aJwg - example of using a hardware breakpoint with kdb
- video 5 of 6: http://www.youtube.com/watch?v=sWiHV5mt8_k - use an address watch (hardware watchpoint) using kgdb (data access hardware breakpoint on tp_address_ref)
- video 6 of 6: http://www.youtube.com/watch?v=nnopzcwvLTs - use of kgdb over serial - Start up the agent-proxy and connect and hit a breakpoint a sys_sync
Documentation, up-to-date as of 2010, for KDB and KGDB is at: http://kernel.org/pub/linux/kernel/people/jwessel/kdb/
See http://www.ibm.com/developerworks/linux/library/l-kdbug/ for a tutorial for the 2.4.20 kernel (from June 2003)
Here's an article from 2002 on KDB vs. KGDB: http://kerneltrap.org/node/112 It has a good discussion excerpt between Andrew Morton and Keith Owens about the relative merits of KDB versus KGDB.
Here are some questions to answer:
- What kernel versions are supported?
- 2.6.35 and on
- what kernel configs are required to be set?
- see below
- how to invoke the debugger?
- sysrq trigger ('echo g >/proc/sysrq-trigger')
- sysrq serial console combo (in minicom, ctrl-a f g (quickly))
The following descriptions are for a 2.6.35 kernel, using KDB over a serial line between host and target:
All these options on are the "Kernel Hacking" menu.
In order to support KDB, "KGDB" support must be turned on first (even if you aren't using kgdb/gdb)
- CONFIG_DEBUG_KERNEL=y - includes debug information in the kernel compilation - required for basic kernel debugging support
- CONFIG_KGDB=Y - turn on basic kernel debug agent support
- CONFIG_KGDB_SERIAL_CONSOLE=Y - to share a serial console with kgdb.
- Sysrq-g must be used to break in initially.
- Selecting this will automatically set:
- CONFIG_MAGIC_SYSRQ=Y - turn on MAGIC-SYSRQ key support
- CONFIG_KGDB_KDB=Y - actually turn on the KDB debugger feature
Optional other configuration settings:
- CONFIG_FRAME_POINTER=Y - this allows for better backtrace support in KDB
- CONFIG_DEBUG_RODATA=N - disable this in order to support hardware breakpoints on data accesses
- CONFIG_KALLSYMS=Y - this adds symbolic information to the kernel, useful to see symbols instead of addresses
- CONFIG_KDB_KEYBOARD - use KDB with an attached keyboard (not for use with serial console)
- CONFIG_KGDB_TESTS - used to turn on kgdb internal self-tests - see the config help for this for more information
Using gdb to see the kernel source listing
You can use the addresses printed out in kdb, with a host-side gdb session, to see the source code or assembly instructions around a particular address.
The target address can come from a backtrace or register dump (e.g. instruction pointer).
To load the source for a kernel, start gdb (or the appropriate arch-specific gdb) with the vmlinux that matches the image running on target. The kernel should have been compiled with debug symbols (CONFIG_DEBUG_KERNEL=y). gdb will start, and load the symbol information for the kernel.
Use the following commands to see various bits of information:
- source file and line number for an instruction address
- info line *0x<target_addr>
- source lines around an instruction address
- list *0x<target_addr>
- assembly instructions at an address
- disas 0x<target_addr>, or
- x/20i 0x<target_addr>
KDB environment variables
- LINES - set the number of lines for paging output from KDB