KDB

Introduction and basic resources
Here is some information about KDB - the in-kernel debugger for the Linux kernel.

The KDB and KGDB official wiki: https://kgdb.wiki.kernel.org/ (this only has 2 pages?)

Jason Wessel is the current KDB maintainer. Here is a presentation from him at LinuxCon 2010 (August 2010): http://kernel.org/pub/linux/kernel/people/jwessel/dbg_webinar/State_Of_kernel_debugging_LinuxCon2010.pdf

Here are some videos showing use of KDB and KGDB:
 * video 1 of 6: http://www.youtube.com/watch?v=V6Qc8ppJ_jc - example of a call to panic from a test module (without a debugger)
 * video 2 of 6: http://www.youtube.com/watch?v=LqAhY8K3XzI - example of catching the panic with KDB, and looking up the source line with gdb
 * video 3 of 6: http://www.youtube.com/watch?v=bBEh_UduX04 - example of a bad access request, and looking up the source line with gdb
 * video 4 of 6: http://www.youtube.com/watch?v=MfJU2E0aJwg - example of using a hardware breakpoint with kdb
 * video 5 of 6: http://www.youtube.com/watch?v=sWiHV5mt8_k - use an address watch (hardware watchpoint) using kgdb (data access hardware breakpoint on tp_address_ref)
 * video 6 of 6: http://www.youtube.com/watch?v=nnopzcwvLTs - use of kgdb over serial - Start up the agent-proxy and connect and hit a breakpoint a sys_sync

Documentation, up-to-date as of 2010, for KDB and KGDB is at: http://kernel.org/pub/linux/kernel/people/jwessel/kdb/

Older Information
See http://www.ibm.com/developerworks/linux/library/l-kdbug/ for a tutorial for the 2.4.20 kernel (from June 2003)

Here's an article from 2002 on KDB vs. KGDB: http://kerneltrap.org/node/112 It has a good discussion excerpt between Andrew Morton and Keith Owens about the relative merits of KDB versus KGDB.

Kernel Versions supported
kgdb was added to the mainline Linux kernel in version 2.6.26.

kdb support was added to the mainline Linux kernel in version 2.6.35.

Before those versions, kgdb and kdb were available as patches which could be applied to the Linux kernel source.

Kernel configuration
The following descriptions are for a 2.6.35 kernel, using KDB over a serial line between host and target:

All these options on are the "Kernel Hacking" menu.

In order to support KDB, "KGDB" support must be turned on first (even if you aren't using kgdb/gdb)


 * CONFIG_DEBUG_KERNEL=Y - includes debug information in the kernel compilation - required for basic kernel debugging support
 * CONFIG_KGDB=Y - turn on basic kernel debug agent support
 * CONFIG_KGDB_SERIAL_CONSOLE=Y - to share a serial console with kgdb.
 * Sysrq-g must be used to break in initially.
 * Selecting this will automatically set:
 * CONFIG_CONSOLE_POLL=N
 * CONFIG_MAGIC_SYSRQ=Y - turn on MAGIC-SYSRQ key support
 * CONFIG_KGDB_KDB=Y - actually turn on the KDB debugger feature

Optional other configuration settings:
 * CONFIG_FRAME_POINTER=Y - this allows for better backtrace support in KDB
 * CONFIG_DEBUG_RODATA=N - disable this in order to support hardware breakpoints on data accesses
 * CONFIG_KALLSYMS=Y - this adds symbolic information to the kernel, useful to see symbols instead of addresses
 * CONFIG_KDB_KEYBOARD - use KDB with an attached keyboard (not for use with serial console)
 * CONFIG_KGDB_TESTS - used to turn on kgdb internal self-tests - see the config help for this for more information

Enabling kdb at kernel runtime
Once the kernel is compiled with kdb support and is running on your target board, you need to enable it. This can be done on a running system, binding the kdb/kgdb feature to a serial port, by writing a value into the sys filesystem.

If your machine starts a serial console on ttyS0, you can bind kdb/kgdb to this serial console by writing the string "ttyS0" to /sys/module/kgdboc/parameters/kgdboc. The kernel will respond with a message indicating that that driver is registered.

$ echo ttyS0 >/sys/module/kgdboc/parameters/kgdboc kgdb: Registered I/O driver kgdboc.

[ANSWER THIS - can you also do this binding at boot time? - you can for kgdb, but what about kdb?]

Invoking kdb
Once the kernel is running, and the kgdb/kdb is bound to the serial console, you can invoke the debugger in numerous ways.

First, you can enter the debugger using a Magic SysRq command.

kdb will also be entered automatically if the kernel panics.

Finally, you can set a breakpoint (either hardware or software), such that the debugger is invoked when the breakpoint condition is met. For a code breakpoint, this means when the instruction is executed at the breakpoint location, and for a data breakpoint, when the particular access is made at the breakpoint address.

Invoking with Magic SysRq 'g'
To invoke the debugger using the Magic SysRq command, you use the 'g' command, which can be issued any of the ways supported by the Magic SysRq feature. This can be done by 1) typing the key sequence on a connected keyboard, 2) echoing a value to /proc/sysrq-trigger, or 3) sending a break key sequence through the serial console

Examples:
 * sysrq trigger from local shell, via procfs: 'echo g >/proc/sysrq-trigger'
 * sysrq trigger via serial console break sequence:
 * in minicom, type 'ctrl-a f g' (quickly)
 * in telnet, through a server that supports sending a break: type

Invocation by panic
When a kernel panic occurs, then something has gone seriously wrong and the kernel automatically enters kdb. From here you can look at memory, do a traceback, examine registers, and do other operations to find out more about the state of the system and debug the problem.

Invocation by breakpoint
To enter kdb using a breakpoint, first invoke kdb using the Magic SysRq key (see above), then set a breakpoint. Then type 'go' to continue execution. When the breakpoint is hit, the debugger shell will appear.

In the example that follows, items in italics are commands typed by a user. Items following a '$' are commands entered at a shell command (normal Linux user-space runtime), and items following 'kgdb>' are commands entered at the kdb interactive shell.

$ echo g >/proc/sysrq-trigger SysRq : DEBUG Entering kdb (current=0xdfdff040, pid 71) due to Keyboard Entry kdb> bp sys_sync+4 Instruction(i) BP #0 at 0xc00c9f00 (sys_sync+0x4) is enabled addr at 00000000c00c9f00, hardtype=0 installed=0 kdb> go $ sync Entering kdb (current=0xdfdaa360, pid 72) due to Breakpoint @ 0xc00c9f00 kdb> bt Stack traceback for pid 72 0xdfdaa360      72       71  1    0   R  0xdfdaa560 *sync [ ] (unwind_backtrace+0x0/0xe4) from [ ] (show_stack+0x10/0x14) [ ] (show_stack+0x10/0x14) from [ ] (kdb_show_stack+0x58/0x80) [ ] (kdb_show_stack+0x58/0x80) from [ ] (kdb_bt1.clone.0+0x7c/0xcc) [ ] (kdb_bt1.clone.0+0x7c/0xcc) from [ ] (kdb_bt+0x2d4/0x338) [ ] (kdb_bt+0x2d4/0x338) from [ ] (kdb_parse+0x4d4/0x5f8) [ ] (kdb_parse+0x4d4/0x5f8) from [ ] (kdb_main_loop+0x448/0x6b0) [ ] (kdb_main_loop+0x448/0x6b0) from [ ] (kdb_stub+0x210/0x398) [ ] (kdb_stub+0x210/0x398) from [ ] (kgdb_handle_exception+0x384/0x574) [ ] (kgdb_handle_exception+0x384/0x574) from [ ] (kgdb_brk_fn+0x18/0x20) [ ] (kgdb_brk_fn+0x18/0x20) from [ ] (do_undefinstr+0x10c/0x1a8) [ ] (do_undefinstr+0x10c/0x1a8) from [ ] (__und_svc+0x44/0x60) Exception stack(0xdfe09f58 to 0xdfe09fa0) 9f40:                                                      00000000 bec93e74 9f60: 000437ac 00034738 00000001 bec93e74 00000049 00000024 c00230e8 dfe08000 9f80: 00000000 bec93e54 00000000 dfe09fa0 c0022f40 c00c9f00 80000013 ffffffff [ ] (__und_svc+0x44/0x60) from [ ] (sys_sync+0x4/0x28) [ ] (sys_sync+0x4/0x28) from [ ] (ret_fast_syscall+0x0/0x30) kdb>

This example shows an invocation of kdb, followed by setting a breakpoint, then resuming execution with 'go'. Then, at the Linux user-space shell, the 'sync' command is run to cause the breakpoint to occur. When kdb is entered due to the breakpoint, then 'bt' is run to get a backtrace from the stack of the current process.

Using gdb to see the kernel source listing
You can use the addresses printed out in kdb, with a host-side gdb session, to see the source code or assembly instructions around a particular address.

The target address can come from a backtrace or register dump (e.g. instruction pointer).

To load the source for a kernel, start gdb (or the appropriate arch-specific gdb) with the vmlinux that matches the image running on target. The kernel should have been compiled with debug symbols (CONFIG_DEBUG_KERNEL=y). gdb will start, and load the symbol information for the kernel.

Use the following commands to see various bits of information:
 * source file and line number for an instruction address
 * info line *0x
 * source lines around an instruction address
 * list *0x
 * assembly instructions at an address
 * disas 0x, or
 * x/20i 0x

KDB environment variables

 * LINES - set the number of lines for paging output from KDB