KDB

Introduction and basic resources
Here is some information about KDB - the in-kernel debugger for the Linux kernel.

The KDB and KGDB official wiki: https://kgdb.wiki.kernel.org/ (this only has 2 pages?)

Jason Wessel is the current KDB maintainer. Here is a presentation from him at LinuxCon 2010 (August 2010): http://kernel.org/pub/linux/kernel/people/jwessel/dbg_webinar/State_Of_kernel_debugging_LinuxCon2010.pdf

Here are some videos showing use of KDB and KGDB:
 * video 1 of 6: http://www.youtube.com/watch?v=V6Qc8ppJ_jc - example of a call to panic from a test module (without a debugger)
 * video 2 of 6: http://www.youtube.com/watch?v=LqAhY8K3XzI - example of catching the panic with KDB, and looking up the source line with gdb
 * video 3 of 6: http://www.youtube.com/watch?v=bBEh_UduX04 - example of a bad access request, and looking up the source line with gdb
 * video 4 of 6: http://www.youtube.com/watch?v=MfJU2E0aJwg - example of using a hardware breakpoint with kdb
 * video 5 of 6: http://www.youtube.com/watch?v=sWiHV5mt8_k - use an address watch (hardware watchpoint) using kgdb (data access hardware breakpoint on tp_address_ref)
 * video 6 of 6: http://www.youtube.com/watch?v=nnopzcwvLTs - use of kgdb over serial - Start up the agent-proxy and connect and hit a breakpoint a sys_sync

Documentation, up-to-date as of 2010, for KDB and KGDB is at: http://kernel.org/pub/linux/kernel/people/jwessel/kdb/

Older Information
See http://www.ibm.com/developerworks/linux/library/l-kdbug/ for a tutorial for the 2.4.20 kernel (from June 2003)

Here's an article from 2002 on KDB vs. KGDB: http://kerneltrap.org/node/112 It has a good discussion excerpt between Andrew Morton and Keith Owens about the relative merits of KDB versus KGDB.

Questions
Here are some questions to answer:
 * What kernel versions are supported?
 * 2.6.35 and on
 * what kernel configs are required to be set?
 * see below
 * how to invoke the debugger?
 * sysrq trigger ('echo g >/proc/sysrq-trigger')
 * sysrq serial console combo (in minicom, ctrl-a f g (quickly))
 * panic
 * breakpoint

Kernel configuration
The following descriptions are for a 2.6.35 kernel, using KDB over a serial line between host and target:

All these options on are the "Kernel Hacking" menu.

In order to support KDB, "KGDB" support must be turned on first (even if you aren't using kgdb/gdb)


 * CONFIG_DEBUG_KERNEL=y - includes debug information in the kernel compilation - required for basic kernel debugging support
 * CONFIG_KGDB=Y - turn on basic kernel debug agent support
 * CONFIG_KGDB_SERIAL_CONSOLE=Y - to share a serial console with kgdb.
 * Sysrq-g must be used to break in initially.
 * Selecting this will automatically set:
 * CONFIG_CONSOLE_POLL=N
 * CONFIG_MAGIC_SYSRQ=Y - turn on MAGIC-SYSRQ key support
 * CONFIG_KGDB_KDB=Y - actually turn on the KDB debugger feature

Optional other configuration settings:
 * CONFIG_FRAME_POINTER=Y - this allows for better backtrace support in KDB
 * CONFIG_DEBUG_RODATA=N - disable this in order to support hardware breakpoints on data accesses
 * CONFIG_KALLSYMS=Y - this adds symbolic information to the kernel, useful to see symbols instead of addresses
 * CONFIG_KDB_KEYBOARD - use KDB with an attached keyboard (not for use with serial console)
 * CONFIG_KGDB_TESTS - used to turn on kgdb internal self-tests - see the config help for this for more information

Using gdb to see the kernel source listing
You can use the addresses printed out in kdb, with a host-side gdb session, to see the source code or assembly instructions around a particular address.

The target address can come from a backtrace or register dump (e.g. instruction pointer).

To load the source for a kernel, start gdb (or the appropriate arch-specific gdb) with the vmlinux that matches the image running on target. The kernel should have been compiled with debug symbols (CONFIG_DEBUG_KERNEL=y). gdb will start, and load the symbol information for the kernel.

Use the following commands to see various bits of information:
 * source file and line number for an instruction address
 * info line *0x
 * source lines around an instruction address
 * list *0x
 * assembly instructions at an address
 * disas 0x, or
 * x/20i 0x

KDB environment variables

 * LINES - set the number of lines for paging output from KDB