Kernel Debugging Tips - Revision history

Peter Huewe: Added links to content consolidated with Debugging_by_printing

2012-04-03T21:33:21Z

Added links to content consolidated with Debugging_by_printing

Peter Huewe: Consolidated with Debugging_by_printing

2012-04-03T21:28:21Z

Consolidated with Debugging_by_printing

Peter Huewe: Moved to Debugging by Printing

2012-04-03T20:45:24Z

Moved to Debugging by Printing

Peter Huewe: Undo revision 113000 by Peter Huewe (talk)

2012-04-03T20:44:35Z

Undo revision 113000 by Peter Huewe (talk)

Peter Huewe: Moved to Debugging by Printing

2012-04-03T20:43:50Z

Moved to Debugging by Printing

Tim Bird: /* Accessing the printk buffer after a silent hang on boot */ fix typo

2012-01-20T18:43:01Z

‎Accessing the printk buffer after a silent hang on boot: fix typo

Tim Bird: /* Debugging early boot problems */

2011-08-24T20:29:48Z

‎Debugging early boot problems

Tim Bird: add info about kernel debug with eclipse and QEMU

2011-07-28T23:39:26Z

add info about kernel debug with eclipse and QEMU

Tim Bird: /* Using kernel symbols */

2010-06-15T19:25:03Z

‎Using kernel symbols

Tim Bird: add link to 'find a kernel function line' page

2010-06-15T19:21:58Z

add link to 'find a kernel function line' page

@@ Line 4: / Line 4: @@
 To add your own debug message to the kernel, you can place a "printk()" in the kernel code.
-These messages are logged to an internal buffer, and eventually displayed by the kernel, being emitted to whatever is the active console, at the time of display.  Since the console is not initialized until part-way through the boot sequence, several messages build up in the log until the console is initialize, at which point all buffered messages are displayed at once.
+See [[Debugging_by_printing#Usage| Debugging by printing -> Usage]] for more details.
-−
-After the messages are displayed on the kernel console, they are left in the log buffer (which is a ring buffer), and they can be (and usually are) retrieved later by various commands.
-−
-printk is a variable-argument function, just like printf.  Note that there is no "printf()" function for the kernel.  "printk()" is used to specifically remind kernel developers that the environment is different.
-−
-printk can use many standard C-style printf format characters, but not all of them.  See the kernel source for examples of what you can do.
 === Log levels ===
-Each kernel message can be pre-pended with a tag indicating the importance of the message. The message tags are shown in the file include/linux/kernel.h.  The available tags are:
+Each kernel message can be pre-pended with a tag indicating the importance of the message.
- KERN_EMERG, KERN_ALERT, KERN_CRIT, KERN_WARNING, KERN_NOTICE, KERN_INFO, KERN_DEBUG.
-−
-Note that these tags are defined as simple strings, with angle brackets and a number:
- #define      KERN_INFO       "<6>"
-−
-You could place the tag onto front of the string without using the #define, but that is considered very bad form, and is discouraged.
-−
-The way the tag is placed on the string is with C string concatenation:
- printk(KERN_DEBUG "This is a debug message\n");
-Note that there is no comma between the tag macro and the message string.
+See [[Debugging_by_printing#Log_Levels| Debugging by printing -> Log_Levels]] for more details.
 === Adding timing information ===

@@ Line 51: / Line 51: @@
 === Changing the size of the printk buffer ===
-The size of the buffer that printk writes to is referred to in the kernel source as the "log buffer".
+See [[Debugging_by_printing#Internals_.2F_Changing_the_size_of_the_printk_buffer | Debugging by printing -> Internals / Changing the size of the printk buffer ]]
-−
-You can make the log buffer larger when you build the kernel. This is useful for when additional printk data (such as extra messages inserted for debugging) is produced by the kernel and it overflows the regular buffer. The log buffer is a ring buffer, so later messages will overwrite the earliest messages in the buffer if too much data is written before the log buffer is drained (by, e.g. klogd or dmesg)
-−
-Step 1: Change the kernel log buffer size:
-−
-On the General Setup menu, change "Kernel log buffer size" to a larger number. This corresponds to LOG_BUF_SHIFT, which is the power-of-two size of the log buffer.
-By default, this is 14, which yeilds a log buffer size of 16KB.  Changing this to 17 will yield a log buffer size of 128KB.
-−
-Set the option, then re-build and install the kernel for booting on the target board.
-−
-Step 2: Tell dmesg to retrieve the larger buffer:
-Even though the buffer is larger in the kernel, dmesg will still default to only reading 16K, unless told otherwise.  To read the full buffer, use the '-s' option
-like so:
- $dmesg -s 128000
 == Using kernel symbols ==

@@ Line 80: / Line 80: @@
 == Debugging early boot problems ==
-=== Accessing the printk buffer after a silent hang on boot ===
+See [[Debugging_by_printing#Debugging_early_boot_problems]]
-Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang.  However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.
-−
-The kernel starts putting messages into the printk buffer as soon as it starts.  They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices).  Even though these messages are not printed before the hang, it is still possible in some circumstances
-to dump the printk buffer and see the messages.
-−
-The tricky parts of doing this are:
-−
-) using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader.  You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
-−
-) figuring out the address to use in the bootloader, based on the address of __log_buf in System.map.  You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address.  However, there may be other offsets present as well (such as TEXT_OFFSET).  Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump.  Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change.  So once you figure it out you are set.
-−
-Quinn Jensen writes:
-−
-Something I've found handy when the console is silent is to dump the printk buffer from the boot loader.  To do it you have to know how your boot loader maps memory compared to the kernel.
-−
-==== Redboot example on a Freescale ADS board ====
-Quinn says: Here's what I do with Redboot on i.MX31:
-−
- fgrep printk_buf System.map
-−
-this shows the linked address of the printk_buf, e.g.:
-−
- c02338f0 b printk_buf.16194
-−
-The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0.  So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:
-−
- dump -b 0x802338f0 -l 10000
-−
-And you see the printk buffer that never got flushed to the UART.  Knowing what's there can be '''very''' useful in debugging your console.
-−
-==== U-boot example on an OMAP OSK board ====
-Tim Bird tried these steps and they worked:
-−
- grep __log_buf System.map
-−
-or
-−
- grep __log_buf /proc/kallsyms
-−
-These show:
-−
- c0352d88 B __log_buf
-−
-In the case of the OSK, this address maps to 0x10352d88.  So I reset the target board and use the following:
-−
- OMAP5912 OSK # md 10352d88
-d88: 4c3e353c 78756e69 72657620 6e6f6973    <5>Linux version
-d98: 362e3220 2e32322e 612d3631 6e5f706c     2.6.22.16-alp_n
-da8: 7428206c 64726962 6d697440 6b736564    l (tbird@timdesk
-db8: 2e6d612e 796e6f73 6d6f632e 67282029    .am.sony.com) (g
-dc8: 76206363 69737265 33206e6f 342e342e    cc version 3.4.4
-dd8: 34232029 45525020 54504d45 65755420    ) #4 PREEMPT Tue
- ...
-−
-=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
-If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel.  These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.
-−
-Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):
-<pre>
-> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
-> We've figured out so far that the kernel dies somewhere between  the gunzip and start_kernel.
-−
-Try enabling DEBUG_LL to see if it's an machine ID error.  If you see:
-−
-Error: unrecognized/unsupported processor variant.
-−
-or:
-−
-Error: unrecognized/unsupported machine ID...
-−
-Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.
-−
-If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,.  You can find more details on these low-level debugging hacks via a little googling...
-</pre>
 == Triggering a kernel event ==

← Older revision		Revision as of 20:44, 3 April 2012
Line 78:		Line 78:
	See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.		See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.

		+
		+	== Debugging early boot problems ==
		+	=== Accessing the printk buffer after a silent hang on boot ===
		+	Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.
		+
		+	The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
		+	to dump the printk buffer and see the messages.
		+
		+	The tricky parts of doing this are:
		+
		+	1) using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
		+
		+	2) figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.
		+
		+	Quinn Jensen writes:
		+
		+	Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.
		+
		+	==== Redboot example on a Freescale ADS board ====
		+	Quinn says: Here's what I do with Redboot on i.MX31:
		+
		+	fgrep printk_buf System.map
		+
		+	this shows the linked address of the printk_buf, e.g.:
		+
		+	c02338f0 b printk_buf.16194
		+
		+	The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:
		+
		+	dump -b 0x802338f0 -l 10000
		+
		+	And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.
		+
		+	==== U-boot example on an OMAP OSK board ====
		+	Tim Bird tried these steps and they worked:
		+
		+	grep __log_buf System.map
		+
		+	or
		+
		+	grep __log_buf /proc/kallsyms
		+
		+	These show:
		+
		+	c0352d88 B __log_buf
		+
		+	In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:
		+
		+	OMAP5912 OSK # md 10352d88
		+	10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
		+	10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
		+	10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
		+	10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
		+	10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
		+	10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
		+	...
		+
		+	=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
		+	If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.
		+
		+	Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):
		+	<pre>
		+	> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
		+	> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.
		+
		+	Try enabling DEBUG_LL to see if it's an machine ID error. If you see:
		+
		+	Error: unrecognized/unsupported processor variant.
		+
		+	or:
		+
		+	Error: unrecognized/unsupported machine ID...
		+
		+	Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.
		+
		+	If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
		+	</pre>

	== Triggering a kernel event ==		== Triggering a kernel event ==

← Older revision		Revision as of 20:43, 3 April 2012
Line 78:		Line 78:
	See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.		See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.

−
−	~~== Debugging early boot problems ==~~
−	~~=== Accessing the printk buffer after a silent hang on boot ===~~
−	Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.
−
−	The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
−	~~to dump the printk buffer and see the messages.~~
−
−	~~The tricky parts of doing this are:~~
−
−	1) using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
−
−	2) figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.
−
−	~~Quinn Jensen writes:~~
−
−	~~Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.~~
−
−	~~==== Redboot example on a Freescale ADS board ====~~
−	~~Quinn says: Here's what I do with Redboot on i.MX31:~~
−
−	~~fgrep printk_buf System.map~~
−
−	~~this shows the linked address of the printk_buf, e.g.:~~
−
−	~~c02338f0 b printk_buf.16194~~
−
−	The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:
−
−	~~dump -b 0x802338f0 -l 10000~~
−
−	~~And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.~~
−
−	~~==== U-boot example on an OMAP OSK board ====~~
−	~~Tim Bird tried these steps and they worked:~~
−
−	~~grep __log_buf System.map~~
−
−	or
−
−	~~grep __log_buf /proc/kallsyms~~
−
−	~~These show:~~
−
−	~~c0352d88 B __log_buf~~
−
−	~~In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:~~
−
−	~~OMAP5912 OSK # md 10352d88~~
−	~~10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version~~
−	~~10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n~~
−	~~10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk~~
−	~~10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g~~
−	~~10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4~~
−	~~10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue~~
−	~~...~~
−
−	~~=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===~~
−	If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.
−
−	~~Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):~~
−	~~<pre>~~
−	~~> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.~~
−	~~> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.~~
−
−	~~Try enabling DEBUG_LL to see if it's an machine ID error. If you see:~~
−
−	~~Error: unrecognized/unsupported processor variant.~~
−
−	~~or:~~
−
−	~~Error: unrecognized/unsupported machine ID...~~
−
−	~~Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.~~
−
−	If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
−	~~</pre>~~

	== Triggering a kernel event ==		== Triggering a kernel event ==

@@ Line 70: / Line 70: @@
 You can look up the source code for a function address using your toolchain's addr2line program.
 See [[Find a kernel function line]] or [[Addr2line for kernel debugging]].
 == Debugging early boot problems ==