Difference between revisions of "Addr2line for kernel debugging"
Line 182: | Line 182: | ||
in That line we have [http://en.wikipedia.org/wiki/Program_counter the program counter] | in That line we have [http://en.wikipedia.org/wiki/Program_counter the program counter] | ||
[ 88.635314] pc : [<c01b063c>] lr : [<c01b0640>] psr: a0000013 | [ 88.635314] pc : [<c01b063c>] lr : [<c01b0640>] psr: a0000013 | ||
− | # arm-none-linux-gnueabi-addr2line -f -e vmlinux c01b063c | + | |
− | msmfb_suspend | + | # arm-none-linux-gnueabi-addr2line -f -e vmlinux c01b063c |
− | /home/gnutoo/embedded/htcdream/SHR/kernel/linux/drivers/video/msm/msm_fb.c:485 | + | msmfb_suspend |
+ | /home/gnutoo/embedded/htcdream/SHR/kernel/linux/drivers/video/msm/msm_fb.c:485 |
Revision as of 13:39, 22 May 2010
Introduction
In this howto we will look how to debug a kernel without KGDB(for instance if we don't have serial available)
Howto
Here's an example of Oops(null pointer exception):
# while true;do dmesg -c;done [ 87.915496] request_suspend_state: sleep (0->3) at 82181848246 (2010-05-22 19:29:45.776824970 UTC) [ 88.028686] deinit sharp panel [ 88.635070] Unable to handle kernel NULL pointer dereference at virtual address 00000fc5 [ 88.635101] pgd = c5db8000 [ 88.635131] [00000fc5] *pgd=16aba031, *pte=00000000, *ppte=00000000 [ 88.635162] Internal error: Oops: 11 [#1] PREEMPT [ 88.635192] last sysfs file: /sys/power/state [ 88.635192] Modules linked in: [ 88.635223] CPU: 0 Not tainted (2.6.32 #385) [ 88.635253] PC is at msmfb_suspend+0x1c/0x2c [ 88.635284] LR is at msmfb_suspend+0x20/0x2c [ 88.635314] pc : [<c01b063c>] lr : [<c01b0640>] psr: a0000013 [ 88.635314] sp : c6847f38 ip : c6847f50 fp : c6847f4c [ 88.635345] r10: c6847f84 r9 : c68009a8 r8 : c04e47e8 [ 88.635345] r7 : c04e4818 r6 : c68e6ea0 r5 : c0535c1c r4 : c68e6ea0 [ 88.635375] r3 : 00000fc5 r2 : c04e1f64 r1 : c6846000 r0 : 00000fc5 [ 88.635406] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel [ 88.635437] Control: 00c5387d Table: 15db8008 DAC: 00000017 [ 88.635437] [ 88.635437] PC: 0xc01b05bc: [ 88.635467] 05bc e58de000 e58dc008 e58dc004 ebfffe7d e24bd014 e89da830 e1a0c00d e92dd830 [ 88.635528] 05dc e24cb004 e24dd010 e1a04001 e1a05000 ebfff6df e594000c e894000e e082e000 [ 88.635559] 05fc e3a0c000 e1a00005 e0813003 e58de000 e58dc008 e58dc004 ebfffe6a e24bd014 [ 88.635620] 061c e89da830 e1a0c00d e92dd818 e24cb004 e5103038 e1a04000 e1a00003 e1a0e00f [ 88.635681] 063c e593f000 e2840074 eb079177 e89da818 e1a0c00d e92ddff0 e24cb004 e24dd02c [ 88.635742] 065c e5907004 e1a04000 e10f6000 f10c0080 e3a00001 ebfa5e63 e24b0034 ebfb087b [ 88.635772] 067c e3a01e11 e51b2034 e18100d4 e51b3030 e0520000 e0c31001 e2502000 b28225ee [ 88.635833] 069c b282296b e59f0290 b2822c0a e1a03fc2 e0e32091 e3530000 da00005b e59f027c [ 88.635894] [ 88.635894] LR: 0xc01b05c0: [ 88.635894] 05c0 e58dc008 e58dc004 ebfffe7d e24bd014 e89da830 e1a0c00d e92dd830 e24cb004 [ 88.635955] 05e0 e24dd010 e1a04001 e1a05000 ebfff6df e594000c e894000e e082e000 e3a0c000 [ 88.636016] 0600 e1a00005 e0813003 e58de000 e58dc008 e58dc004 ebfffe6a e24bd014 e89da830 [ 88.636077] 0620 e1a0c00d e92dd818 e24cb004 e5103038 e1a04000 e1a00003 e1a0e00f e593f000 [ 88.636108] 0640 e2840074 eb079177 e89da818 e1a0c00d e92ddff0 e24cb004 e24dd02c e5907004 [ 88.636169] 0660 e1a04000 e10f6000 f10c0080 e3a00001 ebfa5e63 e24b0034 ebfb087b e3a01e11 [ 88.636230] 0680 e51b2034 e18100d4 e51b3030 e0520000 e0c31001 e2502000 b28225ee b282296b [ 88.636260] 06a0 e59f0290 b2822c0a e1a03fc2 e0e32091 e3530000 da00005b e59f027c e59f127c [ 88.636322] [ 88.636322] SP: 0xc6847eb8: [ 88.636352] 7eb8 c6847f04 c6847ec8 c020a08c c0085e50 c69511a4 00000002 ffffffff c6847f24 [ 88.636383] 7ed8 c68e6ea0 c04e4818 c6847f4c c6847ef0 c0028b4c c0028300 00000fc5 c6846000 [ 88.636444] 7ef8 c04e1f64 00000fc5 c68e6ea0 c0535c1c c68e6ea0 c04e4818 c04e47e8 c68009a8 [ 88.636505] 7f18 c6847f84 c6847f4c c6847f50 c6847f38 c01b0640 c01b063c a0000013 ffffffff [ 88.636566] 7f38 c01b0620 c6846000 c6847f74 c6847f50 c0083dc8 c01b062c c04e47ec c68009a0 [ 88.636596] 7f58 c6846000 c68009a0 c0083cd0 00000000 c6847fc4 c6847f78 c00640f0 c0083cdc [ 88.636657] 7f78 00000000 c04e47ec 00000000 00000000 c6825160 c00684b8 c6847f90 c6847f90 [ 88.636718] 7f98 c6847fc4 c6829ee0 00000002 c0063fa4 c68009a0 00000000 00000000 00000000 [ 88.636749] [ 88.636779] IP: 0xc6847ed0: [ 88.636779] 7ed0 ffffffff c6847f24 c68e6ea0 c04e4818 c6847f4c c6847ef0 c0028b4c c0028300 [ 88.636840] 7ef0 00000fc5 c6846000 c04e1f64 00000fc5 c68e6ea0 c0535c1c c68e6ea0 c04e4818 [ 88.636871] 7f10 c04e47e8 c68009a8 c6847f84 c6847f4c c6847f50 c6847f38 c01b0640 c01b063c [ 88.636932] 7f30 a0000013 ffffffff c01b0620 c6846000 c6847f74 c6847f50 c0083dc8 c01b062c [ 88.636993] 7f50 c04e47ec c68009a0 c6846000 c68009a0 c0083cd0 00000000 c6847fc4 c6847f78 [ 88.637023] 7f70 c00640f0 c0083cdc 00000000 c04e47ec 00000000 00000000 c6825160 c00684b8 [ 88.637084] 7f90 c6847f90 c6847f90 c6847fc4 c6829ee0 00000002 c0063fa4 c68009a0 00000000 [ 88.637145] 7fb0 00000000 00000000 c6847ff4 c6847fc8 c0068260 c0063fb0 00000000 00000000 [ 88.637207] [ 88.637207] FP: 0xc6847ecc: [ 88.637207] 7ecc 00000002 ffffffff c6847f24 c68e6ea0 c04e4818 c6847f4c c6847ef0 c0028b4c [ 88.637268] 7eec c0028300 00000fc5 c6846000 c04e1f64 00000fc5 c68e6ea0 c0535c1c c68e6ea0 [ 88.637298] 7f0c c04e4818 c04e47e8 c68009a8 c6847f84 c6847f4c c6847f50 c6847f38 c01b0640 [ 88.637359] 7f2c c01b063c a0000013 ffffffff c01b0620 c6846000 c6847f74 c6847f50 c0083dc8 [ 88.637420] 7f4c c01b062c c04e47ec c68009a0 c6846000 c68009a0 c0083cd0 00000000 c6847fc4 [ 88.637481] 7f6c c6847f78 c00640f0 c0083cdc 00000000 c04e47ec 00000000 00000000 c6825160 [ 88.637512] 7f8c c00684b8 c [ 88.635314] pc : [<c01b063c>] lr : [<c01b0640>] psr: a00000136847f90 c6847f90 c6847fc4 c6829ee0 00000002 c0063fa4 c68009a0 [ 88.637573] 7fac 00000000 00000000 00000000 c6847ff4 c6847fc8 c0068260 c0063fb0 00000000 [ 88.637634] [ 88.637634] R1: 0xc6845f80: [ 88.637634] 5f80 00000000 00000000 c68245c0 c004b5bc c04e4220 c04e4220 c6845fc4 c6829f40 [ 88.637695] 5fa0 00000002 c006f2f4 00000000 00000000 00000000 00000000 c6845ff4 c6845fc8 [ 88.637756] 5fc0 c0068260 c006f300 00000000 00000000 c6845fd0 c6845fd0 00000000 00000000 [ 88.637786] 5fe0 00000000 00000000 00000000 c6845ff8 c00299d8 c00681e4 ffffffff ffffffff [ 88.637847] 6000 00000000 00000002 00000000 c6825160 c04e2530 00000000 00000017 c6825160 [ 88.637908] 6020 c04e2060 c6846000 c04e2060 c6b58900 c6b58900 c6a08ba0 c6847d24 c6847cc0 [ 88.637939] 6040 c0393f34 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 88.638000] 6060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 88.638061] [ 88.638061] R2: 0xc04e1ee4: [ 88.638061] 1ee4 00000090 00000000 c00417f8 c0435ac4 c0435acc c0435ad0 00000000 00000002 [ 88.638122] 1f04 00000003 00000005 0000001d 00000004 c0435954 c000e478 00000000 00000000 [ 88.638153] 1f24 00000000 00000000 c0434ed0 c04f9820 00000000 00000000 00000000 c01ddfcc [ 88.638214] 1f44 00000000 00000000 00000000 00000000 00000000 c04e1f7c c68f81e0 00000000 [ 88.638275] 1f64 c04e4818 c68e6ea0 00000097 c00424a0 c00423f0 00000001 00000000 00000000 [ 88.638305] 1f84 00000000 00000000 00000000 00000000 00000000 00000000 c0042454 c004240c [ 88.638366] 1fa4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c0042c94 [ 88.638427] 1fc4 c0042be4 00000000 00000000 00000000 c0434ecc c04f9820 00000000 00000000 [ 88.638458] [ 88.638458] R4: 0xc68e6e20: [ 88.638488] 6e20 c68e0910 c692c180 00000001 c7200000 00000000 c053bd2c 00000000 00000000 [ 88.638549] 6e40 c68e6e50 00000000 00000000 00000000 c68e6c00 c68e09c8 00000140 000001e0 [ 88.638580] 6e60 00000000 00000000 00000fc5 00000fc4 00000004 00000650 00000000 00000000 [ 88.638641] 6e80 00000140 000001e0 c680a000 c6973eec c04fa208 00000032 c01b0b28 00000000 [ 88.638702] 6ea0 c04e1f64 c6951188 00000096 c01b0620 c01b1128 00000000 c5d95b58 c053bea0 [ 88.638732] 6ec0 00000101 c0450d78 ffffab02 00000000 00000973 00000000 00000000 00000000 [ 88.638793] 6ee0 1fc8f477 00000015 00000000 00000000 0f8c01d0 00000000 0b248caa 00000052 [ 88.638854] 6f00 00000001 c68e6f04 c68e6f04 c68e6f0c c68e6f0c c6991f20 c6991f40 c68e6f1c [ 88.638885] [ 88.638885] R5: 0xc0535b9c: [ 88.638916] 5b9c 00000000 00000000 00000000 00000000 00000000 c05408b8 c0512988 00000300 [ 88.638977] 5bbc c043bf90 7fffffff 00000000 00000000 00000000 00000000 00000000 00000000 [ 88.639007] 5bdc 00000000 00000000 00000000 00000000 00000000 0d1cef11 00000000 0d1cef11 [ 88.639068] 5bfc 00000000 c05425a0 c050fcd8 c695e348 c695e348 00000000 000001b1 00000000 [ 88.639099] 5c1c 00000003 00000000 00000000 00000000 00000000 c68d5d00 00000114 00000000 [ 88.639160] 5c3c 00000000 00000000 00000000 00000002 00000000 0000253b 00000000 00000000 [ 88.639221] 5c5c 00000000 00000000 00000000 00000000 00000000 00000000 00001d9f 00000000 [ 88.639251] 5c7c 00000000 0000090d 00000000 0000125e 00000000 00000003 00000000 00000000 [ 88.639312] [ 88.639312] R6: 0xc68e6e20: [ 88.639343] 6e20 c68e0910 c692c180 00000001 c7200000 00000000 c053bd2c 00000000 00000000 [ 88.639373] 6e40 c68e6e50 00000000 00000000 00000000 c68e6c00 c68e09c8 00000140 000001e0 [ 88.639434] 6e60 00000000 00000000 00000fc5 00000fc4 00000004 00000650 00000000 00000000 [ 88.639495] 6e80 00000140 000001e0 c680a000 c6973eec c04fa208 00000032 c01b0b28 00000000 [ 88.639526] 6ea0 c04e1f64 c6951188 00000096 c01b0620 c01b1128 00000000 c5d95b58 c053bea0 [ 88.639587] 6ec0 00000101 c0450d78 ffffab02 00000000 00000973 00000000 00000000 00000000 [ 88.639648] 6ee0 1fc8f477 00000015 00000000 00000000 0f8c01d0 00000000 0b248caa 00000052 [ 88.639678] 6f00 00000001 c68e6f04 c68e6f04 c68e6f0c c68e6f0c c6991f20 c6991f40 c68e6f1c [ 88.639739] [ 88.639739] R7: 0xc04e4798: [ 88.639739] 4798 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c00828f8 [ 88.639801] 47b8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 88.639862] 47d8 00000001 00000001 c04e47e0 c04e47e0 00000001 c68009a0 c04e47f0 c04e47f0 [ 88.639892] 47f8 c0083cd0 00000000 c04e4800 c04e4800 c0083abc 00000000 c04e4810 c04e4810 [ 88.639953] 4818 c050033c c04e1f64 c0083eb0 c04e483c 00000000 0000003c 00000043 c68222e0 [ 88.640014] 4838 00000000 c0084170 00000000 00000000 00000078 00008000 0000000a 00000000 [ 88.640045] 4858 00000000 c0535c34 c0086a94 c04dd474 00000000 00000000 00000000 c68bd6c0 [ 88.640106] 4878 00120001 00000000 00000001 00000114 00000000 00000000 00000000 c04e4894 [ 88.640167] [ 88.640167] R8: 0xc04e4768: [ 88.640167] 4768 00000000 74737461 00000000 c0083204 00000000 c05335c0 00000000 c04e4784 [ 88.640228] 4788 c04e4784 c0082e64 00000003 00000000 00000000 00000000 00000000 00000000 [ 88.640289] 47a8 00000000 00000000 00000000 c00828f8 00000000 00000000 00000000 00000000 [ 88.640319] 47c8 00000000 00000000 00000000 00000000 00000001 00000001 c04e47e0 c04e47e0 [ 88.640380] 47e8 00000001 c68009a0 c04e47f0 c04e47f0 c0083cd0 00000000 c04e4800 c04e4800 [ 88.640441] 4808 c0083abc 00000000 c04e4810 c04e4810 c050033c c04e1f64 c0083eb0 c04e483c [ 88.640472] 4828 00000000 0000003c 00000043 c68222e0 00000000 c0084170 00000000 00000000 [ 88.640533] 4848 00000078 00008000 0000000a 00000000 00000000 c0535c34 c0086a94 c04dd474 [ 88.640594] [ 88.640594] R9: 0xc6800928: [ 88.640594] 0928 ffffffff ffffffff ffffffff ffffffff fffbffff ffffffff 65776f70 00302e72 [ 88.640655] 0948 ffffffff ffffffff ffffffff fff7ffff ffffffff ffffffff 76697264 ff007265 [ 88.640716] 0968 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff c68009a0 c6800984 [ 88.640747] 0988 c6800984 c04551e4 00000001 00000000 00000000 00000000 c68009a0 c68009a0 [ 88.640808] 09a8 c68009a8 c68009a8 c04e47ec c6800980 c6825160 00000000 75626564 ffff0067 [ 88.640869] 09c8 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 75626564 ffff0067 [ 88.640899] 09e8 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 66757063 00716572 [ 88.640960] 0a08 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 66757063 00716572 [ 88.641021] [ 88.641021] R10: 0xc6847f04: [ 88.641021] 7f04 c0535c1c c68e6ea0 c04e4818 c04e47e8 c68009a8 c6847f84 c6847f4c c6847f50 [ 88.641082] 7f24 c6847f38 c01b0640 c01b063c a0000013 ffffffff c01b0620 c6846000 c6847f74 [ 88.641143] 7f44 c6847f50 c0083dc8 c01b062c c04e47ec c68009a0 c6846000 c68009a0 c0083cd0 [ 88.641204] 7f64 00000000 c6847fc4 c6847f78 c00640f0 c0083cdc 00000000 c04e47ec 00000000 [ 88.641235] 7f84 00000000 c6825160 c00684b8 c6847f90 c6847f90 c6847fc4 c6829ee0 00000002 [ 88.641296] 7fa4 c0063fa4 c68009a0 00000000 00000000 00000000 c6847ff4 c6847fc8 c0068260 [ 88.641357] 7fc4 c0063fb0 00000000 00000000 c6847fd0 c6847fd0 00000000 00000000 00000000 [ 88.641387] 7fe4 00000000 00000000 c6847ff8 c00299d8 c00681e4 ffffffff ffffffff c6a52720 [ 88.641448] Process suspend (pid: 8, stack limit = 0xc6846268) [ 88.641479] Stack: (0xc6847f38 to 0xc6848000) [ 88.641510] 7f20: c01b0620 c6846000 [ 88.641540] 7f40: c6847f74 c6847f50 c0083dc8 c01b062c c04e47ec c68009a0 c6846000 c68009a0 [ 88.641571] 7f60: c0083cd0 00000000 c6847fc4 c6847f78 c00640f0 c0083cdc 00000000 c04e47ec [ 88.641601] 7f80: 00000000 00000000 c6825160 c00684b8 c6847f90 c6847f90 c6847fc4 c6829ee0 [ 88.641632] 7fa0: 00000002 c0063fa4 c68009a0 00000000 00000000 00000000 c6847ff4 c6847fc8 [ 88.641662] 7fc0: c0068260 c0063fb0 00000000 00000000 c6847fd0 c6847fd0 00000000 00000000 [ 88.641693] 7fe0: 00000000 00000000 00000000 c6847ff8 c00299d8 c00681e4 ffffffff ffffffff [ 88.641784] [<c01b063c>] (msmfb_suspend+0x1c/0x2c) from [<c0083dc8>] (early_suspend+0xf8/0x1a4) [ 88.641845] [<c0083dc8>] (early_suspend+0xf8/0x1a4) from [<c00640f0>] (worker_thread+0x14c/0x240) [ 88.641906] [<c00640f0>] (worker_thread+0x14c/0x240) from [<c0068260>] (kthread+0x88/0x90) [ 88.641967] [<c0068260>] (kthread+0x88/0x90) from [<c00299d8>] (kernel_thread_exit+0x0/0x8) [ 88.641998] Code: e5103038 e1a04000 e1a00003 e1a0e00f (e593f000) [ 88.642059] ---[ end trace a026a5e39ad32e4a ]---
We have function names but not line numbers. Addr2line can solve that: in That line we have the program counter
[ 88.635314] pc : [<c01b063c>] lr : [<c01b0640>] psr: a0000013
# arm-none-linux-gnueabi-addr2line -f -e vmlinux c01b063c msmfb_suspend /home/gnutoo/embedded/htcdream/SHR/kernel/linux/drivers/video/msm/msm_fb.c:485