Root/
1 | NOTE: ksymoops is useless on 2.6. Please use the Oops in its original format |
2 | (from dmesg, etc). Ignore any references in this or other docs to "decoding |
3 | the Oops" or "running it through ksymoops". If you post an Oops from 2.6 that |
4 | has been run through ksymoops, people will just tell you to repost it. |
5 | |
6 | Quick Summary |
7 | ------------- |
8 | |
9 | Find the Oops and send it to the maintainer of the kernel area that seems to be |
10 | involved with the problem. Don't worry too much about getting the wrong person. |
11 | If you are unsure send it to the person responsible for the code relevant to |
12 | what you were doing. If it occurs repeatably try and describe how to recreate |
13 | it. That's worth even more than the oops. |
14 | |
15 | If you are totally stumped as to whom to send the report, send it to |
16 | linux-kernel@vger.kernel.org. Thanks for your help in making Linux as |
17 | stable as humanly possible. |
18 | |
19 | Where is the Oops? |
20 | ---------------------- |
21 | |
22 | Normally the Oops text is read from the kernel buffers by klogd and |
23 | handed to syslogd which writes it to a syslog file, typically |
24 | /var/log/messages (depends on /etc/syslog.conf). Sometimes klogd dies, |
25 | in which case you can run dmesg > file to read the data from the kernel |
26 | buffers and save it. Or you can cat /proc/kmsg > file, however you |
27 | have to break in to stop the transfer, kmsg is a "never ending file". |
28 | If the machine has crashed so badly that you cannot enter commands or |
29 | the disk is not available then you have three options :- |
30 | |
31 | (1) Hand copy the text from the screen and type it in after the machine |
32 | has restarted. Messy but it is the only option if you have not |
33 | planned for a crash. Alternatively, you can take a picture of |
34 | the screen with a digital camera - not nice, but better than |
35 | nothing. If the messages scroll off the top of the console, you |
36 | may find that booting with a higher resolution (eg, vga=791) |
37 | will allow you to read more of the text. (Caveat: This needs vesafb, |
38 | so won't help for 'early' oopses) |
39 | |
40 | (2) Boot with a serial console (see Documentation/serial-console.txt), |
41 | run a null modem to a second machine and capture the output there |
42 | using your favourite communication program. Minicom works well. |
43 | |
44 | (3) Use Kdump (see Documentation/kdump/kdump.txt), |
45 | extract the kernel ring buffer from old memory with using dmesg |
46 | gdbmacro in Documentation/kdump/gdbmacros.txt. |
47 | |
48 | |
49 | Full Information |
50 | ---------------- |
51 | |
52 | NOTE: the message from Linus below applies to 2.4 kernel. I have preserved it |
53 | for historical reasons, and because some of the information in it still |
54 | applies. Especially, please ignore any references to ksymoops. |
55 | |
56 | From: Linus Torvalds <torvalds@osdl.org> |
57 | |
58 | How to track down an Oops.. [originally a mail to linux-kernel] |
59 | |
60 | The main trick is having 5 years of experience with those pesky oops |
61 | messages ;-) |
62 | |
63 | Actually, there are things you can do that make this easier. I have two |
64 | separate approaches: |
65 | |
66 | gdb /usr/src/linux/vmlinux |
67 | gdb> disassemble <offending_function> |
68 | |
69 | That's the easy way to find the problem, at least if the bug-report is |
70 | well made (like this one was - run through ksymoops to get the |
71 | information of which function and the offset in the function that it |
72 | happened in). |
73 | |
74 | Oh, it helps if the report happens on a kernel that is compiled with the |
75 | same compiler and similar setups. |
76 | |
77 | The other thing to do is disassemble the "Code:" part of the bug report: |
78 | ksymoops will do this too with the correct tools, but if you don't have |
79 | the tools you can just do a silly program: |
80 | |
81 | char str[] = "\xXX\xXX\xXX..."; |
82 | main(){} |
83 | |
84 | and compile it with gcc -g and then do "disassemble str" (where the "XX" |
85 | stuff are the values reported by the Oops - you can just cut-and-paste |
86 | and do a replace of spaces to "\x" - that's what I do, as I'm too lazy |
87 | to write a program to automate this all). |
88 | |
89 | Alternatively, you can use the shell script in scripts/decodecode. |
90 | Its usage is: decodecode < oops.txt |
91 | |
92 | The hex bytes that follow "Code:" may (in some architectures) have a series |
93 | of bytes that precede the current instruction pointer as well as bytes at and |
94 | following the current instruction pointer. In some cases, one instruction |
95 | byte or word is surrounded by <> or (), as in "<86>" or "(f00d)". These |
96 | <> or () markings indicate the current instruction pointer. Example from |
97 | i386, split into multiple lines for readability: |
98 | |
99 | Code: f9 0f 8d f9 00 00 00 8d 42 0c e8 dd 26 11 c7 a1 60 ea 2b f9 8b 50 08 a1 |
100 | 64 ea 2b f9 8d 34 82 8b 1e 85 db 74 6d 8b 15 60 ea 2b f9 <8b> 43 04 39 42 54 |
101 | 7e 04 40 89 42 54 8b 43 04 3b 05 00 f6 52 c0 |
102 | |
103 | Finally, if you want to see where the code comes from, you can do |
104 | |
105 | cd /usr/src/linux |
106 | make fs/buffer.s # or whatever file the bug happened in |
107 | |
108 | and then you get a better idea of what happens than with the gdb |
109 | disassembly. |
110 | |
111 | Now, the trick is just then to combine all the data you have: the C |
112 | sources (and general knowledge of what it _should_ do), the assembly |
113 | listing and the code disassembly (and additionally the register dump you |
114 | also get from the "oops" message - that can be useful to see _what_ the |
115 | corrupted pointers were, and when you have the assembler listing you can |
116 | also match the other registers to whatever C expressions they were used |
117 | for). |
118 | |
119 | Essentially, you just look at what doesn't match (in this case it was the |
120 | "Code" disassembly that didn't match with what the compiler generated). |
121 | Then you need to find out _why_ they don't match. Often it's simple - you |
122 | see that the code uses a NULL pointer and then you look at the code and |
123 | wonder how the NULL pointer got there, and if it's a valid thing to do |
124 | you just check against it.. |
125 | |
126 | Now, if somebody gets the idea that this is time-consuming and requires |
127 | some small amount of concentration, you're right. Which is why I will |
128 | mostly just ignore any panic reports that don't have the symbol table |
129 | info etc looked up: it simply gets too hard to look it up (I have some |
130 | programs to search for specific patterns in the kernel code segment, and |
131 | sometimes I have been able to look up those kinds of panics too, but |
132 | that really requires pretty good knowledge of the kernel just to be able |
133 | to pick out the right sequences etc..) |
134 | |
135 | _Sometimes_ it happens that I just see the disassembled code sequence |
136 | from the panic, and I know immediately where it's coming from. That's when |
137 | I get worried that I've been doing this for too long ;-) |
138 | |
139 | Linus |
140 | |
141 | |
142 | --------------------------------------------------------------------------- |
143 | Notes on Oops tracing with klogd: |
144 | |
145 | In order to help Linus and the other kernel developers there has been |
146 | substantial support incorporated into klogd for processing protection |
147 | faults. In order to have full support for address resolution at least |
148 | version 1.3-pl3 of the sysklogd package should be used. |
149 | |
150 | When a protection fault occurs the klogd daemon automatically |
151 | translates important addresses in the kernel log messages to their |
152 | symbolic equivalents. This translated kernel message is then |
153 | forwarded through whatever reporting mechanism klogd is using. The |
154 | protection fault message can be simply cut out of the message files |
155 | and forwarded to the kernel developers. |
156 | |
157 | Two types of address resolution are performed by klogd. The first is |
158 | static translation and the second is dynamic translation. Static |
159 | translation uses the System.map file in much the same manner that |
160 | ksymoops does. In order to do static translation the klogd daemon |
161 | must be able to find a system map file at daemon initialization time. |
162 | See the klogd man page for information on how klogd searches for map |
163 | files. |
164 | |
165 | Dynamic address translation is important when kernel loadable modules |
166 | are being used. Since memory for kernel modules is allocated from the |
167 | kernel's dynamic memory pools there are no fixed locations for either |
168 | the start of the module or for functions and symbols in the module. |
169 | |
170 | The kernel supports system calls which allow a program to determine |
171 | which modules are loaded and their location in memory. Using these |
172 | system calls the klogd daemon builds a symbol table which can be used |
173 | to debug a protection fault which occurs in a loadable kernel module. |
174 | |
175 | At the very minimum klogd will provide the name of the module which |
176 | generated the protection fault. There may be additional symbolic |
177 | information available if the developer of the loadable module chose to |
178 | export symbol information from the module. |
179 | |
180 | Since the kernel module environment can be dynamic there must be a |
181 | mechanism for notifying the klogd daemon when a change in module |
182 | environment occurs. There are command line options available which |
183 | allow klogd to signal the currently executing daemon that symbol |
184 | information should be refreshed. See the klogd manual page for more |
185 | information. |
186 | |
187 | A patch is included with the sysklogd distribution which modifies the |
188 | modules-2.0.0 package to automatically signal klogd whenever a module |
189 | is loaded or unloaded. Applying this patch provides essentially |
190 | seamless support for debugging protection faults which occur with |
191 | kernel loadable modules. |
192 | |
193 | The following is an example of a protection fault in a loadable module |
194 | processed by klogd: |
195 | --------------------------------------------------------------------------- |
196 | Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc |
197 | Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000 |
198 | Aug 29 09:51:01 blizard kernel: *pde = 00000000 |
199 | Aug 29 09:51:01 blizard kernel: Oops: 0002 |
200 | Aug 29 09:51:01 blizard kernel: CPU: 0 |
201 | Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868] |
202 | Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212 |
203 | Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c |
204 | Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c |
205 | Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018 |
206 | Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000) |
207 | Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001 |
208 | Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00 |
209 | Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036 |
210 | Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128] |
211 | Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3 |
212 | --------------------------------------------------------------------------- |
213 | |
214 | Dr. G.W. Wettstein Oncology Research Div. Computing Facility |
215 | Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com |
216 | 820 4th St. N. |
217 | Fargo, ND 58122 |
218 | Phone: 701-234-7556 |
219 | |
220 | |
221 | --------------------------------------------------------------------------- |
222 | Tainted kernels: |
223 | |
224 | Some oops reports contain the string 'Tainted: ' after the program |
225 | counter. This indicates that the kernel has been tainted by some |
226 | mechanism. The string is followed by a series of position-sensitive |
227 | characters, each representing a particular tainted value. |
228 | |
229 | 1: 'G' if all modules loaded have a GPL or compatible license, 'P' if |
230 | any proprietary module has been loaded. Modules without a |
231 | MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by |
232 | insmod as GPL compatible are assumed to be proprietary. |
233 | |
234 | 2: 'F' if any module was force loaded by "insmod -f", ' ' if all |
235 | modules were loaded normally. |
236 | |
237 | 3: 'S' if the oops occurred on an SMP kernel running on hardware that |
238 | hasn't been certified as safe to run multiprocessor. |
239 | Currently this occurs only on various Athlons that are not |
240 | SMP capable. |
241 | |
242 | 4: 'R' if a module was force unloaded by "rmmod -f", ' ' if all |
243 | modules were unloaded normally. |
244 | |
245 | 5: 'M' if any processor has reported a Machine Check Exception, |
246 | ' ' if no Machine Check Exceptions have occurred. |
247 | |
248 | 6: 'B' if a page-release function has found a bad page reference or |
249 | some unexpected page flags. |
250 | |
251 | 7: 'U' if a user or user application specifically requested that the |
252 | Tainted flag be set, ' ' otherwise. |
253 | |
254 | 8: 'D' if the kernel has died recently, i.e. there was an OOPS or BUG. |
255 | |
256 | 9: 'A' if the ACPI table has been overridden. |
257 | |
258 | 10: 'W' if a warning has previously been issued by the kernel. |
259 | (Though some warnings may set more specific taint flags.) |
260 | |
261 | 11: 'C' if a staging driver has been loaded. |
262 | |
263 | 12: 'I' if the kernel is working around a severe bug in the platform |
264 | firmware (BIOS or similar). |
265 | |
266 | The primary reason for the 'Tainted: ' string is to tell kernel |
267 | debuggers if this is a clean kernel or if anything unusual has |
268 | occurred. Tainting is permanent: even if an offending module is |
269 | unloaded, the tainted value remains to indicate that the kernel is not |
270 | trustworthy. |
271 |
Branches:
ben-wpan
ben-wpan-stefan
javiroman/ks7010
jz-2.6.34
jz-2.6.34-rc5
jz-2.6.34-rc6
jz-2.6.34-rc7
jz-2.6.35
jz-2.6.36
jz-2.6.37
jz-2.6.38
jz-2.6.39
jz-3.0
jz-3.1
jz-3.11
jz-3.12
jz-3.13
jz-3.15
jz-3.16
jz-3.18-dt
jz-3.2
jz-3.3
jz-3.4
jz-3.5
jz-3.6
jz-3.6-rc2-pwm
jz-3.9
jz-3.9-clk
jz-3.9-rc8
jz47xx
jz47xx-2.6.38
master
Tags:
od-2011-09-04
od-2011-09-18
v2.6.34-rc5
v2.6.34-rc6
v2.6.34-rc7
v3.9