Root/
1 | |
2 | Introduction |
3 | ============ |
4 | |
5 | This document describes how to use the dynamic debug (dyndbg) feature. |
6 | |
7 | Dynamic debug is designed to allow you to dynamically enable/disable |
8 | kernel code to obtain additional kernel information. Currently, if |
9 | CONFIG_DYNAMIC_DEBUG is set, then all pr_debug()/dev_dbg() and |
10 | print_hex_dump_debug()/print_hex_dump_bytes() calls can be dynamically |
11 | enabled per-callsite. |
12 | |
13 | If CONFIG_DYNAMIC_DEBUG is not set, print_hex_dump_debug() is just |
14 | shortcut for print_hex_dump(KERN_DEBUG). |
15 | |
16 | For print_hex_dump_debug()/print_hex_dump_bytes(), format string is |
17 | its 'prefix_str' argument, if it is constant string; or "hexdump" |
18 | in case 'prefix_str' is build dynamically. |
19 | |
20 | Dynamic debug has even more useful features: |
21 | |
22 | * Simple query language allows turning on and off debugging |
23 | statements by matching any combination of 0 or 1 of: |
24 | |
25 | - source filename |
26 | - function name |
27 | - line number (including ranges of line numbers) |
28 | - module name |
29 | - format string |
30 | |
31 | * Provides a debugfs control file: <debugfs>/dynamic_debug/control |
32 | which can be read to display the complete list of known debug |
33 | statements, to help guide you |
34 | |
35 | Controlling dynamic debug Behaviour |
36 | =================================== |
37 | |
38 | The behaviour of pr_debug()/dev_dbg()s are controlled via writing to a |
39 | control file in the 'debugfs' filesystem. Thus, you must first mount |
40 | the debugfs filesystem, in order to make use of this feature. |
41 | Subsequently, we refer to the control file as: |
42 | <debugfs>/dynamic_debug/control. For example, if you want to enable |
43 | printing from source file 'svcsock.c', line 1603 you simply do: |
44 | |
45 | nullarbor:~ # echo 'file svcsock.c line 1603 +p' > |
46 | <debugfs>/dynamic_debug/control |
47 | |
48 | If you make a mistake with the syntax, the write will fail thus: |
49 | |
50 | nullarbor:~ # echo 'file svcsock.c wtf 1 +p' > |
51 | <debugfs>/dynamic_debug/control |
52 | -bash: echo: write error: Invalid argument |
53 | |
54 | Viewing Dynamic Debug Behaviour |
55 | =========================== |
56 | |
57 | You can view the currently configured behaviour of all the debug |
58 | statements via: |
59 | |
60 | nullarbor:~ # cat <debugfs>/dynamic_debug/control |
61 | # filename:lineno [module]function flags format |
62 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:323 [svcxprt_rdma]svc_rdma_cleanup =_ "SVCRDMA Module Removed, deregister RPC RDMA transport\012" |
63 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:341 [svcxprt_rdma]svc_rdma_init =_ "\011max_inline : %d\012" |
64 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:340 [svcxprt_rdma]svc_rdma_init =_ "\011sq_depth : %d\012" |
65 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:338 [svcxprt_rdma]svc_rdma_init =_ "\011max_requests : %d\012" |
66 | ... |
67 | |
68 | |
69 | You can also apply standard Unix text manipulation filters to this |
70 | data, e.g. |
71 | |
72 | nullarbor:~ # grep -i rdma <debugfs>/dynamic_debug/control | wc -l |
73 | 62 |
74 | |
75 | nullarbor:~ # grep -i tcp <debugfs>/dynamic_debug/control | wc -l |
76 | 42 |
77 | |
78 | The third column shows the currently enabled flags for each debug |
79 | statement callsite (see below for definitions of the flags). The |
80 | default value, with no flags enabled, is "=_". So you can view all |
81 | the debug statement callsites with any non-default flags: |
82 | |
83 | nullarbor:~ # awk '$3 != "=_"' <debugfs>/dynamic_debug/control |
84 | # filename:lineno [module]function flags format |
85 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c:1603 [sunrpc]svc_send p "svc_process: st_sendto returned %d\012" |
86 | |
87 | |
88 | Command Language Reference |
89 | ========================== |
90 | |
91 | At the lexical level, a command comprises a sequence of words separated |
92 | by spaces or tabs. So these are all equivalent: |
93 | |
94 | nullarbor:~ # echo -c 'file svcsock.c line 1603 +p' > |
95 | <debugfs>/dynamic_debug/control |
96 | nullarbor:~ # echo -c ' file svcsock.c line 1603 +p ' > |
97 | <debugfs>/dynamic_debug/control |
98 | nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
99 | <debugfs>/dynamic_debug/control |
100 | |
101 | Command submissions are bounded by a write() system call. |
102 | Multiple commands can be written together, separated by ';' or '\n'. |
103 | |
104 | ~# echo "func pnpacpi_get_resources +p; func pnp_assign_mem +p" \ |
105 | > <debugfs>/dynamic_debug/control |
106 | |
107 | If your query set is big, you can batch them too: |
108 | |
109 | ~# cat query-batch-file > <debugfs>/dynamic_debug/control |
110 | |
111 | At the syntactical level, a command comprises a sequence of match |
112 | specifications, followed by a flags change specification. |
113 | |
114 | command ::= match-spec* flags-spec |
115 | |
116 | The match-spec's are used to choose a subset of the known pr_debug() |
117 | callsites to which to apply the flags-spec. Think of them as a query |
118 | with implicit ANDs between each pair. Note that an empty list of |
119 | match-specs will select all debug statement callsites. |
120 | |
121 | A match specification comprises a keyword, which controls the |
122 | attribute of the callsite to be compared, and a value to compare |
123 | against. Possible keywords are: |
124 | |
125 | match-spec ::= 'func' string | |
126 | 'file' string | |
127 | 'module' string | |
128 | 'format' string | |
129 | 'line' line-range |
130 | |
131 | line-range ::= lineno | |
132 | '-'lineno | |
133 | lineno'-' | |
134 | lineno'-'lineno |
135 | // Note: line-range cannot contain space, e.g. |
136 | // "1-30" is valid range but "1 - 30" is not. |
137 | |
138 | lineno ::= unsigned-int |
139 | |
140 | The meanings of each keyword are: |
141 | |
142 | func |
143 | The given string is compared against the function name |
144 | of each callsite. Example: |
145 | |
146 | func svc_tcp_accept |
147 | |
148 | file |
149 | The given string is compared against either the full pathname, the |
150 | src-root relative pathname, or the basename of the source file of |
151 | each callsite. Examples: |
152 | |
153 | file svcsock.c |
154 | file kernel/freezer.c |
155 | file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c |
156 | |
157 | module |
158 | The given string is compared against the module name |
159 | of each callsite. The module name is the string as |
160 | seen in "lsmod", i.e. without the directory or the .ko |
161 | suffix and with '-' changed to '_'. Examples: |
162 | |
163 | module sunrpc |
164 | module nfsd |
165 | |
166 | format |
167 | The given string is searched for in the dynamic debug format |
168 | string. Note that the string does not need to match the |
169 | entire format, only some part. Whitespace and other |
170 | special characters can be escaped using C octal character |
171 | escape \ooo notation, e.g. the space character is \040. |
172 | Alternatively, the string can be enclosed in double quote |
173 | characters (") or single quote characters ('). |
174 | Examples: |
175 | |
176 | format svcrdma: // many of the NFS/RDMA server pr_debugs |
177 | format readahead // some pr_debugs in the readahead cache |
178 | format nfsd:\040SETATTR // one way to match a format with whitespace |
179 | format "nfsd: SETATTR" // a neater way to match a format with whitespace |
180 | format 'nfsd: SETATTR' // yet another way to match a format with whitespace |
181 | |
182 | line |
183 | The given line number or range of line numbers is compared |
184 | against the line number of each pr_debug() callsite. A single |
185 | line number matches the callsite line number exactly. A |
186 | range of line numbers matches any callsite between the first |
187 | and last line number inclusive. An empty first number means |
188 | the first line in the file, an empty line number means the |
189 | last number in the file. Examples: |
190 | |
191 | line 1603 // exactly line 1603 |
192 | line 1600-1605 // the six lines from line 1600 to line 1605 |
193 | line -1605 // the 1605 lines from line 1 to line 1605 |
194 | line 1600- // all lines from line 1600 to the end of the file |
195 | |
196 | The flags specification comprises a change operation followed |
197 | by one or more flag characters. The change operation is one |
198 | of the characters: |
199 | |
200 | - remove the given flags |
201 | + add the given flags |
202 | = set the flags to the given flags |
203 | |
204 | The flags are: |
205 | |
206 | p enables the pr_debug() callsite. |
207 | f Include the function name in the printed message |
208 | l Include line number in the printed message |
209 | m Include module name in the printed message |
210 | t Include thread ID in messages not generated from interrupt context |
211 | _ No flags are set. (Or'd with others on input) |
212 | |
213 | For print_hex_dump_debug() and print_hex_dump_bytes(), only 'p' flag |
214 | have meaning, other flags ignored. |
215 | |
216 | For display, the flags are preceded by '=' |
217 | (mnemonic: what the flags are currently equal to). |
218 | |
219 | Note the regexp ^[-+=][flmpt_]+$ matches a flags specification. |
220 | To clear all flags at once, use "=_" or "-flmpt". |
221 | |
222 | |
223 | Debug messages during Boot Process |
224 | ================================== |
225 | |
226 | To activate debug messages for core code and built-in modules during |
227 | the boot process, even before userspace and debugfs exists, use |
228 | dyndbg="QUERY", module.dyndbg="QUERY", or ddebug_query="QUERY" |
229 | (ddebug_query is obsoleted by dyndbg, and deprecated). QUERY follows |
230 | the syntax described above, but must not exceed 1023 characters. Your |
231 | bootloader may impose lower limits. |
232 | |
233 | These dyndbg params are processed just after the ddebug tables are |
234 | processed, as part of the arch_initcall. Thus you can enable debug |
235 | messages in all code run after this arch_initcall via this boot |
236 | parameter. |
237 | |
238 | On an x86 system for example ACPI enablement is a subsys_initcall and |
239 | dyndbg="file ec.c +p" |
240 | will show early Embedded Controller transactions during ACPI setup if |
241 | your machine (typically a laptop) has an Embedded Controller. |
242 | PCI (or other devices) initialization also is a hot candidate for using |
243 | this boot parameter for debugging purposes. |
244 | |
245 | If foo module is not built-in, foo.dyndbg will still be processed at |
246 | boot time, without effect, but will be reprocessed when module is |
247 | loaded later. dyndbg_query= and bare dyndbg= are only processed at |
248 | boot. |
249 | |
250 | |
251 | Debug Messages at Module Initialization Time |
252 | ============================================ |
253 | |
254 | When "modprobe foo" is called, modprobe scans /proc/cmdline for |
255 | foo.params, strips "foo.", and passes them to the kernel along with |
256 | params given in modprobe args or /etc/modprob.d/*.conf files, |
257 | in the following order: |
258 | |
259 | 1. # parameters given via /etc/modprobe.d/*.conf |
260 | options foo dyndbg=+pt |
261 | options foo dyndbg # defaults to +p |
262 | |
263 | 2. # foo.dyndbg as given in boot args, "foo." is stripped and passed |
264 | foo.dyndbg=" func bar +p; func buz +mp" |
265 | |
266 | 3. # args to modprobe |
267 | modprobe foo dyndbg==pmf # override previous settings |
268 | |
269 | These dyndbg queries are applied in order, with last having final say. |
270 | This allows boot args to override or modify those from /etc/modprobe.d |
271 | (sensible, since 1 is system wide, 2 is kernel or boot specific), and |
272 | modprobe args to override both. |
273 | |
274 | In the foo.dyndbg="QUERY" form, the query must exclude "module foo". |
275 | "foo" is extracted from the param-name, and applied to each query in |
276 | "QUERY", and only 1 match-spec of each type is allowed. |
277 | |
278 | The dyndbg option is a "fake" module parameter, which means: |
279 | |
280 | - modules do not need to define it explicitly |
281 | - every module gets it tacitly, whether they use pr_debug or not |
282 | - it doesn't appear in /sys/module/$module/parameters/ |
283 | To see it, grep the control file, or inspect /proc/cmdline. |
284 | |
285 | For CONFIG_DYNAMIC_DEBUG kernels, any settings given at boot-time (or |
286 | enabled by -DDEBUG flag during compilation) can be disabled later via |
287 | the sysfs interface if the debug messages are no longer needed: |
288 | |
289 | echo "module module_name -p" > <debugfs>/dynamic_debug/control |
290 | |
291 | Examples |
292 | ======== |
293 | |
294 | // enable the message at line 1603 of file svcsock.c |
295 | nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
296 | <debugfs>/dynamic_debug/control |
297 | |
298 | // enable all the messages in file svcsock.c |
299 | nullarbor:~ # echo -n 'file svcsock.c +p' > |
300 | <debugfs>/dynamic_debug/control |
301 | |
302 | // enable all the messages in the NFS server module |
303 | nullarbor:~ # echo -n 'module nfsd +p' > |
304 | <debugfs>/dynamic_debug/control |
305 | |
306 | // enable all 12 messages in the function svc_process() |
307 | nullarbor:~ # echo -n 'func svc_process +p' > |
308 | <debugfs>/dynamic_debug/control |
309 | |
310 | // disable all 12 messages in the function svc_process() |
311 | nullarbor:~ # echo -n 'func svc_process -p' > |
312 | <debugfs>/dynamic_debug/control |
313 | |
314 | // enable messages for NFS calls READ, READLINK, READDIR and READDIR+. |
315 | nullarbor:~ # echo -n 'format "nfsd: READ" +p' > |
316 | <debugfs>/dynamic_debug/control |
317 | |
318 | // enable all messages |
319 | nullarbor:~ # echo -n '+p' > <debugfs>/dynamic_debug/control |
320 | |
321 | // add module, function to all enabled messages |
322 | nullarbor:~ # echo -n '+mf' > <debugfs>/dynamic_debug/control |
323 | |
324 | // boot-args example, with newlines and comments for readability |
325 | Kernel command line: ... |
326 | // see whats going on in dyndbg=value processing |
327 | dynamic_debug.verbose=1 |
328 | // enable pr_debugs in 2 builtins, #cmt is stripped |
329 | dyndbg="module params +p #cmt ; module sys +p" |
330 | // enable pr_debugs in 2 functions in a module loaded later |
331 | pc87360.dyndbg="func pc87360_init_device +p; func pc87360_find +p" |
332 |
Branches:
ben-wpan
ben-wpan-stefan
javiroman/ks7010
jz-2.6.34
jz-2.6.34-rc5
jz-2.6.34-rc6
jz-2.6.34-rc7
jz-2.6.35
jz-2.6.36
jz-2.6.37
jz-2.6.38
jz-2.6.39
jz-3.0
jz-3.1
jz-3.11
jz-3.12
jz-3.13
jz-3.15
jz-3.16
jz-3.18-dt
jz-3.2
jz-3.3
jz-3.4
jz-3.5
jz-3.6
jz-3.6-rc2-pwm
jz-3.9
jz-3.9-clk
jz-3.9-rc8
jz47xx
jz47xx-2.6.38
master
Tags:
od-2011-09-04
od-2011-09-18
v2.6.34-rc5
v2.6.34-rc6
v2.6.34-rc7
v3.9