Root/
1 | The MSI Driver Guide HOWTO |
2 | Tom L Nguyen tom.l.nguyen@intel.com |
3 | 10/03/2003 |
4 | Revised Feb 12, 2004 by Martine Silbermann |
5 | email: Martine.Silbermann@hp.com |
6 | Revised Jun 25, 2004 by Tom L Nguyen |
7 | Revised Jul 9, 2008 by Matthew Wilcox <willy@linux.intel.com> |
8 | Copyright 2003, 2008 Intel Corporation |
9 | |
10 | 1. About this guide |
11 | |
12 | This guide describes the basics of Message Signaled Interrupts (MSIs), |
13 | the advantages of using MSI over traditional interrupt mechanisms, how |
14 | to change your driver to use MSI or MSI-X and some basic diagnostics to |
15 | try if a device doesn't support MSIs. |
16 | |
17 | |
18 | 2. What are MSIs? |
19 | |
20 | A Message Signaled Interrupt is a write from the device to a special |
21 | address which causes an interrupt to be received by the CPU. |
22 | |
23 | The MSI capability was first specified in PCI 2.2 and was later enhanced |
24 | in PCI 3.0 to allow each interrupt to be masked individually. The MSI-X |
25 | capability was also introduced with PCI 3.0. It supports more interrupts |
26 | per device than MSI and allows interrupts to be independently configured. |
27 | |
28 | Devices may support both MSI and MSI-X, but only one can be enabled at |
29 | a time. |
30 | |
31 | |
32 | 3. Why use MSIs? |
33 | |
34 | There are three reasons why using MSIs can give an advantage over |
35 | traditional pin-based interrupts. |
36 | |
37 | Pin-based PCI interrupts are often shared amongst several devices. |
38 | To support this, the kernel must call each interrupt handler associated |
39 | with an interrupt, which leads to reduced performance for the system as |
40 | a whole. MSIs are never shared, so this problem cannot arise. |
41 | |
42 | When a device writes data to memory, then raises a pin-based interrupt, |
43 | it is possible that the interrupt may arrive before all the data has |
44 | arrived in memory (this becomes more likely with devices behind PCI-PCI |
45 | bridges). In order to ensure that all the data has arrived in memory, |
46 | the interrupt handler must read a register on the device which raised |
47 | the interrupt. PCI transaction ordering rules require that all the data |
48 | arrives in memory before the value can be returned from the register. |
49 | Using MSIs avoids this problem as the interrupt-generating write cannot |
50 | pass the data writes, so by the time the interrupt is raised, the driver |
51 | knows that all the data has arrived in memory. |
52 | |
53 | PCI devices can only support a single pin-based interrupt per function. |
54 | Often drivers have to query the device to find out what event has |
55 | occurred, slowing down interrupt handling for the common case. With |
56 | MSIs, a device can support more interrupts, allowing each interrupt |
57 | to be specialised to a different purpose. One possible design gives |
58 | infrequent conditions (such as errors) their own interrupt which allows |
59 | the driver to handle the normal interrupt handling path more efficiently. |
60 | Other possible designs include giving one interrupt to each packet queue |
61 | in a network card or each port in a storage controller. |
62 | |
63 | |
64 | 4. How to use MSIs |
65 | |
66 | PCI devices are initialised to use pin-based interrupts. The device |
67 | driver has to set up the device to use MSI or MSI-X. Not all machines |
68 | support MSIs correctly, and for those machines, the APIs described below |
69 | will simply fail and the device will continue to use pin-based interrupts. |
70 | |
71 | 4.1 Include kernel support for MSIs |
72 | |
73 | To support MSI or MSI-X, the kernel must be built with the CONFIG_PCI_MSI |
74 | option enabled. This option is only available on some architectures, |
75 | and it may depend on some other options also being set. For example, |
76 | on x86, you must also enable X86_UP_APIC or SMP in order to see the |
77 | CONFIG_PCI_MSI option. |
78 | |
79 | 4.2 Using MSI |
80 | |
81 | Most of the hard work is done for the driver in the PCI layer. It simply |
82 | has to request that the PCI layer set up the MSI capability for this |
83 | device. |
84 | |
85 | 4.2.1 pci_enable_msi |
86 | |
87 | int pci_enable_msi(struct pci_dev *dev) |
88 | |
89 | A successful call will allocate ONE interrupt to the device, regardless |
90 | of how many MSIs the device supports. The device will be switched from |
91 | pin-based interrupt mode to MSI mode. The dev->irq number is changed |
92 | to a new number which represents the message signaled interrupt. |
93 | This function should be called before the driver calls request_irq() |
94 | since enabling MSIs disables the pin-based IRQ and the driver will not |
95 | receive interrupts on the old interrupt. |
96 | |
97 | 4.2.2 pci_enable_msi_block |
98 | |
99 | int pci_enable_msi_block(struct pci_dev *dev, int count) |
100 | |
101 | This variation on the above call allows a device driver to request multiple |
102 | MSIs. The MSI specification only allows interrupts to be allocated in |
103 | powers of two, up to a maximum of 2^5 (32). |
104 | |
105 | If this function returns 0, it has succeeded in allocating at least as many |
106 | interrupts as the driver requested (it may have allocated more in order |
107 | to satisfy the power-of-two requirement). In this case, the function |
108 | enables MSI on this device and updates dev->irq to be the lowest of |
109 | the new interrupts assigned to it. The other interrupts assigned to |
110 | the device are in the range dev->irq to dev->irq + count - 1. |
111 | |
112 | If this function returns a negative number, it indicates an error and |
113 | the driver should not attempt to request any more MSI interrupts for |
114 | this device. If this function returns a positive number, it will be |
115 | less than 'count' and indicate the number of interrupts that could have |
116 | been allocated. In neither case will the irq value have been |
117 | updated, nor will the device have been switched into MSI mode. |
118 | |
119 | The device driver must decide what action to take if |
120 | pci_enable_msi_block() returns a value less than the number asked for. |
121 | Some devices can make use of fewer interrupts than the maximum they |
122 | request; in this case the driver should call pci_enable_msi_block() |
123 | again. Note that it is not guaranteed to succeed, even when the |
124 | 'count' has been reduced to the value returned from a previous call to |
125 | pci_enable_msi_block(). This is because there are multiple constraints |
126 | on the number of vectors that can be allocated; pci_enable_msi_block() |
127 | will return as soon as it finds any constraint that doesn't allow the |
128 | call to succeed. |
129 | |
130 | 4.2.3 pci_disable_msi |
131 | |
132 | void pci_disable_msi(struct pci_dev *dev) |
133 | |
134 | This function should be used to undo the effect of pci_enable_msi() or |
135 | pci_enable_msi_block(). Calling it restores dev->irq to the pin-based |
136 | interrupt number and frees the previously allocated message signaled |
137 | interrupt(s). The interrupt may subsequently be assigned to another |
138 | device, so drivers should not cache the value of dev->irq. |
139 | |
140 | A device driver must always call free_irq() on the interrupt(s) |
141 | for which it has called request_irq() before calling this function. |
142 | Failure to do so will result in a BUG_ON(), the device will be left with |
143 | MSI enabled and will leak its vector. |
144 | |
145 | 4.3 Using MSI-X |
146 | |
147 | The MSI-X capability is much more flexible than the MSI capability. |
148 | It supports up to 2048 interrupts, each of which can be controlled |
149 | independently. To support this flexibility, drivers must use an array of |
150 | `struct msix_entry': |
151 | |
152 | struct msix_entry { |
153 | u16 vector; /* kernel uses to write alloc vector */ |
154 | u16 entry; /* driver uses to specify entry */ |
155 | }; |
156 | |
157 | This allows for the device to use these interrupts in a sparse fashion; |
158 | for example it could use interrupts 3 and 1027 and allocate only a |
159 | two-element array. The driver is expected to fill in the 'entry' value |
160 | in each element of the array to indicate which entries it wants the kernel |
161 | to assign interrupts for. It is invalid to fill in two entries with the |
162 | same number. |
163 | |
164 | 4.3.1 pci_enable_msix |
165 | |
166 | int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec) |
167 | |
168 | Calling this function asks the PCI subsystem to allocate 'nvec' MSIs. |
169 | The 'entries' argument is a pointer to an array of msix_entry structs |
170 | which should be at least 'nvec' entries in size. On success, the |
171 | function will return 0 and the device will have been switched into |
172 | MSI-X interrupt mode. The 'vector' elements in each entry will have |
173 | been filled in with the interrupt number. The driver should then call |
174 | request_irq() for each 'vector' that it decides to use. |
175 | |
176 | If this function returns a negative number, it indicates an error and |
177 | the driver should not attempt to allocate any more MSI-X interrupts for |
178 | this device. If it returns a positive number, it indicates the maximum |
179 | number of interrupt vectors that could have been allocated. See example |
180 | below. |
181 | |
182 | This function, in contrast with pci_enable_msi(), does not adjust |
183 | dev->irq. The device will not generate interrupts for this interrupt |
184 | number once MSI-X is enabled. The device driver is responsible for |
185 | keeping track of the interrupts assigned to the MSI-X vectors so it can |
186 | free them again later. |
187 | |
188 | Device drivers should normally call this function once per device |
189 | during the initialization phase. |
190 | |
191 | It is ideal if drivers can cope with a variable number of MSI-X interrupts, |
192 | there are many reasons why the platform may not be able to provide the |
193 | exact number a driver asks for. |
194 | |
195 | A request loop to achieve that might look like: |
196 | |
197 | static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) |
198 | { |
199 | while (nvec >= FOO_DRIVER_MINIMUM_NVEC) { |
200 | rc = pci_enable_msix(adapter->pdev, |
201 | adapter->msix_entries, nvec); |
202 | if (rc > 0) |
203 | nvec = rc; |
204 | else |
205 | return rc; |
206 | } |
207 | |
208 | return -ENOSPC; |
209 | } |
210 | |
211 | 4.3.2 pci_disable_msix |
212 | |
213 | void pci_disable_msix(struct pci_dev *dev) |
214 | |
215 | This API should be used to undo the effect of pci_enable_msix(). It frees |
216 | the previously allocated message signaled interrupts. The interrupts may |
217 | subsequently be assigned to another device, so drivers should not cache |
218 | the value of the 'vector' elements over a call to pci_disable_msix(). |
219 | |
220 | A device driver must always call free_irq() on the interrupt(s) |
221 | for which it has called request_irq() before calling this function. |
222 | Failure to do so will result in a BUG_ON(), the device will be left with |
223 | MSI enabled and will leak its vector. |
224 | |
225 | 4.3.3 The MSI-X Table |
226 | |
227 | The MSI-X capability specifies a BAR and offset within that BAR for the |
228 | MSI-X Table. This address is mapped by the PCI subsystem, and should not |
229 | be accessed directly by the device driver. If the driver wishes to |
230 | mask or unmask an interrupt, it should call disable_irq() / enable_irq(). |
231 | |
232 | 4.4 Handling devices implementing both MSI and MSI-X capabilities |
233 | |
234 | If a device implements both MSI and MSI-X capabilities, it can |
235 | run in either MSI mode or MSI-X mode but not both simultaneously. |
236 | This is a requirement of the PCI spec, and it is enforced by the |
237 | PCI layer. Calling pci_enable_msi() when MSI-X is already enabled or |
238 | pci_enable_msix() when MSI is already enabled will result in an error. |
239 | If a device driver wishes to switch between MSI and MSI-X at runtime, |
240 | it must first quiesce the device, then switch it back to pin-interrupt |
241 | mode, before calling pci_enable_msi() or pci_enable_msix() and resuming |
242 | operation. This is not expected to be a common operation but may be |
243 | useful for debugging or testing during development. |
244 | |
245 | 4.5 Considerations when using MSIs |
246 | |
247 | 4.5.1 Choosing between MSI-X and MSI |
248 | |
249 | If your device supports both MSI-X and MSI capabilities, you should use |
250 | the MSI-X facilities in preference to the MSI facilities. As mentioned |
251 | above, MSI-X supports any number of interrupts between 1 and 2048. |
252 | In constrast, MSI is restricted to a maximum of 32 interrupts (and |
253 | must be a power of two). In addition, the MSI interrupt vectors must |
254 | be allocated consecutively, so the system may not be able to allocate |
255 | as many vectors for MSI as it could for MSI-X. On some platforms, MSI |
256 | interrupts must all be targetted at the same set of CPUs whereas MSI-X |
257 | interrupts can all be targetted at different CPUs. |
258 | |
259 | 4.5.2 Spinlocks |
260 | |
261 | Most device drivers have a per-device spinlock which is taken in the |
262 | interrupt handler. With pin-based interrupts or a single MSI, it is not |
263 | necessary to disable interrupts (Linux guarantees the same interrupt will |
264 | not be re-entered). If a device uses multiple interrupts, the driver |
265 | must disable interrupts while the lock is held. If the device sends |
266 | a different interrupt, the driver will deadlock trying to recursively |
267 | acquire the spinlock. |
268 | |
269 | There are two solutions. The first is to take the lock with |
270 | spin_lock_irqsave() or spin_lock_irq() (see |
271 | Documentation/DocBook/kernel-locking). The second is to specify |
272 | IRQF_DISABLED to request_irq() so that the kernel runs the entire |
273 | interrupt routine with interrupts disabled. |
274 | |
275 | If your MSI interrupt routine does not hold the lock for the whole time |
276 | it is running, the first solution may be best. The second solution is |
277 | normally preferred as it avoids making two transitions from interrupt |
278 | disabled to enabled and back again. |
279 | |
280 | 4.6 How to tell whether MSI/MSI-X is enabled on a device |
281 | |
282 | Using 'lspci -v' (as root) may show some devices with "MSI", "Message |
283 | Signalled Interrupts" or "MSI-X" capabilities. Each of these capabilities |
284 | has an 'Enable' flag which will be followed with either "+" (enabled) |
285 | or "-" (disabled). |
286 | |
287 | |
288 | 5. MSI quirks |
289 | |
290 | Several PCI chipsets or devices are known not to support MSIs. |
291 | The PCI stack provides three ways to disable MSIs: |
292 | |
293 | 1. globally |
294 | 2. on all devices behind a specific bridge |
295 | 3. on a single device |
296 | |
297 | 5.1. Disabling MSIs globally |
298 | |
299 | Some host chipsets simply don't support MSIs properly. If we're |
300 | lucky, the manufacturer knows this and has indicated it in the ACPI |
301 | FADT table. In this case, Linux will automatically disable MSIs. |
302 | Some boards don't include this information in the table and so we have |
303 | to detect them ourselves. The complete list of these is found near the |
304 | quirk_disable_all_msi() function in drivers/pci/quirks.c. |
305 | |
306 | If you have a board which has problems with MSIs, you can pass pci=nomsi |
307 | on the kernel command line to disable MSIs on all devices. It would be |
308 | in your best interests to report the problem to linux-pci@vger.kernel.org |
309 | including a full 'lspci -v' so we can add the quirks to the kernel. |
310 | |
311 | 5.2. Disabling MSIs below a bridge |
312 | |
313 | Some PCI bridges are not able to route MSIs between busses properly. |
314 | In this case, MSIs must be disabled on all devices behind the bridge. |
315 | |
316 | Some bridges allow you to enable MSIs by changing some bits in their |
317 | PCI configuration space (especially the Hypertransport chipsets such |
318 | as the nVidia nForce and Serverworks HT2000). As with host chipsets, |
319 | Linux mostly knows about them and automatically enables MSIs if it can. |
320 | If you have a bridge which Linux doesn't yet know about, you can enable |
321 | MSIs in configuration space using whatever method you know works, then |
322 | enable MSIs on that bridge by doing: |
323 | |
324 | echo 1 > /sys/bus/pci/devices/$bridge/msi_bus |
325 | |
326 | where $bridge is the PCI address of the bridge you've enabled (eg |
327 | 0000:00:0e.0). |
328 | |
329 | To disable MSIs, echo 0 instead of 1. Changing this value should be |
330 | done with caution as it can break interrupt handling for all devices |
331 | below this bridge. |
332 | |
333 | Again, please notify linux-pci@vger.kernel.org of any bridges that need |
334 | special handling. |
335 | |
336 | 5.3. Disabling MSIs on a single device |
337 | |
338 | Some devices are known to have faulty MSI implementations. Usually this |
339 | is handled in the individual device driver but occasionally it's necessary |
340 | to handle this with a quirk. Some drivers have an option to disable use |
341 | of MSI. While this is a convenient workaround for the driver author, |
342 | it is not good practise, and should not be emulated. |
343 | |
344 | 5.4. Finding why MSIs are disabled on a device |
345 | |
346 | From the above three sections, you can see that there are many reasons |
347 | why MSIs may not be enabled for a given device. Your first step should |
348 | be to examine your dmesg carefully to determine whether MSIs are enabled |
349 | for your machine. You should also check your .config to be sure you |
350 | have enabled CONFIG_PCI_MSI. |
351 | |
352 | Then, 'lspci -t' gives the list of bridges above a device. Reading |
353 | /sys/bus/pci/devices/*/msi_bus will tell you whether MSI are enabled (1) |
354 | or disabled (0). If 0 is found in any of the msi_bus files belonging |
355 | to bridges between the PCI root and the device, MSIs are disabled. |
356 | |
357 | It is also worth checking the device driver to see whether it supports MSIs. |
358 | For example, it may contain calls to pci_enable_msi(), pci_enable_msix() or |
359 | pci_enable_msi_block(). |
360 |
Branches:
ben-wpan
ben-wpan-stefan
javiroman/ks7010
jz-2.6.34
jz-2.6.34-rc5
jz-2.6.34-rc6
jz-2.6.34-rc7
jz-2.6.35
jz-2.6.36
jz-2.6.37
jz-2.6.38
jz-2.6.39
jz-3.0
jz-3.1
jz-3.11
jz-3.12
jz-3.13
jz-3.15
jz-3.16
jz-3.18-dt
jz-3.2
jz-3.3
jz-3.4
jz-3.5
jz-3.6
jz-3.6-rc2-pwm
jz-3.9
jz-3.9-clk
jz-3.9-rc8
jz47xx
jz47xx-2.6.38
master
Tags:
od-2011-09-04
od-2011-09-18
v2.6.34-rc5
v2.6.34-rc6
v2.6.34-rc7
v3.9