Root/Documentation/edac.txt

1
2
3EDAC - Error Detection And Correction
4
5Written by Doug Thompson <dougthompson@xmission.com>
67 Dec 2005
717 Jul 2007 Updated
8
9
10EDAC is maintained and written by:
11
12    Doug Thompson, Dave Jiang, Dave Peterson et al,
13    original author: Thayne Harbaugh,
14
15Contact:
16    website: bluesmoke.sourceforge.net
17    mailing list: bluesmoke-devel@lists.sourceforge.net
18
19"bluesmoke" was the name for this device driver when it was "out-of-tree"
20and maintained at sourceforge.net. When it was pushed into 2.6.16 for the
21first time, it was renamed to 'EDAC'.
22
23The bluesmoke project at sourceforge.net is now utilized as a 'staging area'
24for EDAC development, before it is sent upstream to kernel.org
25
26At the bluesmoke/EDAC project site is a series of quilt patches against
27recent kernels, stored in a SVN repository. For easier downloading, there
28is also a tarball snapshot available.
29
30============================================================================
31EDAC PURPOSE
32
33The 'edac' kernel module goal is to detect and report errors that occur
34within the computer system running under linux.
35
36MEMORY
37
38In the initial release, memory Correctable Errors (CE) and Uncorrectable
39Errors (UE) are the primary errors being harvested. These types of errors
40are harvested by the 'edac_mc' class of device.
41
42Detecting CE events, then harvesting those events and reporting them,
43CAN be a predictor of future UE events. With CE events, the system can
44continue to operate, but with less safety. Preventive maintenance and
45proactive part replacement of memory DIMMs exhibiting CEs can reduce
46the likelihood of the dreaded UE events and system 'panics'.
47
48NON-MEMORY
49
50A new feature for EDAC, the edac_device class of device, was added in
51the 2.6.23 version of the kernel.
52
53This new device type allows for non-memory type of ECC hardware detectors
54to have their states harvested and presented to userspace via the sysfs
55interface.
56
57Some architectures have ECC detectors for L1, L2 and L3 caches, along with DMA
58engines, fabric switches, main data path switches, interconnections,
59and various other hardware data paths. If the hardware reports it, then
60a edac_device device probably can be constructed to harvest and present
61that to userspace.
62
63
64PCI BUS SCANNING
65
66In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
67in order to determine if errors are occurring on data transfers.
68
69The presence of PCI Parity errors must be examined with a grain of salt.
70There are several add-in adapters that do NOT follow the PCI specification
71with regards to Parity generation and reporting. The specification says
72the vendor should tie the parity status bits to 0 if they do not intend
73to generate parity. Some vendors do not do this, and thus the parity bit
74can "float" giving false positives.
75
76In the kernel there is a PCI device attribute located in sysfs that is
77checked by the EDAC PCI scanning code. If that attribute is set,
78PCI parity/error scanning is skipped for that device. The attribute
79is:
80
81    broken_parity_status
82
83as is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for
84PCI devices.
85
86FUTURE HARDWARE SCANNING
87
88EDAC will have future error detectors that will be integrated with
89EDAC or added to it, in the following list:
90
91    MCE Machine Check Exception
92    MCA Machine Check Architecture
93    NMI NMI notification of ECC errors
94    MSRs Machine Specific Register error cases
95    and other mechanisms.
96
97These errors are usually bus errors, ECC errors, thermal throttling
98and the like.
99
100
101============================================================================
102EDAC VERSIONING
103
104EDAC is composed of a "core" module (edac_core.ko) and several Memory
105Controller (MC) driver modules. On a given system, the CORE
106is loaded and one MC driver will be loaded. Both the CORE and
107the MC driver (or edac_device driver) have individual versions that reflect
108current release level of their respective modules.
109
110Thus, to "report" on what version a system is running, one must report both
111the CORE's and the MC driver's versions.
112
113
114LOADING
115
116If 'edac' was statically linked with the kernel then no loading is
117necessary. If 'edac' was built as modules then simply modprobe the
118'edac' pieces that you need. You should be able to modprobe
119hardware-specific modules and have the dependencies load the necessary core
120modules.
121
122Example:
123
124$> modprobe amd76x_edac
125
126loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
127core module.
128
129
130============================================================================
131EDAC sysfs INTERFACE
132
133EDAC presents a 'sysfs' interface for control, reporting and attribute
134reporting purposes.
135
136EDAC lives in the /sys/devices/system/edac directory.
137
138Within this directory there currently reside 2 'edac' components:
139
140    mc memory controller(s) system
141    pci PCI control and status system
142
143
144============================================================================
145Memory Controller (mc) Model
146
147First a background on the memory controller's model abstracted in EDAC.
148Each 'mc' device controls a set of DIMM memory modules. These modules are
149laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can
150be multiple csrows and multiple channels.
151
152Memory controllers allow for several csrows, with 8 csrows being a typical value.
153Yet, the actual number of csrows depends on the electrical "loading"
154of a given motherboard, memory controller and DIMM characteristics.
155
156Dual channels allows for 128 bit data transfers to the CPU from memory.
157Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs
158(FB-DIMMs). The following example will assume 2 channels:
159
160
161        Channel 0 Channel 1
162    ===================================
163    csrow0 | DIMM_A0 | DIMM_B0 |
164    csrow1 | DIMM_A0 | DIMM_B0 |
165    ===================================
166
167    ===================================
168    csrow2 | DIMM_A1 | DIMM_B1 |
169    csrow3 | DIMM_A1 | DIMM_B1 |
170    ===================================
171
172In the above example table there are 4 physical slots on the motherboard
173for memory DIMMs:
174
175    DIMM_A0
176    DIMM_B0
177    DIMM_A1
178    DIMM_B1
179
180Labels for these slots are usually silk screened on the motherboard. Slots
181labeled 'A' are channel 0 in this example. Slots labeled 'B'
182are channel 1. Notice that there are two csrows possible on a
183physical DIMM. These csrows are allocated their csrow assignment
184based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
185is placed in each Channel, the csrows cross both DIMMs.
186
187Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
188Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
189will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
190when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
191csrow1 will be populated. The pattern repeats itself for csrow2 and
192csrow3.
193
194The representation of the above is reflected in the directory tree
195in EDAC's sysfs interface. Starting in directory
196/sys/devices/system/edac/mc each memory controller will be represented
197by its own 'mcX' directory, where 'X" is the index of the MC.
198
199
200    ..../edac/mc/
201           |
202           |->mc0
203           |->mc1
204           |->mc2
205           ....
206
207Under each 'mcX' directory each 'csrowX' is again represented by a
208'csrowX', where 'X" is the csrow index:
209
210
211    .../mc/mc0/
212        |
213        |->csrow0
214        |->csrow2
215        |->csrow3
216        ....
217
218Notice that there is no csrow1, which indicates that csrow0 is
219composed of a single ranked DIMMs. This should also apply in both
220Channels, in order to have dual-channel mode be operational. Since
221both csrow2 and csrow3 are populated, this indicates a dual ranked
222set of DIMMs for channels 0 and 1.
223
224
225Within each of the 'mcX' and 'csrowX' directories are several
226EDAC control and attribute files.
227
228============================================================================
229'mcX' DIRECTORIES
230
231
232In 'mcX' directories are EDAC control and attribute files for
233this 'X" instance of the memory controllers:
234
235
236Counter reset control file:
237
238    'reset_counters'
239
240    This write-only control file will zero all the statistical counters
241    for UE and CE errors. Zeroing the counters will also reset the timer
242    indicating how long since the last counter zero. This is useful
243    for computing errors/time. Since the counters are always reset at
244    driver initialization time, no module/kernel parameter is available.
245
246    RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset
247
248        This resets the counters on memory controller 0
249
250
251Seconds since last counter reset control file:
252
253    'seconds_since_reset'
254
255    This attribute file displays how many seconds have elapsed since the
256    last counter reset. This can be used with the error counters to
257    measure error rates.
258
259
260
261Memory Controller name attribute file:
262
263    'mc_name'
264
265    This attribute file displays the type of memory controller
266    that is being utilized.
267
268
269Total memory managed by this memory controller attribute file:
270
271    'size_mb'
272
273    This attribute file displays, in count of megabytes, of memory
274    that this instance of memory controller manages.
275
276
277Total Uncorrectable Errors count attribute file:
278
279    'ue_count'
280
281    This attribute file displays the total count of uncorrectable
282    errors that have occurred on this memory controller. If panic_on_ue
283    is set this counter will not have a chance to increment,
284    since EDAC will panic the system.
285
286
287Total UE count that had no information attribute fileY:
288
289    'ue_noinfo_count'
290
291    This attribute file displays the number of UEs that have occurred
292    with no information as to which DIMM slot is having errors.
293
294
295Total Correctable Errors count attribute file:
296
297    'ce_count'
298
299    This attribute file displays the total count of correctable
300    errors that have occurred on this memory controller. This
301    count is very important to examine. CEs provide early
302    indications that a DIMM is beginning to fail. This count
303    field should be monitored for non-zero values and report
304    such information to the system administrator.
305
306
307Total Correctable Errors count attribute file:
308
309    'ce_noinfo_count'
310
311    This attribute file displays the number of CEs that
312    have occurred wherewith no informations as to which DIMM slot
313    is having errors. Memory is handicapped, but operational,
314    yet no information is available to indicate which slot
315    the failing memory is in. This count field should be also
316    be monitored for non-zero values.
317
318Device Symlink:
319
320    'device'
321
322    Symlink to the memory controller device.
323
324Sdram memory scrubbing rate:
325
326    'sdram_scrub_rate'
327
328    Read/Write attribute file that controls memory scrubbing. The scrubbing
329    rate is set by writing a minimum bandwidth in bytes/sec to the attribute
330    file. The rate will be translated to an internal value that gives at
331    least the specified rate.
332
333    Reading the file will return the actual scrubbing rate employed.
334
335    If configuration fails or memory scrubbing is not implemented, the value
336    of the attribute file will be -1.
337
338
339
340============================================================================
341'csrowX' DIRECTORIES
342
343In the 'csrowX' directories are EDAC control and attribute files for
344this 'X" instance of csrow:
345
346
347Total Uncorrectable Errors count attribute file:
348
349    'ue_count'
350
351    This attribute file displays the total count of uncorrectable
352    errors that have occurred on this csrow. If panic_on_ue is set
353    this counter will not have a chance to increment, since EDAC
354    will panic the system.
355
356
357Total Correctable Errors count attribute file:
358
359    'ce_count'
360
361    This attribute file displays the total count of correctable
362    errors that have occurred on this csrow. This
363    count is very important to examine. CEs provide early
364    indications that a DIMM is beginning to fail. This count
365    field should be monitored for non-zero values and report
366    such information to the system administrator.
367
368
369Total memory managed by this csrow attribute file:
370
371    'size_mb'
372
373    This attribute file displays, in count of megabytes, of memory
374    that this csrow contains.
375
376
377Memory Type attribute file:
378
379    'mem_type'
380
381    This attribute file will display what type of memory is currently
382    on this csrow. Normally, either buffered or unbuffered memory.
383    Examples:
384        Registered-DDR
385        Unbuffered-DDR
386
387
388EDAC Mode of operation attribute file:
389
390    'edac_mode'
391
392    This attribute file will display what type of Error detection
393    and correction is being utilized.
394
395
396Device type attribute file:
397
398    'dev_type'
399
400    This attribute file will display what type of DRAM device is
401    being utilized on this DIMM.
402    Examples:
403        x1
404        x2
405        x4
406        x8
407
408
409Channel 0 CE Count attribute file:
410
411    'ch0_ce_count'
412
413    This attribute file will display the count of CEs on this
414    DIMM located in channel 0.
415
416
417Channel 0 UE Count attribute file:
418
419    'ch0_ue_count'
420
421    This attribute file will display the count of UEs on this
422    DIMM located in channel 0.
423
424
425Channel 0 DIMM Label control file:
426
427    'ch0_dimm_label'
428
429    This control file allows this DIMM to have a label assigned
430    to it. With this label in the module, when errors occur
431    the output can provide the DIMM label in the system log.
432    This becomes vital for panic events to isolate the
433    cause of the UE event.
434
435    DIMM Labels must be assigned after booting, with information
436    that correctly identifies the physical slot with its
437    silk screen label. This information is currently very
438    motherboard specific and determination of this information
439    must occur in userland at this time.
440
441
442Channel 1 CE Count attribute file:
443
444    'ch1_ce_count'
445
446    This attribute file will display the count of CEs on this
447    DIMM located in channel 1.
448
449
450Channel 1 UE Count attribute file:
451
452    'ch1_ue_count'
453
454    This attribute file will display the count of UEs on this
455    DIMM located in channel 0.
456
457
458Channel 1 DIMM Label control file:
459
460    'ch1_dimm_label'
461
462    This control file allows this DIMM to have a label assigned
463    to it. With this label in the module, when errors occur
464    the output can provide the DIMM label in the system log.
465    This becomes vital for panic events to isolate the
466    cause of the UE event.
467
468    DIMM Labels must be assigned after booting, with information
469    that correctly identifies the physical slot with its
470    silk screen label. This information is currently very
471    motherboard specific and determination of this information
472    must occur in userland at this time.
473
474============================================================================
475SYSTEM LOGGING
476
477If logging for UEs and CEs are enabled then system logs will have
478error notices indicating errors that have been detected:
479
480EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
481channel 1 "DIMM_B1": amd76x_edac
482
483EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
484channel 1 "DIMM_B1": amd76x_edac
485
486
487The structure of the message is:
488    the memory controller (MC0)
489    Error type (CE)
490    memory page (0x283)
491    offset in the page (0xce0)
492    the byte granularity (grain 8)
493        or resolution of the error
494    the error syndrome (0xb741)
495    memory row (row 0)
496    memory channel (channel 1)
497    DIMM label, if set prior (DIMM B1
498    and then an optional, driver-specific message that may
499        have additional information.
500
501Both UEs and CEs with no info will lack all but memory controller,
502error type, a notice of "no info" and then an optional,
503driver-specific error message.
504
505
506============================================================================
507PCI Bus Parity Detection
508
509
510On Header Type 00 devices the primary status is looked at
511for any parity error regardless of whether Parity is enabled on the
512device. (The spec indicates parity is generated in some cases).
513On Header Type 01 bridges, the secondary status register is also
514looked at to see if parity occurred on the bus on the other side of
515the bridge.
516
517
518SYSFS CONFIGURATION
519
520Under /sys/devices/system/edac/pci are control and attribute files as follows:
521
522
523Enable/Disable PCI Parity checking control file:
524
525    'check_pci_parity'
526
527
528    This control file enables or disables the PCI Bus Parity scanning
529    operation. Writing a 1 to this file enables the scanning. Writing
530    a 0 to this file disables the scanning.
531
532    Enable:
533    echo "1" >/sys/devices/system/edac/pci/check_pci_parity
534
535    Disable:
536    echo "0" >/sys/devices/system/edac/pci/check_pci_parity
537
538
539Parity Count:
540
541    'pci_parity_count'
542
543    This attribute file will display the number of parity errors that
544    have been detected.
545
546
547============================================================================
548MODULE PARAMETERS
549
550Panic on UE control file:
551
552    'edac_mc_panic_on_ue'
553
554    An uncorrectable error will cause a machine panic. This is usually
555    desirable. It is a bad idea to continue when an uncorrectable error
556    occurs - it is indeterminate what was uncorrected and the operating
557    system context might be so mangled that continuing will lead to further
558    corruption. If the kernel has MCE configured, then EDAC will never
559    notice the UE.
560
561    LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1]
562
563    RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
564
565
566Log UE control file:
567
568    'edac_mc_log_ue'
569
570    Generate kernel messages describing uncorrectable errors. These errors
571    are reported through the system message log system. UE statistics
572    will be accumulated even when UE logging is disabled.
573
574    LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1]
575
576    RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
577
578
579Log CE control file:
580
581    'edac_mc_log_ce'
582
583    Generate kernel messages describing correctable errors. These
584    errors are reported through the system message log system.
585    CE statistics will be accumulated even when CE logging is disabled.
586
587    LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1]
588
589    RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
590
591
592Polling period control file:
593
594    'edac_mc_poll_msec'
595
596    The time period, in milliseconds, for polling for error information.
597    Too small a value wastes resources. Too large a value might delay
598    necessary handling of errors and might loose valuable information for
599    locating the error. 1000 milliseconds (once each second) is the current
600    default. Systems which require all the bandwidth they can get, may
601    increase this.
602
603    LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1]
604
605    RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
606
607
608Panic on PCI PARITY Error:
609
610    'panic_on_pci_parity'
611
612
613    This control files enables or disables panicking when a parity
614    error has been detected.
615
616
617    module/kernel parameter: edac_panic_on_pci_pe=[0|1]
618
619    Enable:
620    echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
621
622    Disable:
623    echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
624
625
626
627=======================================================================
628
629
630EDAC_DEVICE type of device
631
632In the header file, edac_core.h, there is a series of edac_device structures
633and APIs for the EDAC_DEVICE.
634
635User space access to an edac_device is through the sysfs interface.
636
637At the location /sys/devices/system/edac (sysfs) new edac_device devices will
638appear.
639
640There is a three level tree beneath the above 'edac' directory. For example,
641the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website)
642installs itself as:
643
644    /sys/devices/systm/edac/test-instance
645
646in this directory are various controls, a symlink and one or more 'instance'
647directorys.
648
649The standard default controls are:
650
651    log_ce boolean to log CE events
652    log_ue boolean to log UE events
653    panic_on_ue boolean to 'panic' the system if an UE is encountered
654            (default off, can be set true via startup script)
655    poll_msec time period between POLL cycles for events
656
657The test_device_edac device adds at least one of its own custom control:
658
659    test_bits which in the current test driver does nothing but
660            show how it is installed. A ported driver can
661            add one or more such controls and/or attributes
662            for specific uses.
663            One out-of-tree driver uses controls here to allow
664            for ERROR INJECTION operations to hardware
665            injection registers
666
667The symlink points to the 'struct dev' that is registered for this edac_device.
668
669INSTANCES
670
671One or more instance directories are present. For the 'test_device_edac' case:
672
673    test-instance0
674
675
676In this directory there are two default counter attributes, which are totals of
677counter in deeper subdirectories.
678
679    ce_count total of CE events of subdirectories
680    ue_count total of UE events of subdirectories
681
682BLOCKS
683
684At the lowest directory level is the 'block' directory. There can be 0, 1
685or more blocks specified in each instance.
686
687    test-block0
688
689
690In this directory the default attributes are:
691
692    ce_count which is counter of CE events for this 'block'
693            of hardware being monitored
694    ue_count which is counter of UE events for this 'block'
695            of hardware being monitored
696
697
698The 'test_device_edac' device adds 4 attributes and 1 control:
699
700    test-block-bits-0 for every POLL cycle this counter
701                is incremented
702    test-block-bits-1 every 10 cycles, this counter is bumped once,
703                and test-block-bits-0 is set to 0
704    test-block-bits-2 every 100 cycles, this counter is bumped once,
705                and test-block-bits-1 is set to 0
706    test-block-bits-3 every 1000 cycles, this counter is bumped once,
707                and test-block-bits-2 is set to 0
708
709
710    reset-counters writing ANY thing to this control will
711                reset all the above counters.
712
713
714Use of the 'test_device_edac' driver should any others to create their own
715unique drivers for their hardware systems.
716
717The 'test_device_edac' sample driver is located at the
718bluesmoke.sourceforge.net project site for EDAC.
719
720

Archive Download this file



interactive