1224 lines · plain
1.. SPDX-License-Identifier: GPL-2.02.. include:: <isonum.txt>3 4==================================================5Reliability, Availability and Serviceability (RAS)6==================================================7 8This documents different aspects of the RAS functionality present in the9kernel.10 11RAS concepts12************13 14Reliability, Availability and Serviceability (RAS) is a concept used on15servers meant to measure their robustness.16 17Reliability18 is the probability that a system will produce correct outputs.19 20 * Generally measured as Mean Time Between Failures (MTBF)21 * Enhanced by features that help to avoid, detect and repair hardware faults22 23Availability24 is the probability that a system is operational at a given time25 26 * Generally measured as a percentage of downtime per a period of time27 * Often uses mechanisms to detect and correct hardware faults in28 runtime;29 30Serviceability (or maintainability)31 is the simplicity and speed with which a system can be repaired or32 maintained33 34 * Generally measured on Mean Time Between Repair (MTBR)35 36Improving RAS37-------------38 39In order to reduce systems downtime, a system should be capable of detecting40hardware errors, and, when possible correcting them in runtime. It should41also provide mechanisms to detect hardware degradation, in order to warn42the system administrator to take the action of replacing a component before43it causes data loss or system downtime.44 45Among the monitoring measures, the most usual ones include:46 47* CPU – detect errors at instruction execution and at L1/L2/L3 caches;48* Memory – add error correction logic (ECC) to detect and correct errors;49* I/O – add CRC checksums for transferred data;50* Storage – RAID, journal file systems, checksums,51 Self-Monitoring, Analysis and Reporting Technology (SMART).52 53By monitoring the number of occurrences of error detections, it is possible54to identify if the probability of hardware errors is increasing, and, on such55case, do a preventive maintenance to replace a degraded component while56those errors are correctable.57 58Types of errors59---------------60 61Most mechanisms used on modern systems use technologies like Hamming62Codes that allow error correction when the number of errors on a bit packet63is below a threshold. If the number of errors is above, those mechanisms64can indicate with a high degree of confidence that an error happened, but65they can't correct.66 67Also, sometimes an error occur on a component that it is not used. For68example, a part of the memory that it is not currently allocated.69 70That defines some categories of errors:71 72* **Correctable Error (CE)** - the error detection mechanism detected and73 corrected the error. Such errors are usually not fatal, although some74 Kernel mechanisms allow the system administrator to consider them as fatal.75 76* **Uncorrected Error (UE)** - the amount of errors happened above the error77 correction threshold, and the system was unable to auto-correct.78 79* **Fatal Error** - when an UE error happens on a critical component of the80 system (for example, a piece of the Kernel got corrupted by an UE), the81 only reliable way to avoid data corruption is to hang or reboot the machine.82 83* **Non-fatal Error** - when an UE error happens on an unused component,84 like a CPU in power down state or an unused memory bank, the system may85 still run, eventually replacing the affected hardware by a hot spare,86 if available.87 88 Also, when an error happens on a userspace process, it is also possible to89 kill such process and let userspace restart it.90 91The mechanism for handling non-fatal errors is usually complex and may92require the help of some userspace application, in order to apply the93policy desired by the system administrator.94 95Identifying a bad hardware component96------------------------------------97 98Just detecting a hardware flaw is usually not enough, as the system needs99to pinpoint to the minimal replaceable unit (MRU) that should be exchanged100to make the hardware reliable again.101 102So, it requires not only error logging facilities, but also mechanisms that103will translate the error message to the silkscreen or component label for104the MRU.105 106Typically, it is very complex for memory, as modern CPUs interlace memory107from different memory modules, in order to provide a better performance. The108DMI BIOS usually have a list of memory module labels, with can be obtained109using the ``dmidecode`` tool. For example, on a desktop machine, it shows::110 111 Memory Device112 Total Width: 64 bits113 Data Width: 64 bits114 Size: 16384 MB115 Form Factor: SODIMM116 Set: None117 Locator: ChannelA-DIMM0118 Bank Locator: BANK 0119 Type: DDR4120 Type Detail: Synchronous121 Speed: 2133 MHz122 Rank: 2123 Configured Clock Speed: 2133 MHz124 125On the above example, a DDR4 SO-DIMM memory module is located at the126system's memory labeled as "BANK 0", as given by the *bank locator* field.127Please notice that, on such system, the *total width* is equal to the128*data width*. It means that such memory module doesn't have error129detection/correction mechanisms.130 131Unfortunately, not all systems use the same field to specify the memory132bank. On this example, from an older server, ``dmidecode`` shows::133 134 Memory Device135 Array Handle: 0x1000136 Error Information Handle: Not Provided137 Total Width: 72 bits138 Data Width: 64 bits139 Size: 8192 MB140 Form Factor: DIMM141 Set: 1142 Locator: DIMM_A1143 Bank Locator: Not Specified144 Type: DDR3145 Type Detail: Synchronous Registered (Buffered)146 Speed: 1600 MHz147 Rank: 2148 Configured Clock Speed: 1600 MHz149 150There, the DDR3 RDIMM memory module is located at the system's memory labeled151as "DIMM_A1", as given by the *locator* field. Please notice that this152memory module has 64 bits of *data width* and 72 bits of *total width*. So,153it has 8 extra bits to be used by error detection and correction mechanisms.154Such kind of memory is called Error-correcting code memory (ECC memory).155 156To make things even worse, it is not uncommon that systems with different157labels on their system's board to use exactly the same BIOS, meaning that158the labels provided by the BIOS won't match the real ones.159 160ECC memory161----------162 163As mentioned in the previous section, ECC memory has extra bits to be164used for error correction. In the above example, a memory module has16564 bits of *data width*, and 72 bits of *total width*. The extra 8166bits which are used for the error detection and correction mechanisms167are referred to as the *syndrome*\ [#f1]_\ [#f2]_.168 169So, when the cpu requests the memory controller to write a word with170*data width*, the memory controller calculates the *syndrome* in real time,171using Hamming code, or some other error correction code, like SECDED+,172producing a code with *total width* size. Such code is then written173on the memory modules.174 175At read, the *total width* bits code is converted back, using the same176ECC code used on write, producing a word with *data width* and a *syndrome*.177The word with *data width* is sent to the CPU, even when errors happen.178 179The memory controller also looks at the *syndrome* in order to check if180there was an error, and if the ECC code was able to fix such error.181If the error was corrected, a Corrected Error (CE) happened. If not, an182Uncorrected Error (UE) happened.183 184The information about the CE/UE errors is stored on some special registers185at the memory controller and can be accessed by reading such registers,186either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64187bit CPUs, such errors can also be retrieved via the Machine Check188Architecture (MCA)\ [#f3]_.189 190.. [#f1] Please notice that several memory controllers allow operation on a191 mode called "Lock-Step", where it groups two memory modules together,192 doing 128-bit reads/writes. That gives 16 bits for error correction, with193 significantly improves the error correction mechanism, at the expense194 that, when an error happens, there's no way to know what memory module is195 to blame. So, it has to blame both memory modules.196 197.. [#f2] Some memory controllers also allow using memory in mirror mode.198 On such mode, the same data is written to two memory modules. At read,199 the system checks both memory modules, in order to check if both provide200 identical data. On such configuration, when an error happens, there's no201 way to know what memory module is to blame. So, it has to blame both202 memory modules (or 4 memory modules, if the system is also on Lock-step203 mode).204 205.. [#f3] For more details about the Machine Check Architecture (MCA),206 please read Documentation/arch/x86/x86_64/machinecheck.rst at the Kernel tree.207 208EDAC - Error Detection And Correction209*************************************210 211.. note::212 213 "bluesmoke" was the name for this device driver subsystem when it214 was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net.215 That site is mostly archaic now and can be used only for historical216 purposes.217 218 When the subsystem was pushed upstream for the first time, on219 Kernel 2.6.16, it was renamed to ``EDAC``.220 221Purpose222-------223 224The ``edac`` kernel module's goal is to detect and report hardware errors225that occur within the computer system running under linux.226 227Memory228------229 230Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the231primary errors being harvested. These types of errors are harvested by232the ``edac_mc`` device.233 234Detecting CE events, then harvesting those events and reporting them,235**can** but must not necessarily be a predictor of future UE events. With236CE events only, the system can and will continue to operate as no data237has been damaged yet.238 239However, preventive maintenance and proactive part replacement of memory240modules exhibiting CEs can reduce the likelihood of the dreaded UE events241and system panics.242 243Other hardware elements244-----------------------245 246A new feature for EDAC, the ``edac_device`` class of device, was added in247the 2.6.23 version of the kernel.248 249This new device type allows for non-memory type of ECC hardware detectors250to have their states harvested and presented to userspace via the sysfs251interface.252 253Some architectures have ECC detectors for L1, L2 and L3 caches,254along with DMA engines, fabric switches, main data path switches,255interconnections, and various other hardware data paths. If the hardware256reports it, then a edac_device device probably can be constructed to257harvest and present that to userspace.258 259 260PCI bus scanning261----------------262 263In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors264in order to determine if errors are occurring during data transfers.265 266The presence of PCI Parity errors must be examined with a grain of salt.267There are several add-in adapters that do **not** follow the PCI specification268with regards to Parity generation and reporting. The specification says269the vendor should tie the parity status bits to 0 if they do not intend270to generate parity. Some vendors do not do this, and thus the parity bit271can "float" giving false positives.272 273There is a PCI device attribute located in sysfs that is checked by274the EDAC PCI scanning code. If that attribute is set, PCI parity/error275scanning is skipped for that device. The attribute is::276 277 broken_parity_status278 279and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for280PCI devices.281 282 283Versioning284----------285 286EDAC is composed of a "core" module (``edac_core.ko``) and several Memory287Controller (MC) driver modules. On a given system, the CORE is loaded288and one MC driver will be loaded. Both the CORE and the MC driver (or289``edac_device`` driver) have individual versions that reflect current290release level of their respective modules.291 292Thus, to "report" on what version a system is running, one must report293both the CORE's and the MC driver's versions.294 295 296Loading297-------298 299If ``edac`` was statically linked with the kernel then no loading300is necessary. If ``edac`` was built as modules then simply modprobe301the ``edac`` pieces that you need. You should be able to modprobe302hardware-specific modules and have the dependencies load the necessary303core modules.304 305Example::306 307 $ modprobe amd76x_edac308 309loads both the ``amd76x_edac.ko`` memory controller module and the310``edac_mc.ko`` core module.311 312 313Sysfs interface314---------------315 316EDAC presents a ``sysfs`` interface for control and reporting purposes. It317lives in the /sys/devices/system/edac directory.318 319Within this directory there currently reside 2 components:320 321 ======= ==============================322 mc memory controller(s) system323 pci PCI control and status system324 ======= ==============================325 326 327 328Memory Controller (mc) Model329----------------------------330 331Each ``mc`` device controls a set of memory modules [#f4]_. These modules332are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``).333There can be multiple csrows and multiple channels.334 335.. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely336 used to refer to a memory module, although there are other memory337 packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI338 specification (Version 2.7) defines a memory module in the Common339 Platform Error Record (CPER) section to be an SMBIOS Memory Device340 (Type 17). Along this document, and inside the EDAC subsystem, the term341 "dimm" is used for all memory modules, even when they use a342 different kind of packaging.343 344Memory controllers allow for several csrows, with 8 csrows being a345typical value. Yet, the actual number of csrows depends on the layout of346a given motherboard, memory controller and memory module characteristics.347 348Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems)349data transfers to/from the CPU from/to memory. Some newer chipsets allow350for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory351controllers. The following example will assume 2 channels:352 353 +------------+-----------------------+354 | CS Rows | Channels |355 +------------+-----------+-----------+356 | | ``ch0`` | ``ch1`` |357 +============+===========+===========+358 | |**DIMM_A0**|**DIMM_B0**|359 +------------+-----------+-----------+360 | ``csrow0`` | rank0 | rank0 |361 +------------+-----------+-----------+362 | ``csrow1`` | rank1 | rank1 |363 +------------+-----------+-----------+364 | |**DIMM_A1**|**DIMM_B1**|365 +------------+-----------+-----------+366 | ``csrow2`` | rank0 | rank0 |367 +------------+-----------+-----------+368 | ``csrow3`` | rank1 | rank1 |369 +------------+-----------+-----------+370 371In the above example, there are 4 physical slots on the motherboard372for memory DIMMs:373 374 +---------+---------+375 | DIMM_A0 | DIMM_B0 |376 +---------+---------+377 | DIMM_A1 | DIMM_B1 |378 +---------+---------+379 380Labels for these slots are usually silk-screened on the motherboard.381Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are382channel 1. Notice that there are two csrows possible on a physical DIMM.383These csrows are allocated their csrow assignment based on the slot into384which the memory DIMM is placed. Thus, when 1 DIMM is placed in each385Channel, the csrows cross both DIMMs.386 387Memory DIMMs come single or dual "ranked". A rank is a populated csrow.388In the example above 2 dual ranked DIMMs are similarly placed. Thus,389both csrow0 and csrow1 are populated. On the other hand, when 2 single390ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will391have just one csrow (csrow0) and csrow1 will be empty. The pattern392repeats itself for csrow2 and csrow3. Also note that some memory393controllers don't have any logic to identify the memory module, see394``rankX`` directories below.395 396The representation of the above is reflected in the directory397tree in EDAC's sysfs interface. Starting in directory398``/sys/devices/system/edac/mc``, each memory controller will be399represented by its own ``mcX`` directory, where ``X`` is the400index of the MC::401 402 ..../edac/mc/403 |404 |->mc0405 |->mc1406 |->mc2407 ....408 409Under each ``mcX`` directory each ``csrowX`` is again represented by a410``csrowX``, where ``X`` is the csrow index::411 412 .../mc/mc0/413 |414 |->csrow0415 |->csrow2416 |->csrow3417 ....418 419Notice that there is no csrow1, which indicates that csrow0 is composed420of a single ranked DIMMs. This should also apply in both Channels, in421order to have dual-channel mode be operational. Since both csrow2 and422csrow3 are populated, this indicates a dual ranked set of DIMMs for423channels 0 and 1.424 425Within each of the ``mcX`` and ``csrowX`` directories are several EDAC426control and attribute files.427 428``mcX`` directories429-------------------430 431In ``mcX`` directories are EDAC control and attribute files for432this ``X`` instance of the memory controllers.433 434For a description of the sysfs API, please see:435 436 Documentation/ABI/testing/sysfs-devices-edac437 438 439``dimmX`` or ``rankX`` directories440----------------------------------441 442The recommended way to use the EDAC subsystem is to look at the information443provided by the ``dimmX`` or ``rankX`` directories [#f5]_.444 445A typical EDAC system has the following structure under446``/sys/devices/system/edac/``\ [#f6]_::447 448 /sys/devices/system/edac/449 ├── mc450 │ ├── mc0451 │ │ ├── ce_count452 │ │ ├── ce_noinfo_count453 │ │ ├── dimm0454 │ │ │ ├── dimm_ce_count455 │ │ │ ├── dimm_dev_type456 │ │ │ ├── dimm_edac_mode457 │ │ │ ├── dimm_label458 │ │ │ ├── dimm_location459 │ │ │ ├── dimm_mem_type460 │ │ │ ├── dimm_ue_count461 │ │ │ ├── size462 │ │ │ └── uevent463 │ │ ├── max_location464 │ │ ├── mc_name465 │ │ ├── reset_counters466 │ │ ├── seconds_since_reset467 │ │ ├── size_mb468 │ │ ├── ue_count469 │ │ ├── ue_noinfo_count470 │ │ └── uevent471 │ ├── mc1472 │ │ ├── ce_count473 │ │ ├── ce_noinfo_count474 │ │ ├── dimm0475 │ │ │ ├── dimm_ce_count476 │ │ │ ├── dimm_dev_type477 │ │ │ ├── dimm_edac_mode478 │ │ │ ├── dimm_label479 │ │ │ ├── dimm_location480 │ │ │ ├── dimm_mem_type481 │ │ │ ├── dimm_ue_count482 │ │ │ ├── size483 │ │ │ └── uevent484 │ │ ├── max_location485 │ │ ├── mc_name486 │ │ ├── reset_counters487 │ │ ├── seconds_since_reset488 │ │ ├── size_mb489 │ │ ├── ue_count490 │ │ ├── ue_noinfo_count491 │ │ └── uevent492 │ └── uevent493 └── uevent494 495In the ``dimmX`` directories are EDAC control and attribute files for496this ``X`` memory module:497 498- ``size`` - Total memory managed by this csrow attribute file499 500 This attribute file displays, in count of megabytes, the memory501 that this csrow contains.502 503- ``dimm_ue_count`` - Uncorrectable Errors count attribute file504 505 This attribute file displays the total count of uncorrectable506 errors that have occurred on this DIMM. If panic_on_ue is set507 this counter will not have a chance to increment, since EDAC508 will panic the system.509 510- ``dimm_ce_count`` - Correctable Errors count attribute file511 512 This attribute file displays the total count of correctable513 errors that have occurred on this DIMM. This count is very514 important to examine. CEs provide early indications that a515 DIMM is beginning to fail. This count field should be516 monitored for non-zero values and report such information517 to the system administrator.518 519- ``dimm_dev_type`` - Device type attribute file520 521 This attribute file will display what type of DRAM device is522 being utilized on this DIMM.523 Examples:524 525 - x1526 - x2527 - x4528 - x8529 530- ``dimm_edac_mode`` - EDAC Mode of operation attribute file531 532 This attribute file will display what type of Error detection533 and correction is being utilized.534 535- ``dimm_label`` - memory module label control file536 537 This control file allows this DIMM to have a label assigned538 to it. With this label in the module, when errors occur539 the output can provide the DIMM label in the system log.540 This becomes vital for panic events to isolate the541 cause of the UE event.542 543 DIMM Labels must be assigned after booting, with information544 that correctly identifies the physical slot with its545 silk screen label. This information is currently very546 motherboard specific and determination of this information547 must occur in userland at this time.548 549- ``dimm_location`` - location of the memory module550 551 The location can have up to 3 levels, and describe how the552 memory controller identifies the location of a memory module.553 Depending on the type of memory and memory controller, it554 can be:555 556 - *csrow* and *channel* - used when the memory controller557 doesn't identify a single DIMM - e. g. in ``rankX`` dir;558 - *branch*, *channel*, *slot* - typically used on FB-DIMM memory559 controllers;560 - *channel*, *slot* - used on Nehalem and newer Intel drivers.561 562- ``dimm_mem_type`` - Memory Type attribute file563 564 This attribute file will display what type of memory is currently565 on this csrow. Normally, either buffered or unbuffered memory.566 Examples:567 568 - Registered-DDR569 - Unbuffered-DDR570 571.. [#f5] On some systems, the memory controller doesn't have any logic572 to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories.573 On modern Intel memory controllers, the memory controller identifies the574 memory modules directly. On such systems, the directory is called ``dimmX``.575 576.. [#f6] There are also some ``power`` directories and ``subsystem``577 symlinks inside the sysfs mapping that are automatically created by578 the sysfs subsystem. Currently, they serve no purpose.579 580``csrowX`` directories581----------------------582 583When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the ``csrowX``584directories. As this API doesn't work properly for Rambus, FB-DIMMs and585modern Intel Memory Controllers, this is being deprecated in favor of586``dimmX`` directories.587 588In the ``csrowX`` directories are EDAC control and attribute files for589this ``X`` instance of csrow:590 591 592- ``ue_count`` - Total Uncorrectable Errors count attribute file593 594 This attribute file displays the total count of uncorrectable595 errors that have occurred on this csrow. If panic_on_ue is set596 this counter will not have a chance to increment, since EDAC597 will panic the system.598 599 600- ``ce_count`` - Total Correctable Errors count attribute file601 602 This attribute file displays the total count of correctable603 errors that have occurred on this csrow. This count is very604 important to examine. CEs provide early indications that a605 DIMM is beginning to fail. This count field should be606 monitored for non-zero values and report such information607 to the system administrator.608 609 610- ``size_mb`` - Total memory managed by this csrow attribute file611 612 This attribute file displays, in count of megabytes, the memory613 that this csrow contains.614 615 616- ``mem_type`` - Memory Type attribute file617 618 This attribute file will display what type of memory is currently619 on this csrow. Normally, either buffered or unbuffered memory.620 Examples:621 622 - Registered-DDR623 - Unbuffered-DDR624 625 626- ``edac_mode`` - EDAC Mode of operation attribute file627 628 This attribute file will display what type of Error detection629 and correction is being utilized.630 631 632- ``dev_type`` - Device type attribute file633 634 This attribute file will display what type of DRAM device is635 being utilized on this DIMM.636 Examples:637 638 - x1639 - x2640 - x4641 - x8642 643 644- ``ch0_ce_count`` - Channel 0 CE Count attribute file645 646 This attribute file will display the count of CEs on this647 DIMM located in channel 0.648 649 650- ``ch0_ue_count`` - Channel 0 UE Count attribute file651 652 This attribute file will display the count of UEs on this653 DIMM located in channel 0.654 655 656- ``ch0_dimm_label`` - Channel 0 DIMM Label control file657 658 659 This control file allows this DIMM to have a label assigned660 to it. With this label in the module, when errors occur661 the output can provide the DIMM label in the system log.662 This becomes vital for panic events to isolate the663 cause of the UE event.664 665 DIMM Labels must be assigned after booting, with information666 that correctly identifies the physical slot with its667 silk screen label. This information is currently very668 motherboard specific and determination of this information669 must occur in userland at this time.670 671 672- ``ch1_ce_count`` - Channel 1 CE Count attribute file673 674 675 This attribute file will display the count of CEs on this676 DIMM located in channel 1.677 678 679- ``ch1_ue_count`` - Channel 1 UE Count attribute file680 681 682 This attribute file will display the count of UEs on this683 DIMM located in channel 0.684 685 686- ``ch1_dimm_label`` - Channel 1 DIMM Label control file687 688 This control file allows this DIMM to have a label assigned689 to it. With this label in the module, when errors occur690 the output can provide the DIMM label in the system log.691 This becomes vital for panic events to isolate the692 cause of the UE event.693 694 DIMM Labels must be assigned after booting, with information695 that correctly identifies the physical slot with its696 silk screen label. This information is currently very697 motherboard specific and determination of this information698 must occur in userland at this time.699 700 701System Logging702--------------703 704If logging for UEs and CEs is enabled, then system logs will contain705information indicating that errors have been detected::706 707 EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac708 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac709 710 711The structure of the message is:712 713 +---------------------------------------+-------------+714 | Content | Example |715 +=======================================+=============+716 | The memory controller | MC0 |717 +---------------------------------------+-------------+718 | Error type | CE |719 +---------------------------------------+-------------+720 | Memory page | 0x283 |721 +---------------------------------------+-------------+722 | Offset in the page | 0xce0 |723 +---------------------------------------+-------------+724 | The byte granularity | grain 8 |725 | or resolution of the error | |726 +---------------------------------------+-------------+727 | The error syndrome | 0xb741 |728 +---------------------------------------+-------------+729 | Memory row | row 0 |730 +---------------------------------------+-------------+731 | Memory channel | channel 1 |732 +---------------------------------------+-------------+733 | DIMM label, if set prior | DIMM B1 |734 +---------------------------------------+-------------+735 | And then an optional, driver-specific | |736 | message that may have additional | |737 | information. | |738 +---------------------------------------+-------------+739 740Both UEs and CEs with no info will lack all but memory controller, error741type, a notice of "no info" and then an optional, driver-specific error742message.743 744 745PCI Bus Parity Detection746------------------------747 748On Header Type 00 devices, the primary status is looked at for any749parity error regardless of whether parity is enabled on the device or750not. (The spec indicates parity is generated in some cases). On Header751Type 01 bridges, the secondary status register is also looked at to see752if parity occurred on the bus on the other side of the bridge.753 754 755Sysfs configuration756-------------------757 758Under ``/sys/devices/system/edac/pci`` are control and attribute files as759follows:760 761 762- ``check_pci_parity`` - Enable/Disable PCI Parity checking control file763 764 This control file enables or disables the PCI Bus Parity scanning765 operation. Writing a 1 to this file enables the scanning. Writing766 a 0 to this file disables the scanning.767 768 Enable::769 770 echo "1" >/sys/devices/system/edac/pci/check_pci_parity771 772 Disable::773 774 echo "0" >/sys/devices/system/edac/pci/check_pci_parity775 776 777- ``pci_parity_count`` - Parity Count778 779 This attribute file will display the number of parity errors that780 have been detected.781 782 783Module parameters784-----------------785 786- ``edac_mc_panic_on_ue`` - Panic on UE control file787 788 An uncorrectable error will cause a machine panic. This is usually789 desirable. It is a bad idea to continue when an uncorrectable error790 occurs - it is indeterminate what was uncorrected and the operating791 system context might be so mangled that continuing will lead to further792 corruption. If the kernel has MCE configured, then EDAC will never793 notice the UE.794 795 LOAD TIME::796 797 module/kernel parameter: edac_mc_panic_on_ue=[0|1]798 799 RUN TIME::800 801 echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue802 803 804- ``edac_mc_log_ue`` - Log UE control file805 806 807 Generate kernel messages describing uncorrectable errors. These errors808 are reported through the system message log system. UE statistics809 will be accumulated even when UE logging is disabled.810 811 LOAD TIME::812 813 module/kernel parameter: edac_mc_log_ue=[0|1]814 815 RUN TIME::816 817 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue818 819 820- ``edac_mc_log_ce`` - Log CE control file821 822 823 Generate kernel messages describing correctable errors. These824 errors are reported through the system message log system.825 CE statistics will be accumulated even when CE logging is disabled.826 827 LOAD TIME::828 829 module/kernel parameter: edac_mc_log_ce=[0|1]830 831 RUN TIME::832 833 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce834 835 836- ``edac_mc_poll_msec`` - Polling period control file837 838 839 The time period, in milliseconds, for polling for error information.840 Too small a value wastes resources. Too large a value might delay841 necessary handling of errors and might loose valuable information for842 locating the error. 1000 milliseconds (once each second) is the current843 default. Systems which require all the bandwidth they can get, may844 increase this.845 846 LOAD TIME::847 848 module/kernel parameter: edac_mc_poll_msec=[0|1]849 850 RUN TIME::851 852 echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec853 854 855- ``panic_on_pci_parity`` - Panic on PCI PARITY Error856 857 858 This control file enables or disables panicking when a parity859 error has been detected.860 861 862 module/kernel parameter::863 864 edac_panic_on_pci_pe=[0|1]865 866 Enable::867 868 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe869 870 Disable::871 872 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe873 874 875 876EDAC device type877----------------878 879In the header file, edac_pci.h, there is a series of edac_device structures880and APIs for the EDAC_DEVICE.881 882User space access to an edac_device is through the sysfs interface.883 884At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices885will appear.886 887There is a three level tree beneath the above ``edac`` directory. For example,888the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net889website) installs itself as::890 891 /sys/devices/system/edac/test-instance892 893in this directory are various controls, a symlink and one or more ``instance``894directories.895 896The standard default controls are:897 898 ============== =======================================================899 log_ce boolean to log CE events900 log_ue boolean to log UE events901 panic_on_ue boolean to ``panic`` the system if an UE is encountered902 (default off, can be set true via startup script)903 poll_msec time period between POLL cycles for events904 ============== =======================================================905 906The test_device_edac device adds at least one of its own custom control:907 908 ============== ==================================================909 test_bits which in the current test driver does nothing but910 show how it is installed. A ported driver can911 add one or more such controls and/or attributes912 for specific uses.913 One out-of-tree driver uses controls here to allow914 for ERROR INJECTION operations to hardware915 injection registers916 ============== ==================================================917 918The symlink points to the 'struct dev' that is registered for this edac_device.919 920Instances921---------922 923One or more instance directories are present. For the ``test_device_edac``924case:925 926 +----------------+927 | test-instance0 |928 +----------------+929 930 931In this directory there are two default counter attributes, which are totals of932counter in deeper subdirectories.933 934 ============== ====================================935 ce_count total of CE events of subdirectories936 ue_count total of UE events of subdirectories937 ============== ====================================938 939Blocks940------941 942At the lowest directory level is the ``block`` directory. There can be 0, 1943or more blocks specified in each instance:944 945 +-------------+946 | test-block0 |947 +-------------+948 949In this directory the default attributes are:950 951 ============== ================================================952 ce_count which is counter of CE events for this ``block``953 of hardware being monitored954 ue_count which is counter of UE events for this ``block``955 of hardware being monitored956 ============== ================================================957 958 959The ``test_device_edac`` device adds 4 attributes and 1 control:960 961 ================== ====================================================962 test-block-bits-0 for every POLL cycle this counter963 is incremented964 test-block-bits-1 every 10 cycles, this counter is bumped once,965 and test-block-bits-0 is set to 0966 test-block-bits-2 every 100 cycles, this counter is bumped once,967 and test-block-bits-1 is set to 0968 test-block-bits-3 every 1000 cycles, this counter is bumped once,969 and test-block-bits-2 is set to 0970 ================== ====================================================971 972 973 ================== ====================================================974 reset-counters writing ANY thing to this control will975 reset all the above counters.976 ================== ====================================================977 978 979Use of the ``test_device_edac`` driver should enable any others to create their own980unique drivers for their hardware systems.981 982The ``test_device_edac`` sample driver is located at the983http://bluesmoke.sourceforge.net project site for EDAC.984 985 986Usage of EDAC APIs on Nehalem and newer Intel CPUs987--------------------------------------------------988 989On older Intel architectures, the memory controller was part of the North990Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and991newer Intel architectures integrated an enhanced version of the memory992controller (MC) inside the CPUs.993 994This chapter will cover the differences of the enhanced memory controllers995found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and996``sbx_edac`` drivers.997 998.. note::999 1000 The Xeon E7 processor families use a separate chip for the memory1001 controller, called Intel Scalable Memory Buffer. This section doesn't1002 apply for such families.1003 10041) There is one Memory Controller per Quick Patch Interconnect1005 (QPI). At the driver, the term "socket" means one QPI. This is1006 associated with a physical CPU socket.1007 1008 Each MC have 3 physical read channels, 3 physical write channels and1009 3 logic channels. The driver currently sees it as just 3 channels.1010 Each channel can have up to 3 DIMMs.1011 1012 The minimum known unity is DIMMs. There are no information about csrows.1013 As EDAC API maps the minimum unity is csrows, the driver sequentially1014 maps channel/DIMM into different csrows.1015 1016 For example, supposing the following layout::1017 1018 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs1019 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x4001020 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x4001021 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs1022 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x4001023 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs1024 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x4001025 1026 The driver will map it as::1027 1028 csrow0: channel 0, dimm01029 csrow1: channel 0, dimm11030 csrow2: channel 1, dimm01031 csrow3: channel 2, dimm01032 1033 exports one DIMM per csrow.1034 1035 Each QPI is exported as a different memory controller.1036 10372) The MC has the ability to inject errors to test drivers. The drivers1038 implement this functionality via some error injection nodes:1039 1040 For injecting a memory error, there are some sysfs nodes, under1041 ``/sys/devices/system/edac/mc/mc?/``:1042 1043 - ``inject_addrmatch/*``:1044 Controls the error injection mask register. It is possible to specify1045 several characteristics of the address to match an error code::1046 1047 dimm = the affected dimm. Numbers are relative to a channel;1048 rank = the memory rank;1049 channel = the channel that will generate an error;1050 bank = the affected bank;1051 page = the page address;1052 column (or col) = the address column.1053 1054 each of the above values can be set to "any" to match any valid value.1055 1056 At driver init, all values are set to any.1057 1058 For example, to generate an error at rank 1 of dimm 2, for any channel,1059 any bank, any page, any column::1060 1061 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm1062 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank1063 1064 To return to the default behaviour of matching any, you can do::1065 1066 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm1067 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank1068 1069 - ``inject_eccmask``:1070 specifies what bits will have troubles,1071 1072 - ``inject_section``:1073 specifies what ECC cache section will get the error::1074 1075 3 for both1076 2 for the highest1077 1 for the lowest1078 1079 - ``inject_type``:1080 specifies the type of error, being a combination of the following bits::1081 1082 bit 0 - repeat1083 bit 1 - ecc1084 bit 2 - parity1085 1086 - ``inject_enable``:1087 starts the error generation when something different than 0 is written.1088 1089 All inject vars can be read. root permission is needed for write.1090 1091 Datasheet states that the error will only be generated after a write on an1092 address that matches inject_addrmatch. It seems, however, that reading will1093 also produce an error.1094 1095 For example, the following code will generate an error for any write access1096 at socket 0, on any DIMM/address on channel 2::1097 1098 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel1099 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type1100 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask1101 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section1102 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable1103 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null1104 1105 For socket 1, it is needed to replace "mc0" by "mc1" at the above1106 commands.1107 1108 The generated error message will look like::1109 1110 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))1111 11123) Corrected Error memory register counters1113 1114 Those newer MCs have some registers to count memory errors. The driver1115 uses those registers to report Corrected Errors on devices with Registered1116 DIMMs.1117 1118 However, those counters don't work with Unregistered DIMM. As the chipset1119 offers some counters that also work with UDIMMs (but with a worse level of1120 granularity than the default ones), the driver exposes those registers for1121 UDIMM memories.1122 1123 They can be read by looking at the contents of ``all_channel_counts/``::1124 1125 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done1126 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm01127 01128 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm11129 01130 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm21131 01132 1133 What happens here is that errors on different csrows, but at the same1134 dimm number will increment the same counter.1135 So, in this memory mapping::1136 1137 csrow0: channel 0, dimm01138 csrow1: channel 0, dimm11139 csrow2: channel 1, dimm01140 csrow3: channel 2, dimm01141 1142 The hardware will increment udimm0 for an error at the first dimm at either1143 csrow0, csrow2 or csrow3;1144 1145 The hardware will increment udimm1 for an error at the second dimm at either1146 csrow0, csrow2 or csrow3;1147 1148 The hardware will increment udimm2 for an error at the third dimm at either1149 csrow0, csrow2 or csrow3;1150 11514) Standard error counters1152 1153 The standard error counters are generated when an mcelog error is received1154 by the driver. Since, with UDIMM, this is counted by software, it is1155 possible that some errors could be lost. With RDIMM's, they display the1156 contents of the registers1157 1158Reference documents used on ``amd64_edac``1159------------------------------------------1160 1161``amd64_edac`` module is based on the following documents1162(available from http://support.amd.com/en-us/search/tech-docs):1163 11641. :Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD1165 Opteron Processors1166 :AMD publication #: 260941167 :Revision: 3.261168 :Link: http://support.amd.com/TechDocs/26094.PDF1169 11702. :Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh1171 Processors1172 :AMD publication #: 325591173 :Revision: 3.001174 :Issue Date: May 20061175 :Link: http://support.amd.com/TechDocs/32559.pdf1176 11773. :Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h1178 Processors1179 :AMD publication #: 311161180 :Revision: 3.001181 :Issue Date: September 07, 20071182 :Link: http://support.amd.com/TechDocs/31116.pdf1183 11844. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h1185 Models 30h-3Fh Processors1186 :AMD publication #: 491251187 :Revision: 3.061188 :Issue Date: 2/12/2015 (latest release)1189 :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf1190 11915. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h1192 Models 60h-6Fh Processors1193 :AMD publication #: 507421194 :Revision: 3.011195 :Issue Date: 7/23/2015 (latest release)1196 :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf1197 11986. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h1199 Models 00h-0Fh Processors1200 :AMD publication #: 487511201 :Revision: 3.031202 :Issue Date: 2/23/2015 (latest release)1203 :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf1204 1205Credits1206=======1207 1208* Written by Doug Thompson <dougthompson@xmission.com>1209 1210 - 7 Dec 20051211 - 17 Jul 2007 Updated1212 1213* |copy| Mauro Carvalho Chehab1214 1215 - 05 Aug 2009 Nehalem interface1216 - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section1217 1218* EDAC authors/maintainers:1219 1220 - Doug Thompson, Dave Jiang, Dave Peterson et al,1221 - Mauro Carvalho Chehab1222 - Borislav Petkov1223 - original author: Thayne Harbaugh1224