299 lines · plain
1Error Detection And Correction (EDAC) Devices2=============================================3 4Main Concepts used at the EDAC subsystem5----------------------------------------6 7There are several things to be aware of that aren't at all obvious, like8*sockets, *socket sets*, *banks*, *rows*, *chip-select rows*, *channels*,9etc...10 11These are some of the many terms that are thrown about that don't always12mean what people think they mean (Inconceivable!). In the interest of13creating a common ground for discussion, terms and their definitions14will be established.15 16* Memory devices17 18The individual DRAM chips on a memory stick. These devices commonly19output 4 and 8 bits each (x4, x8). Grouping several of these in parallel20provides the number of bits that the memory controller expects:21typically 72 bits, in order to provide 64 bits + 8 bits of ECC data.22 23* Memory Stick24 25A printed circuit board that aggregates multiple memory devices in26parallel. In general, this is the Field Replaceable Unit (FRU) which27gets replaced, in the case of excessive errors. Most often it is also28called DIMM (Dual Inline Memory Module).29 30* Memory Socket31 32A physical connector on the motherboard that accepts a single memory33stick. Also called as "slot" on several datasheets.34 35* Channel36 37A memory controller channel, responsible to communicate with a group of38DIMMs. Each channel has its own independent control (command) and data39bus, and can be used independently or grouped with other channels.40 41* Branch42 43It is typically the highest hierarchy on a Fully-Buffered DIMM memory44controller. Typically, it contains two channels. Two channels at the45same branch can be used in single mode or in lockstep mode. When46lockstep is enabled, the cacheline is doubled, but it generally brings47some performance penalty. Also, it is generally not possible to point to48just one memory stick when an error occurs, as the error correction code49is calculated using two DIMMs instead of one. Due to that, it is capable50of correcting more errors than on single mode.51 52* Single-channel53 54The data accessed by the memory controller is contained into one dimm55only. E. g. if the data is 64 bits-wide, the data flows to the CPU using56one 64 bits parallel access. Typically used with SDR, DDR, DDR2 and DDR357memories. FB-DIMM and RAMBUS use a different concept for channel, so58this concept doesn't apply there.59 60* Double-channel61 62The data size accessed by the memory controller is interlaced into two63dimms, accessed at the same time. E. g. if the DIMM is 64 bits-wide (7264bits with ECC), the data flows to the CPU using a 128 bits parallel65access.66 67* Chip-select row68 69This is the name of the DRAM signal used to select the DRAM ranks to be70accessed. Common chip-select rows for single channel are 64 bits, for71dual channel 128 bits. It may not be visible by the memory controller,72as some DIMM types have a memory buffer that can hide direct access to73it from the Memory Controller.74 75* Single-Ranked stick76 77A Single-ranked stick has 1 chip-select row of memory. Motherboards78commonly drive two chip-select pins to a memory stick. A single-ranked79stick, will occupy only one of those rows. The other will be unused.80 81.. _doubleranked:82 83* Double-Ranked stick84 85A double-ranked stick has two chip-select rows which access different86sets of memory devices. The two rows cannot be accessed concurrently.87 88* Double-sided stick89 90**DEPRECATED TERM**, see :ref:`Double-Ranked stick <doubleranked>`.91 92A double-sided stick has two chip-select rows which access different sets93of memory devices. The two rows cannot be accessed concurrently.94"Double-sided" is irrespective of the memory devices being mounted on95both sides of the memory stick.96 97* Socket set98 99All of the memory sticks that are required for a single memory access or100all of the memory sticks spanned by a chip-select row. A single socket101set has two chip-select rows and if double-sided sticks are used these102will occupy those chip-select rows.103 104* Bank105 106This term is avoided because it is unclear when needing to distinguish107between chip-select rows and socket sets.108 109* High Bandwidth Memory (HBM)110 111HBM is a new memory type with low power consumption and ultra-wide112communication lanes. It uses vertically stacked memory chips (DRAM dies)113interconnected by microscopic wires called "through-silicon vias," or114TSVs.115 116Several stacks of HBM chips connect to the CPU or GPU through an ultra-fast117interconnect called the "interposer". Therefore, HBM's characteristics118are nearly indistinguishable from on-chip integrated RAM.119 120Memory Controllers121------------------122 123Most of the EDAC core is focused on doing Memory Controller error detection.124The :c:func:`edac_mc_alloc`. It uses internally the struct ``mem_ctl_info``125to describe the memory controllers, with is an opaque struct for the EDAC126drivers. Only the EDAC core is allowed to touch it.127 128.. kernel-doc:: include/linux/edac.h129 130.. kernel-doc:: drivers/edac/edac_mc.h131 132PCI Controllers133---------------134 135The EDAC subsystem provides a mechanism to handle PCI controllers by calling136the :c:func:`edac_pci_alloc_ctl_info`. It will use the struct137:c:type:`edac_pci_ctl_info` to describe the PCI controllers.138 139.. kernel-doc:: drivers/edac/edac_pci.h140 141EDAC Blocks142-----------143 144The EDAC subsystem also provides a generic mechanism to report errors on145other parts of the hardware via :c:func:`edac_device_alloc_ctl_info` function.146 147The structures :c:type:`edac_dev_sysfs_block_attribute`,148:c:type:`edac_device_block`, :c:type:`edac_device_instance` and149:c:type:`edac_device_ctl_info` provide a generic or abstract 'edac_device'150representation at sysfs.151 152This set of structures and the code that implements the APIs for the same, provide for registering EDAC type devices which are NOT standard memory or153PCI, like:154 155- CPU caches (L1 and L2)156- DMA engines157- Core CPU switches158- Fabric switch units159- PCIe interface controllers160- other EDAC/ECC type devices that can be monitored for161 errors, etc.162 163It allows for a 2 level set of hierarchy.164 165For example, a cache could be composed of L1, L2 and L3 levels of cache.166Each CPU core would have its own L1 cache, while sharing L2 and maybe L3167caches. On such case, those can be represented via the following sysfs168nodes::169 170 /sys/devices/system/edac/..171 172 pci/ <existing pci directory (if available)>173 mc/ <existing memory device directory>174 cpu/cpu0/.. <L1 and L2 block directory>175 /L1-cache/ce_count176 /ue_count177 /L2-cache/ce_count178 /ue_count179 cpu/cpu1/.. <L1 and L2 block directory>180 /L1-cache/ce_count181 /ue_count182 /L2-cache/ce_count183 /ue_count184 ...185 186 the L1 and L2 directories would be "edac_device_block's"187 188.. kernel-doc:: drivers/edac/edac_device.h189 190 191Heterogeneous system support192----------------------------193 194An AMD heterogeneous system is built by connecting the data fabrics of195both CPUs and GPUs via custom xGMI links. Thus, the data fabric on the196GPU nodes can be accessed the same way as the data fabric on CPU nodes.197 198The MI200 accelerators are data center GPUs. They have 2 data fabrics,199and each GPU data fabric contains four Unified Memory Controllers (UMC).200Each UMC contains eight channels. Each UMC channel controls one 128-bit201HBM2e (2GB) channel (equivalent to 8 X 2GB ranks). This creates a total202of 4096-bits of DRAM data bus.203 204While the UMC is interfacing a 16GB (8high X 2GB DRAM) HBM stack, each UMC205channel is interfacing 2GB of DRAM (represented as rank).206 207Memory controllers on AMD GPU nodes can be represented in EDAC thusly:208 209 GPU DF / GPU Node -> EDAC MC210 GPU UMC -> EDAC CSROW211 GPU UMC channel -> EDAC CHANNEL212 213For example: a heterogeneous system with 1 AMD CPU is connected to2144 MI200 (Aldebaran) GPUs using xGMI.215 216Some more heterogeneous hardware details:217 218- The CPU UMC (Unified Memory Controller) is mostly the same as the GPU UMC.219 They have chip selects (csrows) and channels. However, the layouts are different220 for performance, physical layout, or other reasons.221- CPU UMCs use 1 channel, In this case UMC = EDAC channel. This follows the222 marketing speak. CPU has X memory channels, etc.223- CPU UMCs use up to 4 chip selects, So UMC chip select = EDAC CSROW.224- GPU UMCs use 1 chip select, So UMC = EDAC CSROW.225- GPU UMCs use 8 channels, So UMC channel = EDAC channel.226 227The EDAC subsystem provides a mechanism to handle AMD heterogeneous228systems by calling system specific ops for both CPUs and GPUs.229 230AMD GPU nodes are enumerated in sequential order based on the PCI231hierarchy, and the first GPU node is assumed to have a Node ID value232following those of the CPU nodes after latter are fully populated::233 234 $ ls /sys/devices/system/edac/mc/235 mc0 - CPU MC node 0236 mc1 |237 mc2 |- GPU card[0] => node 0(mc1), node 1(mc2)238 mc3 |239 mc4 |- GPU card[1] => node 0(mc3), node 1(mc4)240 mc5 |241 mc6 |- GPU card[2] => node 0(mc5), node 1(mc6)242 mc7 |243 mc8 |- GPU card[3] => node 0(mc7), node 1(mc8)244 245For example, a heterogeneous system with one AMD CPU is connected to246four MI200 (Aldebaran) GPUs using xGMI. This topology can be represented247via the following sysfs entries::248 249 /sys/devices/system/edac/mc/..250 251 CPU # CPU node252 ├── mc 0253 254 GPU Nodes are enumerated sequentially after CPU nodes have been populated255 GPU card 1 # Each MI200 GPU has 2 nodes/mcs256 ├── mc 1 # GPU node 0 == mc1, Each MC node has 4 UMCs/CSROWs257 │ ├── csrow 0 # UMC 0258 │ │ ├── channel 0 # Each UMC has 8 channels259 │ │ ├── channel 1 # size of each channel is 2 GB, so each UMC has 16 GB260 │ │ ├── channel 2261 │ │ ├── channel 3262 │ │ ├── channel 4263 │ │ ├── channel 5264 │ │ ├── channel 6265 │ │ ├── channel 7266 │ ├── csrow 1 # UMC 1267 │ │ ├── channel 0268 │ │ ├── ..269 │ │ ├── channel 7270 │ ├── .. ..271 │ ├── csrow 3 # UMC 3272 │ │ ├── channel 0273 │ │ ├── ..274 │ │ ├── channel 7275 │ ├── rank 0276 │ ├── .. ..277 │ ├── rank 31 # total 32 ranks/dimms from 4 UMCs278 ├279 ├── mc 2 # GPU node 1 == mc2280 │ ├── .. # each GPU has total 64 GB281 282 GPU card 2283 ├── mc 3284 │ ├── ..285 ├── mc 4286 │ ├── ..287 288 GPU card 3289 ├── mc 5290 │ ├── ..291 ├── mc 6292 │ ├── ..293 294 GPU card 4295 ├── mc 7296 │ ├── ..297 ├── mc 8298 │ ├── ..299