658 lines · plain
1===============================2LIBNVDIMM: Non-Volatile Devices3===============================4 5libnvdimm - kernel / libndctl - userspace helper library6 7nvdimm@lists.linux.dev8 9Version 1310 11.. contents:12 13 Glossary14 Overview15 Supporting Documents16 Git Trees17 LIBNVDIMM PMEM18 PMEM-REGIONs, Atomic Sectors, and DAX19 Example NVDIMM Platform20 LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API21 LIBNDCTL: Context22 libndctl: instantiate a new library context example23 LIBNVDIMM/LIBNDCTL: Bus24 libnvdimm: control class device in /sys/class25 libnvdimm: bus26 libndctl: bus enumeration example27 LIBNVDIMM/LIBNDCTL: DIMM (NMEM)28 libnvdimm: DIMM (NMEM)29 libndctl: DIMM enumeration example30 LIBNVDIMM/LIBNDCTL: Region31 libnvdimm: region32 libndctl: region enumeration example33 Why Not Encode the Region Type into the Region Name?34 How Do I Determine the Major Type of a Region?35 LIBNVDIMM/LIBNDCTL: Namespace36 libnvdimm: namespace37 libndctl: namespace enumeration example38 libndctl: namespace creation example39 Why the Term "namespace"?40 LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"41 libnvdimm: btt layout42 libndctl: btt creation example43 Summary LIBNDCTL Diagram44 45 46Glossary47========48 49PMEM:50 A system-physical-address range where writes are persistent. A51 block device composed of PMEM is capable of DAX. A PMEM address range52 may span an interleave of several DIMMs.53 54DPA:55 DIMM Physical Address, is a DIMM-relative offset. With one DIMM in56 the system there would be a 1:1 system-physical-address:DPA association.57 Once more DIMMs are added a memory controller interleave must be58 decoded to determine the DPA associated with a given59 system-physical-address.60 61DAX:62 File system extensions to bypass the page cache and block layer to63 mmap persistent memory, from a PMEM block device, directly into a64 process address space.65 66DSM:67 Device Specific Method: ACPI method to control specific68 device - in this case the firmware.69 70DCR:71 NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5.72 It defines a vendor-id, device-id, and interface format for a given DIMM.73 74BTT:75 Block Translation Table: Persistent memory is byte addressable.76 Existing software may have an expectation that the power-fail-atomicity77 of writes is at least one sector, 512 bytes. The BTT is an indirection78 table with atomic update semantics to front a PMEM block device79 driver and present arbitrary atomic sector sizes.80 81LABEL:82 Metadata stored on a DIMM device that partitions and identifies83 (persistently names) capacity allocated to different PMEM namespaces. It84 also indicates whether an address abstraction like a BTT is applied to85 the namespace. Note that traditional partition tables, GPT/MBR, are86 layered on top of a PMEM namespace, or an address abstraction like BTT87 if present, but partition support is deprecated going forward.88 89 90Overview91========92 93The LIBNVDIMM subsystem provides support for PMEM described by platform94firmware or a device driver. On ACPI based systems the platform firmware95conveys persistent memory resource via the ACPI NFIT "NVDIMM Firmware96Interface Table" in ACPI 6. While the LIBNVDIMM subsystem implementation97is generic and supports pre-NFIT platforms, it was guided by the98superset of capabilities need to support this ACPI 6 definition for99NVDIMM resources. The original implementation supported the100block-window-aperture capability described in the NFIT, but that support101has since been abandoned and never shipped in a product.102 103Supporting Documents104--------------------105 106ACPI 6:107 https://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf108NVDIMM Namespace:109 https://pmem.io/documents/NVDIMM_Namespace_Spec.pdf110DSM Interface Example:111 https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf112Driver Writer's Guide:113 https://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf114 115Git Trees116---------117 118LIBNVDIMM:119 https://git.kernel.org/cgit/linux/kernel/git/nvdimm/nvdimm.git120LIBNDCTL:121 https://github.com/pmem/ndctl.git122 123 124LIBNVDIMM PMEM125==============126 127Prior to the arrival of the NFIT, non-volatile memory was described to a128system in various ad-hoc ways. Usually only the bare minimum was129provided, namely, a single system-physical-address range where writes130are expected to be durable after a system power loss. Now, the NFIT131specification standardizes not only the description of PMEM, but also132platform message-passing entry points for control and configuration.133 134PMEM (nd_pmem.ko): Drives a system-physical-address range. This range is135contiguous in system memory and may be interleaved (hardware memory controller136striped) across multiple DIMMs. When interleaved the platform may optionally137provide details of which DIMMs are participating in the interleave.138 139It is worth noting that when the labeling capability is detected (a EFI140namespace label index block is found), then no block device is created141by default as userspace needs to do at least one allocation of DPA to142the PMEM range. In contrast ND_NAMESPACE_IO ranges, once registered,143can be immediately attached to nd_pmem. This latter mode is called144label-less or "legacy".145 146PMEM-REGIONs, Atomic Sectors, and DAX147-------------------------------------148 149For the cases where an application or filesystem still needs atomic sector150update guarantees it can register a BTT on a PMEM device or partition. See151LIBNVDIMM/NDCTL: Block Translation Table "btt"152 153 154Example NVDIMM Platform155=======================156 157For the remainder of this document the following diagram will be158referenced for any example sysfs layouts::159 160 161 (a) (b) DIMM162 +-------------------+--------+--------+--------+163 +------+ | pm0.0 | free | pm1.0 | free | 0164 | imc0 +--+- - - region0- - - +--------+ +--------+165 +--+---+ | pm0.0 | free | pm1.0 | free | 1166 | +-------------------+--------v v--------+167 +--+---+ | |168 | cpu0 | region1169 +--+---+ | |170 | +----------------------------^ ^--------+171 +--+---+ | free | pm1.0 | free | 2172 | imc1 +--+----------------------------| +--------+173 +------+ | free | pm1.0 | free | 3174 +----------------------------+--------+--------+175 176In this platform we have four DIMMs and two memory controllers in one177socket. Each PMEM interleave set is identified by a region device with178a dynamically assigned id.179 180 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A181 single PMEM namespace is created in the REGION0-SPA-range that spans most182 of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that183 interleaved system-physical-address range is left free for184 another PMEM namespace to be defined.185 186 2. In the last portion of DIMM0 and DIMM1 we have an interleaved187 system-physical-address range, REGION1, that spans those two DIMMs as188 well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace189 named "pm1.0".190 191 This bus is provided by the kernel under the device192 /sys/devices/platform/nfit_test.0 when the nfit_test.ko module from193 tools/testing/nvdimm is loaded. This module is a unit test for194 LIBNVDIMM and the acpi_nfit.ko driver.195 196 197LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API198========================================================199 200What follows is a description of the LIBNVDIMM sysfs layout and a201corresponding object hierarchy diagram as viewed through the LIBNDCTL202API. The example sysfs paths and diagrams are relative to the Example203NVDIMM Platform which is also the LIBNVDIMM bus used in the LIBNDCTL unit204test.205 206LIBNDCTL: Context207-----------------208 209Every API call in the LIBNDCTL library requires a context that holds the210logging parameters and other library instance state. The library is211based on the libabc template:212 213 https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git214 215LIBNDCTL: instantiate a new library context example216^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^217 218::219 220 struct ndctl_ctx *ctx;221 222 if (ndctl_new(&ctx) == 0)223 return ctx;224 else225 return NULL;226 227LIBNVDIMM/LIBNDCTL: Bus228-----------------------229 230A bus has a 1:1 relationship with an NFIT. The current expectation for231ACPI based systems is that there is only ever one platform-global NFIT.232That said, it is trivial to register multiple NFITs, the specification233does not preclude it. The infrastructure supports multiple busses and234we use this capability to test multiple NFIT configurations in the unit235test.236 237LIBNVDIMM: control class device in /sys/class238---------------------------------------------239 240This character device accepts DSM messages to be passed to DIMM241identified by its NFIT handle::242 243 /sys/class/nd/ndctl0244 |-- dev245 |-- device -> ../../../ndbus0246 |-- subsystem -> ../../../../../../../class/nd247 248 249 250LIBNVDIMM: bus251--------------252 253::254 255 struct nvdimm_bus *nvdimm_bus_register(struct device *parent,256 struct nvdimm_bus_descriptor *nfit_desc);257 258::259 260 /sys/devices/platform/nfit_test.0/ndbus0261 |-- commands262 |-- nd263 |-- nfit264 |-- nmem0265 |-- nmem1266 |-- nmem2267 |-- nmem3268 |-- power269 |-- provider270 |-- region0271 |-- region1272 |-- region2273 |-- region3274 |-- region4275 |-- region5276 |-- uevent277 `-- wait_probe278 279LIBNDCTL: bus enumeration example280^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^281 282Find the bus handle that describes the bus from Example NVDIMM Platform::283 284 static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx,285 const char *provider)286 {287 struct ndctl_bus *bus;288 289 ndctl_bus_foreach(ctx, bus)290 if (strcmp(provider, ndctl_bus_get_provider(bus)) == 0)291 return bus;292 293 return NULL;294 }295 296 bus = get_bus_by_provider(ctx, "nfit_test.0");297 298 299LIBNVDIMM/LIBNDCTL: DIMM (NMEM)300-------------------------------301 302The DIMM device provides a character device for sending commands to303hardware, and it is a container for LABELs. If the DIMM is defined by304NFIT then an optional 'nfit' attribute sub-directory is available to add305NFIT-specifics.306 307Note that the kernel device name for "DIMMs" is "nmemX". The NFIT308describes these devices via "Memory Device to System Physical Address309Range Mapping Structure", and there is no requirement that they actually310be physical DIMMs, so we use a more generic name.311 312LIBNVDIMM: DIMM (NMEM)313^^^^^^^^^^^^^^^^^^^^^^314 315::316 317 struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data,318 const struct attribute_group **groups, unsigned long flags,319 unsigned long *dsm_mask);320 321::322 323 /sys/devices/platform/nfit_test.0/ndbus0324 |-- nmem0325 | |-- available_slots326 | |-- commands327 | |-- dev328 | |-- devtype329 | |-- driver -> ../../../../../bus/nd/drivers/nvdimm330 | |-- modalias331 | |-- nfit332 | | |-- device333 | | |-- format334 | | |-- handle335 | | |-- phys_id336 | | |-- rev_id337 | | |-- serial338 | | `-- vendor339 | |-- state340 | |-- subsystem -> ../../../../../bus/nd341 | `-- uevent342 |-- nmem1343 [..]344 345 346LIBNDCTL: DIMM enumeration example347^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^348 349Note, in this example we are assuming NFIT-defined DIMMs which are350identified by an "nfit_handle" a 32-bit value where:351 352 - Bit 3:0 DIMM number within the memory channel353 - Bit 7:4 memory channel number354 - Bit 11:8 memory controller ID355 - Bit 15:12 socket ID (within scope of a Node controller if node356 controller is present)357 - Bit 27:16 Node Controller ID358 - Bit 31:28 Reserved359 360::361 362 static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus,363 unsigned int handle)364 {365 struct ndctl_dimm *dimm;366 367 ndctl_dimm_foreach(bus, dimm)368 if (ndctl_dimm_get_handle(dimm) == handle)369 return dimm;370 371 return NULL;372 }373 374 #define DIMM_HANDLE(n, s, i, c, d) \375 (((n & 0xfff) << 16) | ((s & 0xf) << 12) | ((i & 0xf) << 8) \376 | ((c & 0xf) << 4) | (d & 0xf))377 378 dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0));379 380LIBNVDIMM/LIBNDCTL: Region381--------------------------382 383A generic REGION device is registered for each PMEM interleave-set /384range. Per the example there are 2 PMEM regions on the "nfit_test.0"385bus. The primary role of regions are to be a container of "mappings". A386mapping is a tuple of <DIMM, DPA-start-offset, length>.387 388LIBNVDIMM provides a built-in driver for REGION devices. This driver389is responsible for all parsing LABELs, if present, and then emitting NAMESPACE390devices for the nd_pmem driver to consume.391 392In addition to the generic attributes of "mapping"s, "interleave_ways"393and "size" the REGION device also exports some convenience attributes.394"nstype" indicates the integer type of namespace-device this region395emits, "devtype" duplicates the DEVTYPE variable stored by udev at the396'add' event, "modalias" duplicates the MODALIAS variable stored by udev397at the 'add' event, and finally, the optional "spa_index" is provided in398the case where the region is defined by a SPA.399 400LIBNVDIMM: region::401 402 struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus,403 struct nd_region_desc *ndr_desc);404 405::406 407 /sys/devices/platform/nfit_test.0/ndbus0408 |-- region0409 | |-- available_size410 | |-- btt0411 | |-- btt_seed412 | |-- devtype413 | |-- driver -> ../../../../../bus/nd/drivers/nd_region414 | |-- init_namespaces415 | |-- mapping0416 | |-- mapping1417 | |-- mappings418 | |-- modalias419 | |-- namespace0.0420 | |-- namespace_seed421 | |-- numa_node422 | |-- nfit423 | | `-- spa_index424 | |-- nstype425 | |-- set_cookie426 | |-- size427 | |-- subsystem -> ../../../../../bus/nd428 | `-- uevent429 |-- region1430 [..]431 432LIBNDCTL: region enumeration example433^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^434 435Sample region retrieval routines based on NFIT-unique data like436"spa_index" (interleave set id).437 438::439 440 static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus,441 unsigned int spa_index)442 {443 struct ndctl_region *region;444 445 ndctl_region_foreach(bus, region) {446 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_PMEM)447 continue;448 if (ndctl_region_get_spa_index(region) == spa_index)449 return region;450 }451 return NULL;452 }453 454 455LIBNVDIMM/LIBNDCTL: Namespace456-----------------------------457 458A REGION, after resolving DPA aliasing and LABEL specified boundaries, surfaces459one or more "namespace" devices. The arrival of a "namespace" device currently460triggers the nd_pmem driver to load and register a disk/block device.461 462LIBNVDIMM: namespace463^^^^^^^^^^^^^^^^^^^^464 465Here is a sample layout from the 2 major types of NAMESPACE where namespace0.0466represents DIMM-info-backed PMEM (note that it has a 'uuid' attribute), and467namespace1.0 represents an anonymous PMEM namespace (note that has no 'uuid'468attribute due to not support a LABEL)469 470::471 472 /sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0473 |-- alt_name474 |-- devtype475 |-- dpa_extents476 |-- force_raw477 |-- modalias478 |-- numa_node479 |-- resource480 |-- size481 |-- subsystem -> ../../../../../../bus/nd482 |-- type483 |-- uevent484 `-- uuid485 /sys/devices/platform/nfit_test.1/ndbus1/region1/namespace1.0486 |-- block487 | `-- pmem0488 |-- devtype489 |-- driver -> ../../../../../../bus/nd/drivers/pmem490 |-- force_raw491 |-- modalias492 |-- numa_node493 |-- resource494 |-- size495 |-- subsystem -> ../../../../../../bus/nd496 |-- type497 `-- uevent498 499LIBNDCTL: namespace enumeration example500^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^501Namespaces are indexed relative to their parent region, example below.502These indexes are mostly static from boot to boot, but subsystem makes503no guarantees in this regard. For a static namespace identifier use its504'uuid' attribute.505 506::507 508 static struct ndctl_namespace509 *get_namespace_by_id(struct ndctl_region *region, unsigned int id)510 {511 struct ndctl_namespace *ndns;512 513 ndctl_namespace_foreach(region, ndns)514 if (ndctl_namespace_get_id(ndns) == id)515 return ndns;516 517 return NULL;518 }519 520LIBNDCTL: namespace creation example521^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^522 523Idle namespaces are automatically created by the kernel if a given524region has enough available capacity to create a new namespace.525Namespace instantiation involves finding an idle namespace and526configuring it. For the most part the setting of namespace attributes527can occur in any order, the only constraint is that 'uuid' must be set528before 'size'. This enables the kernel to track DPA allocations529internally with a static identifier::530 531 static int configure_namespace(struct ndctl_region *region,532 struct ndctl_namespace *ndns,533 struct namespace_parameters *parameters)534 {535 char devname[50];536 537 snprintf(devname, sizeof(devname), "namespace%d.%d",538 ndctl_region_get_id(region), paramaters->id);539 540 ndctl_namespace_set_alt_name(ndns, devname);541 /* 'uuid' must be set prior to setting size! */542 ndctl_namespace_set_uuid(ndns, paramaters->uuid);543 ndctl_namespace_set_size(ndns, paramaters->size);544 /* unlike pmem namespaces, blk namespaces have a sector size */545 if (parameters->lbasize)546 ndctl_namespace_set_sector_size(ndns, parameters->lbasize);547 ndctl_namespace_enable(ndns);548 }549 550 551Why the Term "namespace"?552^^^^^^^^^^^^^^^^^^^^^^^^^553 554 1. Why not "volume" for instance? "volume" ran the risk of confusing555 ND (libnvdimm subsystem) to a volume manager like device-mapper.556 557 2. The term originated to describe the sub-devices that can be created558 within a NVME controller (see the nvme specification:559 https://www.nvmexpress.org/specifications/), and NFIT namespaces are560 meant to parallel the capabilities and configurability of561 NVME-namespaces.562 563 564LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"565-------------------------------------------------566 567A BTT (design document: https://pmem.io/2014/09/23/btt.html) is a568personality driver for a namespace that fronts entire namespace as an569'address abstraction'.570 571LIBNVDIMM: btt layout572^^^^^^^^^^^^^^^^^^^^^573 574Every region will start out with at least one BTT device which is the575seed device. To activate it set the "namespace", "uuid", and576"sector_size" attributes and then bind the device to the nd_pmem or577nd_blk driver depending on the region type::578 579 /sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/580 |-- namespace581 |-- delete582 |-- devtype583 |-- modalias584 |-- numa_node585 |-- sector_size586 |-- subsystem -> ../../../../../bus/nd587 |-- uevent588 `-- uuid589 590LIBNDCTL: btt creation example591^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^592 593Similar to namespaces an idle BTT device is automatically created per594region. Each time this "seed" btt device is configured and enabled a new595seed is created. Creating a BTT configuration involves two steps of596finding and idle BTT and assigning it to consume a namespace.597 598::599 600 static struct ndctl_btt *get_idle_btt(struct ndctl_region *region)601 {602 struct ndctl_btt *btt;603 604 ndctl_btt_foreach(region, btt)605 if (!ndctl_btt_is_enabled(btt)606 && !ndctl_btt_is_configured(btt))607 return btt;608 609 return NULL;610 }611 612 static int configure_btt(struct ndctl_region *region,613 struct btt_parameters *parameters)614 {615 btt = get_idle_btt(region);616 617 ndctl_btt_set_uuid(btt, parameters->uuid);618 ndctl_btt_set_sector_size(btt, parameters->sector_size);619 ndctl_btt_set_namespace(btt, parameters->ndns);620 /* turn off raw mode device */621 ndctl_namespace_disable(parameters->ndns);622 /* turn on btt access */623 ndctl_btt_enable(btt);624 }625 626Once instantiated a new inactive btt seed device will appear underneath627the region.628 629Once a "namespace" is removed from a BTT that instance of the BTT device630will be deleted or otherwise reset to default values. This deletion is631only at the device model level. In order to destroy a BTT the "info632block" needs to be destroyed. Note, that to destroy a BTT the media633needs to be written in raw mode. By default, the kernel will autodetect634the presence of a BTT and disable raw mode. This autodetect behavior635can be suppressed by enabling raw mode for the namespace via the636ndctl_namespace_set_raw_mode() API.637 638 639Summary LIBNDCTL Diagram640------------------------641 642For the given example above, here is the view of the objects as seen by the643LIBNDCTL API::644 645 +---+646 |CTX|647 +-+-+648 |649 +-------+ |650 | DIMM0 <-+ | +---------+ +--------------+ +---------------+651 +-------+ | | +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" |652 | DIMM1 <-+ +-v--+ | +---------+ +--------------+ +---------------+653 +-------+ +-+BUS0+-| +---------+ +--------------+ +----------------------+654 | DIMM2 <-+ +----+ +-> REGION1 +---> NAMESPACE1.0 +--> PMEM6 "pm1.0" | BTT1 |655 +-------+ | | +---------+ +--------------+ +---------------+------+656 | DIMM3 <-+657 +-------+658