697 lines · plain
1==================2Memory Hot(Un)Plug3==================4 5This document describes generic Linux support for memory hot(un)plug with6a focus on System RAM, including ZONE_MOVABLE support.7 8.. contents:: :local:9 10Introduction11============12 13Memory hot(un)plug allows for increasing and decreasing the size of physical14memory available to a machine at runtime. In the simplest case, it consists of15physically plugging or unplugging a DIMM at runtime, coordinated with the16operating system.17 18Memory hot(un)plug is used for various purposes:19 20- The physical memory available to a machine can be adjusted at runtime, up- or21 downgrading the memory capacity. This dynamic memory resizing, sometimes22 referred to as "capacity on demand", is frequently used with virtual machines23 and logical partitions.24 25- Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One26 example is replacing failing memory modules.27 28- Reducing energy consumption either by physically unplugging memory modules or29 by logically unplugging (parts of) memory modules from Linux.30 31Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also32used to expose persistent memory, other performance-differentiated memory and33reserved memory regions as ordinary system RAM to Linux.34 35Linux only supports memory hot(un)plug on selected 64 bit architectures, such as36x86_64, arm64, ppc64 and s390x.37 38Memory Hot(Un)Plug Granularity39------------------------------40 41Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the42physical memory address space into chunks of the same size: memory sections. The43size of a memory section is architecture dependent. For example, x86_64 uses44128 MiB and ppc64 uses 16 MiB.45 46Memory sections are combined into chunks referred to as "memory blocks". The47size of a memory block is architecture dependent and corresponds to the smallest48granularity that can be hot(un)plugged. The default size of a memory block is49the same as memory section size, unless an architecture specifies otherwise.50 51All memory blocks have the same size.52 53Phases of Memory Hotplug54------------------------55 56Memory hotplug consists of two phases:57 58(1) Adding the memory to Linux59(2) Onlining memory blocks60 61In the first phase, metadata, such as the memory map ("memmap") and page tables62for the direct mapping, is allocated and initialized, and memory blocks are63created; the latter also creates sysfs files for managing newly created memory64blocks.65 66In the second phase, added memory is exposed to the page allocator. After this67phase, the memory is visible in memory statistics, such as free and total68memory, of the system.69 70Phases of Memory Hotunplug71--------------------------72 73Memory hotunplug consists of two phases:74 75(1) Offlining memory blocks76(2) Removing the memory from Linux77 78In the first phase, memory is "hidden" from the page allocator again, for79example, by migrating busy memory to other memory locations and removing all80relevant free pages from the page allocator After this phase, the memory is no81longer visible in memory statistics of the system.82 83In the second phase, the memory blocks are removed and metadata is freed.84 85Memory Hotplug Notifications86============================87 88There are various ways how Linux is notified about memory hotplug events such89that it can start adding hotplugged memory. This description is limited to90systems that support ACPI; mechanisms specific to other firmware interfaces or91virtual machines are not described.92 93ACPI Notifications94------------------95 96Platforms that support ACPI, such as x86_64, can support memory hotplug97notifications via ACPI.98 99In general, a firmware supporting memory hotplug defines a memory class object100HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI101driver will hotplug the memory to Linux.102 103If the firmware supports hotplug of NUMA nodes, it defines an object _HID104"ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all105assigned memory devices are added to Linux by the ACPI driver.106 107Similarly, Linux can be notified about requests to hotunplug a memory device or108a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory109blocks, and, if successful, hotunplug the memory from Linux.110 111Manual Probing112--------------113 114On some architectures, the firmware may not be able to notify the operating115system about a memory hotplug event. Instead, the memory has to be manually116probed from user space.117 118The probe interface is located at::119 120 /sys/devices/system/memory/probe121 122Only complete memory blocks can be probed. Individual memory blocks are probed123by providing the physical start address of the memory block::124 125 % echo addr > /sys/devices/system/memory/probe126 127Which results in a memory block for the range [addr, addr + memory_block_size)128being created.129 130.. note::131 132 Using the probe interface is discouraged as it is easy to crash the kernel,133 because Linux cannot validate user input; this interface might be removed in134 the future.135 136Onlining and Offlining Memory Blocks137====================================138 139After a memory block has been created, Linux has to be instructed to actually140make use of that memory: the memory block has to be "online".141 142Before a memory block can be removed, Linux has to stop using any memory part of143the memory block: the memory block has to be "offlined".144 145The Linux kernel can be configured to automatically online added memory blocks146and drivers automatically trigger offlining of memory blocks when trying147hotunplug of memory. Memory blocks can only be removed once offlining succeeded148and drivers may trigger offlining of memory blocks when attempting hotunplug of149memory.150 151Onlining Memory Blocks Manually152-------------------------------153 154If auto-onlining of memory blocks isn't enabled, user-space has to manually155trigger onlining of memory blocks. Often, udev rules are used to automate this156task in user space.157 158Onlining of a memory block can be triggered via::159 160 % echo online > /sys/devices/system/memory/memoryXXX/state161 162Or alternatively::163 164 % echo 1 > /sys/devices/system/memory/memoryXXX/online165 166The kernel will select the target zone automatically, depending on the167configured ``online_policy``.168 169One can explicitly request to associate an offline memory block with170ZONE_MOVABLE by::171 172 % echo online_movable > /sys/devices/system/memory/memoryXXX/state173 174Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by::175 176 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state177 178In any case, if onlining succeeds, the state of the memory block is changed to179be "online". If it fails, the state of the memory block will remain unchanged180and the above commands will fail.181 182Onlining Memory Blocks Automatically183------------------------------------184 185The kernel can be configured to try auto-onlining of newly added memory blocks.186If this feature is disabled, the memory blocks will stay offline until187explicitly onlined from user space.188 189The configured auto-online behavior can be observed via::190 191 % cat /sys/devices/system/memory/auto_online_blocks192 193Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or194``online_movable`` to that file, like::195 196 % echo online > /sys/devices/system/memory/auto_online_blocks197 198Similarly to manual onlining, with ``online`` the kernel will select the199target zone automatically, depending on the configured ``online_policy``.200 201Modifying the auto-online behavior will only affect all subsequently added202memory blocks only.203 204.. note::205 206 In corner cases, auto-onlining can fail. The kernel won't retry. Note that207 auto-onlining is not expected to fail in default configurations.208 209.. note::210 211 DLPAR on ppc64 ignores the ``offline`` setting and will still online added212 memory blocks; if onlining fails, memory blocks are removed again.213 214Offlining Memory Blocks215-----------------------216 217In the current implementation, Linux's memory offlining will try migrating all218movable pages off the affected memory block. As most kernel allocations, such as219page tables, are unmovable, page migration can fail and, therefore, inhibit220memory offlining from succeeding.221 222Having the memory provided by memory block managed by ZONE_MOVABLE significantly223increases memory offlining reliability; still, memory offlining can fail in224some corner cases.225 226Further, memory offlining might retry for a long time (or even forever), until227aborted by the user.228 229Offlining of a memory block can be triggered via::230 231 % echo offline > /sys/devices/system/memory/memoryXXX/state232 233Or alternatively::234 235 % echo 0 > /sys/devices/system/memory/memoryXXX/online236 237If offlining succeeds, the state of the memory block is changed to be "offline".238If it fails, the state of the memory block will remain unchanged and the above239commands will fail, for example, via::240 241 bash: echo: write error: Device or resource busy242 243or via::244 245 bash: echo: write error: Invalid argument246 247Observing the State of Memory Blocks248------------------------------------249 250The state (online/offline/going-offline) of a memory block can be observed251either via::252 253 % cat /sys/devices/system/memory/memoryXXX/state254 255Or alternatively (1/0) via::256 257 % cat /sys/devices/system/memory/memoryXXX/online258 259For an online memory block, the managing zone can be observed via::260 261 % cat /sys/devices/system/memory/memoryXXX/valid_zones262 263Configuring Memory Hot(Un)Plug264==============================265 266There are various ways how system administrators can configure memory267hot(un)plug and interact with memory blocks, especially, to online them.268 269Memory Hot(Un)Plug Configuration via Sysfs270------------------------------------------271 272Some memory hot(un)plug properties can be configured or inspected via sysfs in::273 274 /sys/devices/system/memory/275 276The following files are currently defined:277 278====================== =========================================================279``auto_online_blocks`` read-write: set or get the default state of new memory280 blocks; configure auto-onlining.281 282 The default value depends on the283 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel configuration284 option.285 286 See the ``state`` property of memory blocks for details.287``block_size_bytes`` read-only: the size in bytes of a memory block.288``probe`` write-only: add (probe) selected memory blocks manually289 from user space by supplying the physical start address.290 291 Availability depends on the CONFIG_ARCH_MEMORY_PROBE292 kernel configuration option.293``uevent`` read-write: generic udev file for device subsystems.294``crash_hotplug`` read-only: when changes to the system memory map295 occur due to hot un/plug of memory, this file contains296 '1' if the kernel updates the kdump capture kernel memory297 map itself (via elfcorehdr and other relevant kexec298 segments), or '0' if userspace must update the kdump299 capture kernel memory map.300 301 Availability depends on the CONFIG_MEMORY_HOTPLUG kernel302 configuration option.303====================== =========================================================304 305.. note::306 307 When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two308 additional files ``hard_offline_page`` and ``soft_offline_page`` are available309 to trigger hwpoisoning of pages, for example, for testing purposes. Note that310 this functionality is not really related to memory hot(un)plug or actual311 offlining of memory blocks.312 313Memory Block Configuration via Sysfs314------------------------------------315 316Each memory block is represented as a memory block device that can be317onlined or offlined. All memory blocks have their device information located in318sysfs. Each present memory block is listed under319``/sys/devices/system/memory`` as::320 321 /sys/devices/system/memory/memoryXXX322 323where XXX is the memory block id; the number of digits is variable.324 325A present memory block indicates that some memory in the range is present;326however, a memory block might span memory holes. A memory block spanning memory327holes cannot be offlined.328 329For example, assume 1 GiB memory block size. A device for a memory starting at3300x100000000 is ``/sys/devices/system/memory/memory4``::331 332 (0x100000000 / 1Gib = 4)333 334This device covers address range [0x100000000 ... 0x140000000)335 336The following files are currently defined:337 338=================== ============================================================339``online`` read-write: simplified interface to trigger onlining /340 offlining and to observe the state of a memory block.341 When onlining, the zone is selected automatically.342``phys_device`` read-only: legacy interface only ever used on s390x to343 expose the covered storage increment.344``phys_index`` read-only: the memory block id (XXX).345``removable`` read-only: legacy interface that indicated whether a memory346 block was likely to be offlineable or not. Nowadays, the347 kernel return ``1`` if and only if it supports memory348 offlining.349``state`` read-write: advanced interface to trigger onlining /350 offlining and to observe the state of a memory block.351 352 When writing, ``online``, ``offline``, ``online_kernel`` and353 ``online_movable`` are supported.354 355 ``online_movable`` specifies onlining to ZONE_MOVABLE.356 ``online_kernel`` specifies onlining to the default kernel357 zone for the memory block, such as ZONE_NORMAL.358 ``online`` let's the kernel select the zone automatically.359 360 When reading, ``online``, ``offline`` and ``going-offline``361 may be returned.362``uevent`` read-write: generic uevent file for devices.363``valid_zones`` read-only: when a block is online, shows the zone it364 belongs to; when a block is offline, shows what zone will365 manage it when the block will be onlined.366 367 For online memory blocks, ``DMA``, ``DMA32``, ``Normal``,368 ``Movable`` and ``none`` may be returned. ``none`` indicates369 that memory provided by a memory block is managed by370 multiple zones or spans multiple nodes; such memory blocks371 cannot be offlined. ``Movable`` indicates ZONE_MOVABLE.372 Other values indicate a kernel zone.373 374 For offline memory blocks, the first column shows the375 zone the kernel would select when onlining the memory block376 right now without further specifying a zone.377 378 Availability depends on the CONFIG_MEMORY_HOTREMOVE379 kernel configuration option.380=================== ============================================================381 382.. note::383 384 If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/385 directories can also be accessed via symbolic links located in the386 ``/sys/devices/system/node/node*`` directories.387 388 For example::389 390 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9391 392 A backlink will also be created::393 394 /sys/devices/system/memory/memory9/node0 -> ../../node/node0395 396Command Line Parameters397-----------------------398 399Some command line parameters affect memory hot(un)plug handling. The following400command line parameters are relevant:401 402======================== =======================================================403``memhp_default_state`` configure auto-onlining by essentially setting404 ``/sys/devices/system/memory/auto_online_blocks``.405``movable_node`` configure automatic zone selection in the kernel when406 using the ``contig-zones`` online policy. When407 set, the kernel will default to ZONE_MOVABLE when408 onlining a memory block, unless other zones can be kept409 contiguous.410======================== =======================================================411 412See Documentation/admin-guide/kernel-parameters.txt for a more generic413description of these command line parameters.414 415Module Parameters416------------------417 418Instead of additional command line parameters or sysfs files, the419``memory_hotplug`` subsystem now provides a dedicated namespace for module420parameters. Module parameters can be set via the command line by predicating421them with ``memory_hotplug.`` such as::422 423 memory_hotplug.memmap_on_memory=1424 425and they can be observed (and some even modified at runtime) via::426 427 /sys/module/memory_hotplug/parameters/428 429The following module parameters are currently defined:430 431================================ ===============================================432``memmap_on_memory`` read-write: Allocate memory for the memmap from433 the added memory block itself. Even if enabled,434 actual support depends on various other system435 properties and should only be regarded as a436 hint whether the behavior would be desired.437 438 While allocating the memmap from the memory439 block itself makes memory hotplug less likely440 to fail and keeps the memmap on the same NUMA441 node in any case, it can fragment physical442 memory in a way that huge pages in bigger443 granularity cannot be formed on hotplugged444 memory.445 446 With value "force" it could result in memory447 wastage due to memmap size limitations. For448 example, if the memmap for a memory block449 requires 1 MiB, but the pageblock size is 2450 MiB, 1 MiB of hotplugged memory will be wasted.451 Note that there are still cases where the452 feature cannot be enforced: for example, if the453 memmap is smaller than a single page, or if the454 architecture does not support the forced mode455 in all configurations.456 457``online_policy`` read-write: Set the basic policy used for458 automatic zone selection when onlining memory459 blocks without specifying a target zone.460 ``contig-zones`` has been the kernel default461 before this parameter was added. After an462 online policy was configured and memory was463 online, the policy should not be changed464 anymore.465 466 When set to ``contig-zones``, the kernel will467 try keeping zones contiguous. If a memory block468 intersects multiple zones or no zone, the469 behavior depends on the ``movable_node`` kernel470 command line parameter: default to ZONE_MOVABLE471 if set, default to the applicable kernel zone472 (usually ZONE_NORMAL) if not set.473 474 When set to ``auto-movable``, the kernel will475 try onlining memory blocks to ZONE_MOVABLE if476 possible according to the configuration and477 memory device details. With this policy, one478 can avoid zone imbalances when eventually479 hotplugging a lot of memory later and still480 wanting to be able to hotunplug as much as481 possible reliably, very desirable in482 virtualized environments. This policy ignores483 the ``movable_node`` kernel command line484 parameter and isn't really applicable in485 environments that require it (e.g., bare metal486 with hotunpluggable nodes) where hotplugged487 memory might be exposed via the488 firmware-provided memory map early during boot489 to the system instead of getting detected,490 added and onlined later during boot (such as491 done by virtio-mem or by some hypervisors492 implementing emulated DIMMs). As one example, a493 hotplugged DIMM will be onlined either494 completely to ZONE_MOVABLE or completely to495 ZONE_NORMAL, not a mixture.496 As another example, as many memory blocks497 belonging to a virtio-mem device will be498 onlined to ZONE_MOVABLE as possible,499 special-casing units of memory blocks that can500 only get hotunplugged together. *This policy501 does not protect from setups that are502 problematic with ZONE_MOVABLE and does not503 change the zone of memory blocks dynamically504 after they were onlined.*505``auto_movable_ratio`` read-write: Set the maximum MOVABLE:KERNEL506 memory ratio in % for the ``auto-movable``507 online policy. Whether the ratio applies only508 for the system across all NUMA nodes or also509 per NUMA nodes depends on the510 ``auto_movable_numa_aware`` configuration.511 512 All accounting is based on present memory pages513 in the zones combined with accounting per514 memory device. Memory dedicated to the CMA515 allocator is accounted as MOVABLE, although516 residing on one of the kernel zones. The517 possible ratio depends on the actual workload.518 The kernel default is "301" %, for example,519 allowing for hotplugging 24 GiB to a 8 GiB VM520 and automatically onlining all hotplugged521 memory to ZONE_MOVABLE in many setups. The522 additional 1% deals with some pages being not523 present, for example, because of some firmware524 allocations.525 526 Note that ZONE_NORMAL memory provided by one527 memory device does not allow for more528 ZONE_MOVABLE memory for a different memory529 device. As one example, onlining memory of a530 hotplugged DIMM to ZONE_NORMAL will not allow531 for another hotplugged DIMM to get onlined to532 ZONE_MOVABLE automatically. In contrast, memory533 hotplugged by a virtio-mem device that got534 onlined to ZONE_NORMAL will allow for more535 ZONE_MOVABLE memory within *the same*536 virtio-mem device.537``auto_movable_numa_aware`` read-write: Configure whether the538 ``auto_movable_ratio`` in the ``auto-movable``539 online policy also applies per NUMA540 node in addition to the whole system across all541 NUMA nodes. The kernel default is "Y".542 543 Disabling NUMA awareness can be helpful when544 dealing with NUMA nodes that should be545 completely hotunpluggable, onlining the memory546 completely to ZONE_MOVABLE automatically if547 possible.548 549 Parameter availability depends on CONFIG_NUMA.550================================ ===============================================551 552ZONE_MOVABLE553============554 555ZONE_MOVABLE is an important mechanism for more reliable memory offlining.556Further, having system RAM managed by ZONE_MOVABLE instead of one of the557kernel zones can increase the number of possible transparent huge pages and558dynamically allocated huge pages.559 560Most kernel allocations are unmovable. Important examples include the memory561map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations562can only be served from the kernel zones.563 564Most user space pages, such as anonymous memory, and page cache pages are565movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones.566 567Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable568allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is569absolutely no guarantee whether a memory block can be offlined successfully.570 571Zone Imbalances572---------------573 574Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance,575which can harm the system or degrade performance. As one example, the kernel576might crash because it runs out of free memory for unmovable allocations,577although there is still plenty of free memory left in ZONE_MOVABLE.578 579Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1580are definitely impossible due to the overhead for the memory map.581 582Actual safe zone ratios depend on the workload. Extreme cases, like excessive583long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all.584 585.. note::586 587 CMA memory part of a kernel zone essentially behaves like memory in588 ZONE_MOVABLE and similar considerations apply, especially when combining589 CMA with ZONE_MOVABLE.590 591ZONE_MOVABLE Sizing Considerations592----------------------------------593 594We usually expect that a large portion of available system RAM will actually595be consumed by user space, either directly or indirectly via the page cache. In596the normal case, ZONE_MOVABLE can be used when allocating such pages just fine.597 598With that in mind, it makes sense that we can have a big portion of system RAM599managed by ZONE_MOVABLE. However, there are some things to consider when using600ZONE_MOVABLE, especially when fine-tuning zone ratios:601 602- Having a lot of offline memory blocks. Even offline memory blocks consume603 memory for metadata and page tables in the direct map; having a lot of offline604 memory blocks is not a typical case, though.605 606- Memory ballooning without balloon compaction is incompatible with607 ZONE_MOVABLE. Only some implementations, such as virtio-balloon and608 pseries CMM, fully support balloon compaction.609 610 Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be611 disabled. In that case, balloon inflation will only perform unmovable612 allocations and silently create a zone imbalance, usually triggered by613 inflation requests from the hypervisor.614 615- Gigantic pages are unmovable, resulting in user space consuming a616 lot of unmovable memory.617 618- Huge pages are unmovable when an architectures does not support huge619 page migration, resulting in a similar issue as with gigantic pages.620 621- Page tables are unmovable. Excessive swapping, mapping extremely large622 files or ZONE_DEVICE memory can be problematic, although only really relevant623 in corner cases. When we manage a lot of user space memory that has been624 swapped out or is served from a file/persistent memory/... we still need a lot625 of page tables to manage that memory once user space accessed that memory.626 627- In certain DAX configurations the memory map for the device memory will be628 allocated from the kernel zones.629 630- KASAN can have a significant memory overhead, for example, consuming 1/8th of631 the total system memory size as (unmovable) tracking metadata.632 633- Long-term pinning of pages. Techniques that rely on long-term pinnings634 (especially, RDMA and vfio/mdev) are fundamentally problematic with635 ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside636 on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they637 have to be migrated off that zone while pinning. Pinning a page can fail638 even if there is plenty of free memory in ZONE_MOVABLE.639 640 In addition, using ZONE_MOVABLE might make page pinning more expensive,641 because of the page migration overhead.642 643By default, all the memory configured at boot time is managed by the kernel644zones and ZONE_MOVABLE is not used.645 646To enable ZONE_MOVABLE to include the memory present at boot and to control the647ratio between movable and kernel zones there are two command line options:648``kernelcore=`` and ``movablecore=``. See649Documentation/admin-guide/kernel-parameters.rst for their description.650 651Memory Offlining and ZONE_MOVABLE652---------------------------------653 654Even with ZONE_MOVABLE, there are some corner cases where offlining a memory655block might fail:656 657- Memory blocks with memory holes; this applies to memory blocks present during658 boot and can apply to memory blocks hotplugged via the XEN balloon and the659 Hyper-V balloon.660 661- Mixed NUMA nodes and mixed zones within a single memory block prevent memory662 offlining; this applies to memory blocks present during boot only.663 664- Special memory blocks prevented by the system from getting offlined. Examples665 include any memory available during boot on arm64 or memory blocks spanning666 the crashkernel area on s390x; this usually applies to memory blocks present667 during boot only.668 669- Memory blocks overlapping with CMA areas cannot be offlined, this applies to670 memory blocks present during boot only.671 672- Concurrent activity that operates on the same physical memory area, such as673 allocating gigantic pages, can result in temporary offlining failures.674 675- Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap676 Optimization (HVO) is enabled.677 678 Offlining code may be able to migrate huge page contents, but may not be able679 to dissolve the source huge page because it fails allocating (unmovable) pages680 for the vmemmap, because the system might not have free memory in the kernel681 zones left.682 683 Users that depend on memory offlining to succeed for movable zones should684 carefully consider whether the memory savings gained from this feature are685 worth the risk of possibly not being able to offline memory in certain686 situations.687 688Further, when running into out of memory situations while migrating pages, or689when still encountering permanently unmovable pages within ZONE_MOVABLE690(-> BUG), memory offlining will keep retrying until it eventually succeeds.691 692When offlining is triggered from user space, the offlining context can be693terminated by sending a signal. A timeout based offlining can easily be694implemented via::695 696 % timeout $TIMEOUT offline_block | failure_handling697