524 lines · plain
1==================2NUMA Memory Policy3==================4 5What is NUMA Memory Policy?6============================7 8In the Linux kernel, "memory policy" determines from which node the kernel will9allocate memory in a NUMA system or in an emulated NUMA system. Linux has10supported platforms with Non-Uniform Memory Access architectures since 2.4.?.11The current memory policy support was added to Linux 2.6 around May 2004. This12document attempts to describe the concepts and APIs of the 2.6 memory policy13support.14 15Memory policies should not be confused with cpusets16(``Documentation/admin-guide/cgroup-v1/cpusets.rst``)17which is an administrative mechanism for restricting the nodes from which18memory may be allocated by a set of processes. Memory policies are a19programming interface that a NUMA-aware application can take advantage of. When20both cpusets and policies are applied to a task, the restrictions of the cpuset21takes priority. See :ref:`Memory Policies and cpusets <mem_pol_and_cpusets>`22below for more details.23 24Memory Policy Concepts25======================26 27Scope of Memory Policies28------------------------29 30The Linux kernel supports _scopes_ of memory policy, described here from31most general to most specific:32 33System Default Policy34 this policy is "hard coded" into the kernel. It is the policy35 that governs all page allocations that aren't controlled by36 one of the more specific policy scopes discussed below. When37 the system is "up and running", the system default policy will38 use "local allocation" described below. However, during boot39 up, the system default policy will be set to interleave40 allocations across all nodes with "sufficient" memory, so as41 not to overload the initial boot node with boot-time42 allocations.43 44Task/Process Policy45 this is an optional, per-task policy. When defined for a46 specific task, this policy controls all page allocations made47 by or on behalf of the task that aren't controlled by a more48 specific scope. If a task does not define a task policy, then49 all page allocations that would have been controlled by the50 task policy "fall back" to the System Default Policy.51 52 The task policy applies to the entire address space of a task. Thus,53 it is inheritable, and indeed is inherited, across both fork()54 [clone() w/o the CLONE_VM flag] and exec*(). This allows a parent task55 to establish the task policy for a child task exec()'d from an56 executable image that has no awareness of memory policy. See the57 :ref:`Memory Policy APIs <memory_policy_apis>` section,58 below, for an overview of the system call59 that a task may use to set/change its task/process policy.60 61 In a multi-threaded task, task policies apply only to the thread62 [Linux kernel task] that installs the policy and any threads63 subsequently created by that thread. Any sibling threads existing64 at the time a new task policy is installed retain their current65 policy.66 67 A task policy applies only to pages allocated after the policy is68 installed. Any pages already faulted in by the task when the task69 changes its task policy remain where they were allocated based on70 the policy at the time they were allocated.71 72.. _vma_policy:73 74VMA Policy75 A "VMA" or "Virtual Memory Area" refers to a range of a task's76 virtual address space. A task may define a specific policy for a range77 of its virtual address space. See the78 :ref:`Memory Policy APIs <memory_policy_apis>` section,79 below, for an overview of the mbind() system call used to set a VMA80 policy.81 82 A VMA policy will govern the allocation of pages that back83 this region of the address space. Any regions of the task's84 address space that don't have an explicit VMA policy will fall85 back to the task policy, which may itself fall back to the86 System Default Policy.87 88 VMA policies have a few complicating details:89 90 * VMA policy applies ONLY to anonymous pages. These include91 pages allocated for anonymous segments, such as the task92 stack and heap, and any regions of the address space93 mmap()ed with the MAP_ANONYMOUS flag. If a VMA policy is94 applied to a file mapping, it will be ignored if the mapping95 used the MAP_SHARED flag. If the file mapping used the96 MAP_PRIVATE flag, the VMA policy will only be applied when97 an anonymous page is allocated on an attempt to write to the98 mapping-- i.e., at Copy-On-Write.99 100 * VMA policies are shared between all tasks that share a101 virtual address space--a.k.a. threads--independent of when102 the policy is installed; and they are inherited across103 fork(). However, because VMA policies refer to a specific104 region of a task's address space, and because the address105 space is discarded and recreated on exec*(), VMA policies106 are NOT inheritable across exec(). Thus, only NUMA-aware107 applications may use VMA policies.108 109 * A task may install a new VMA policy on a sub-range of a110 previously mmap()ed region. When this happens, Linux splits111 the existing virtual memory area into 2 or 3 VMAs, each with112 its own policy.113 114 * By default, VMA policy applies only to pages allocated after115 the policy is installed. Any pages already faulted into the116 VMA range remain where they were allocated based on the117 policy at the time they were allocated. However, since118 2.6.16, Linux supports page migration via the mbind() system119 call, so that page contents can be moved to match a newly120 installed policy.121 122Shared Policy123 Conceptually, shared policies apply to "memory objects" mapped124 shared into one or more tasks' distinct address spaces. An125 application installs shared policies the same way as VMA126 policies--using the mbind() system call specifying a range of127 virtual addresses that map the shared object. However, unlike128 VMA policies, which can be considered to be an attribute of a129 range of a task's address space, shared policies apply130 directly to the shared object. Thus, all tasks that attach to131 the object share the policy, and all pages allocated for the132 shared object, by any task, will obey the shared policy.133 134 As of 2.6.22, only shared memory segments, created by shmget() or135 mmap(MAP_ANONYMOUS|MAP_SHARED), support shared policy. When shared136 policy support was added to Linux, the associated data structures were137 added to hugetlbfs shmem segments. At the time, hugetlbfs did not138 support allocation at fault time--a.k.a lazy allocation--so hugetlbfs139 shmem segments were never "hooked up" to the shared policy support.140 Although hugetlbfs segments now support lazy allocation, their support141 for shared policy has not been completed.142 143 As mentioned above in :ref:`VMA policies <vma_policy>` section,144 allocations of page cache pages for regular files mmap()ed145 with MAP_SHARED ignore any VMA policy installed on the virtual146 address range backed by the shared file mapping. Rather,147 shared page cache pages, including pages backing private148 mappings that have not yet been written by the task, follow149 task policy, if any, else System Default Policy.150 151 The shared policy infrastructure supports different policies on subset152 ranges of the shared object. However, Linux still splits the VMA of153 the task that installs the policy for each range of distinct policy.154 Thus, different tasks that attach to a shared memory segment can have155 different VMA configurations mapping that one shared object. This156 can be seen by examining the /proc/<pid>/numa_maps of tasks sharing157 a shared memory region, when one task has installed shared policy on158 one or more ranges of the region.159 160Components of Memory Policies161-----------------------------162 163A NUMA memory policy consists of a "mode", optional mode flags, and164an optional set of nodes. The mode determines the behavior of the165policy, the optional mode flags determine the behavior of the mode,166and the optional set of nodes can be viewed as the arguments to the167policy behavior.168 169Internally, memory policies are implemented by a reference counted170structure, struct mempolicy. Details of this structure will be171discussed in context, below, as required to explain the behavior.172 173NUMA memory policy supports the following 4 behavioral modes:174 175Default Mode--MPOL_DEFAULT176 This mode is only used in the memory policy APIs. Internally,177 MPOL_DEFAULT is converted to the NULL memory policy in all178 policy scopes. Any existing non-default policy will simply be179 removed when MPOL_DEFAULT is specified. As a result,180 MPOL_DEFAULT means "fall back to the next most specific policy181 scope."182 183 For example, a NULL or default task policy will fall back to the184 system default policy. A NULL or default vma policy will fall185 back to the task policy.186 187 When specified in one of the memory policy APIs, the Default mode188 does not use the optional set of nodes.189 190 It is an error for the set of nodes specified for this policy to191 be non-empty.192 193MPOL_BIND194 This mode specifies that memory must come from the set of195 nodes specified by the policy. Memory will be allocated from196 the node in the set with sufficient free memory that is197 closest to the node where the allocation takes place.198 199MPOL_PREFERRED200 This mode specifies that the allocation should be attempted201 from the single node specified in the policy. If that202 allocation fails, the kernel will search other nodes, in order203 of increasing distance from the preferred node based on204 information provided by the platform firmware.205 206 Internally, the Preferred policy uses a single node--the207 preferred_node member of struct mempolicy. When the internal208 mode flag MPOL_F_LOCAL is set, the preferred_node is ignored209 and the policy is interpreted as local allocation. "Local"210 allocation policy can be viewed as a Preferred policy that211 starts at the node containing the cpu where the allocation212 takes place.213 214 It is possible for the user to specify that local allocation215 is always preferred by passing an empty nodemask with this216 mode. If an empty nodemask is passed, the policy cannot use217 the MPOL_F_STATIC_NODES or MPOL_F_RELATIVE_NODES flags218 described below.219 220MPOL_INTERLEAVED221 This mode specifies that page allocations be interleaved, on a222 page granularity, across the nodes specified in the policy.223 This mode also behaves slightly differently, based on the224 context where it is used:225 226 For allocation of anonymous pages and shared memory pages,227 Interleave mode indexes the set of nodes specified by the228 policy using the page offset of the faulting address into the229 segment [VMA] containing the address modulo the number of230 nodes specified by the policy. It then attempts to allocate a231 page, starting at the selected node, as if the node had been232 specified by a Preferred policy or had been selected by a233 local allocation. That is, allocation will follow the per234 node zonelist.235 236 For allocation of page cache pages, Interleave mode indexes237 the set of nodes specified by the policy using a node counter238 maintained per task. This counter wraps around to the lowest239 specified node after it reaches the highest specified node.240 This will tend to spread the pages out over the nodes241 specified by the policy based on the order in which they are242 allocated, rather than based on any page offset into an243 address range or file. During system boot up, the temporary244 interleaved system default policy works in this mode.245 246MPOL_PREFERRED_MANY247 This mode specifies that the allocation should be preferably248 satisfied from the nodemask specified in the policy. If there is249 a memory pressure on all nodes in the nodemask, the allocation250 can fall back to all existing numa nodes. This is effectively251 MPOL_PREFERRED allowed for a mask rather than a single node.252 253MPOL_WEIGHTED_INTERLEAVE254 This mode operates the same as MPOL_INTERLEAVE, except that255 interleaving behavior is executed based on weights set in256 /sys/kernel/mm/mempolicy/weighted_interleave/257 258 Weighted interleave allocates pages on nodes according to a259 weight. For example if nodes [0,1] are weighted [5,2], 5 pages260 will be allocated on node0 for every 2 pages allocated on node1.261 262NUMA memory policy supports the following optional mode flags:263 264MPOL_F_STATIC_NODES265 This flag specifies that the nodemask passed by266 the user should not be remapped if the task or VMA's set of allowed267 nodes changes after the memory policy has been defined.268 269 Without this flag, any time a mempolicy is rebound because of a270 change in the set of allowed nodes, the preferred nodemask (Preferred271 Many), preferred node (Preferred) or nodemask (Bind, Interleave) is272 remapped to the new set of allowed nodes. This may result in nodes273 being used that were previously undesired.274 275 With this flag, if the user-specified nodes overlap with the276 nodes allowed by the task's cpuset, then the memory policy is277 applied to their intersection. If the two sets of nodes do not278 overlap, the Default policy is used.279 280 For example, consider a task that is attached to a cpuset with281 mems 1-3 that sets an Interleave policy over the same set. If282 the cpuset's mems change to 3-5, the Interleave will now occur283 over nodes 3, 4, and 5. With this flag, however, since only node284 3 is allowed from the user's nodemask, the "interleave" only285 occurs over that node. If no nodes from the user's nodemask are286 now allowed, the Default behavior is used.287 288 MPOL_F_STATIC_NODES cannot be combined with the289 MPOL_F_RELATIVE_NODES flag. It also cannot be used for290 MPOL_PREFERRED policies that were created with an empty nodemask291 (local allocation).292 293MPOL_F_RELATIVE_NODES294 This flag specifies that the nodemask passed295 by the user will be mapped relative to the set of the task or VMA's296 set of allowed nodes. The kernel stores the user-passed nodemask,297 and if the allowed nodes changes, then that original nodemask will298 be remapped relative to the new set of allowed nodes.299 300 Without this flag (and without MPOL_F_STATIC_NODES), anytime a301 mempolicy is rebound because of a change in the set of allowed302 nodes, the node (Preferred) or nodemask (Bind, Interleave) is303 remapped to the new set of allowed nodes. That remap may not304 preserve the relative nature of the user's passed nodemask to its305 set of allowed nodes upon successive rebinds: a nodemask of306 1,3,5 may be remapped to 7-9 and then to 1-3 if the set of307 allowed nodes is restored to its original state.308 309 With this flag, the remap is done so that the node numbers from310 the user's passed nodemask are relative to the set of allowed311 nodes. In other words, if nodes 0, 2, and 4 are set in the user's312 nodemask, the policy will be effected over the first (and in the313 Bind or Interleave case, the third and fifth) nodes in the set of314 allowed nodes. The nodemask passed by the user represents nodes315 relative to task or VMA's set of allowed nodes.316 317 If the user's nodemask includes nodes that are outside the range318 of the new set of allowed nodes (for example, node 5 is set in319 the user's nodemask when the set of allowed nodes is only 0-3),320 then the remap wraps around to the beginning of the nodemask and,321 if not already set, sets the node in the mempolicy nodemask.322 323 For example, consider a task that is attached to a cpuset with324 mems 2-5 that sets an Interleave policy over the same set with325 MPOL_F_RELATIVE_NODES. If the cpuset's mems change to 3-7, the326 interleave now occurs over nodes 3,5-7. If the cpuset's mems327 then change to 0,2-3,5, then the interleave occurs over nodes328 0,2-3,5.329 330 Thanks to the consistent remapping, applications preparing331 nodemasks to specify memory policies using this flag should332 disregard their current, actual cpuset imposed memory placement333 and prepare the nodemask as if they were always located on334 memory nodes 0 to N-1, where N is the number of memory nodes the335 policy is intended to manage. Let the kernel then remap to the336 set of memory nodes allowed by the task's cpuset, as that may337 change over time.338 339 MPOL_F_RELATIVE_NODES cannot be combined with the340 MPOL_F_STATIC_NODES flag. It also cannot be used for341 MPOL_PREFERRED policies that were created with an empty nodemask342 (local allocation).343 344Memory Policy Reference Counting345================================346 347To resolve use/free races, struct mempolicy contains an atomic reference348count field. Internal interfaces, mpol_get()/mpol_put() increment and349decrement this reference count, respectively. mpol_put() will only free350the structure back to the mempolicy kmem cache when the reference count351goes to zero.352 353When a new memory policy is allocated, its reference count is initialized354to '1', representing the reference held by the task that is installing the355new policy. When a pointer to a memory policy structure is stored in another356structure, another reference is added, as the task's reference will be dropped357on completion of the policy installation.358 359During run-time "usage" of the policy, we attempt to minimize atomic operations360on the reference count, as this can lead to cache lines bouncing between cpus361and NUMA nodes. "Usage" here means one of the following:362 3631) querying of the policy, either by the task itself [using the get_mempolicy()364 API discussed below] or by another task using the /proc/<pid>/numa_maps365 interface.366 3672) examination of the policy to determine the policy mode and associated node368 or node lists, if any, for page allocation. This is considered a "hot369 path". Note that for MPOL_BIND, the "usage" extends across the entire370 allocation process, which may sleep during page reclamation, because the371 BIND policy nodemask is used, by reference, to filter ineligible nodes.372 373We can avoid taking an extra reference during the usages listed above as374follows:375 3761) we never need to get/free the system default policy as this is never377 changed nor freed, once the system is up and running.378 3792) for querying the policy, we do not need to take an extra reference on the380 target task's task policy nor vma policies because we always acquire the381 task's mm's mmap_lock for read during the query. The set_mempolicy() and382 mbind() APIs [see below] always acquire the mmap_lock for write when383 installing or replacing task or vma policies. Thus, there is no possibility384 of a task or thread freeing a policy while another task or thread is385 querying it.386 3873) Page allocation usage of task or vma policy occurs in the fault path where388 we hold them mmap_lock for read. Again, because replacing the task or vma389 policy requires that the mmap_lock be held for write, the policy can't be390 freed out from under us while we're using it for page allocation.391 3924) Shared policies require special consideration. One task can replace a393 shared memory policy while another task, with a distinct mmap_lock, is394 querying or allocating a page based on the policy. To resolve this395 potential race, the shared policy infrastructure adds an extra reference396 to the shared policy during lookup while holding a spin lock on the shared397 policy management structure. This requires that we drop this extra398 reference when we're finished "using" the policy. We must drop the399 extra reference on shared policies in the same query/allocation paths400 used for non-shared policies. For this reason, shared policies are marked401 as such, and the extra reference is dropped "conditionally"--i.e., only402 for shared policies.403 404 Because of this extra reference counting, and because we must lookup405 shared policies in a tree structure under spinlock, shared policies are406 more expensive to use in the page allocation path. This is especially407 true for shared policies on shared memory regions shared by tasks running408 on different NUMA nodes. This extra overhead can be avoided by always409 falling back to task or system default policy for shared memory regions,410 or by prefaulting the entire shared memory region into memory and locking411 it down. However, this might not be appropriate for all applications.412 413.. _memory_policy_apis:414 415Memory Policy APIs416==================417 418Linux supports 4 system calls for controlling memory policy. These APIS419always affect only the calling task, the calling task's address space, or420some shared object mapped into the calling task's address space.421 422.. note::423 the headers that define these APIs and the parameter data types for424 user space applications reside in a package that is not part of the425 Linux kernel. The kernel system call interfaces, with the 'sys\_'426 prefix, are defined in <linux/syscalls.h>; the mode and flag427 definitions are defined in <linux/mempolicy.h>.428 429Set [Task] Memory Policy::430 431 long set_mempolicy(int mode, const unsigned long *nmask,432 unsigned long maxnode);433 434Set's the calling task's "task/process memory policy" to mode435specified by the 'mode' argument and the set of nodes defined by436'nmask'. 'nmask' points to a bit mask of node ids containing at least437'maxnode' ids. Optional mode flags may be passed by combining the438'mode' argument with the flag (for example: MPOL_INTERLEAVE |439MPOL_F_STATIC_NODES).440 441See the set_mempolicy(2) man page for more details442 443 444Get [Task] Memory Policy or Related Information::445 446 long get_mempolicy(int *mode,447 const unsigned long *nmask, unsigned long maxnode,448 void *addr, int flags);449 450Queries the "task/process memory policy" of the calling task, or the451policy or location of a specified virtual address, depending on the452'flags' argument.453 454See the get_mempolicy(2) man page for more details455 456 457Install VMA/Shared Policy for a Range of Task's Address Space::458 459 long mbind(void *start, unsigned long len, int mode,460 const unsigned long *nmask, unsigned long maxnode,461 unsigned flags);462 463mbind() installs the policy specified by (mode, nmask, maxnodes) as a464VMA policy for the range of the calling task's address space specified465by the 'start' and 'len' arguments. Additional actions may be466requested via the 'flags' argument.467 468See the mbind(2) man page for more details.469 470Set home node for a Range of Task's Address Spacec::471 472 long sys_set_mempolicy_home_node(unsigned long start, unsigned long len,473 unsigned long home_node,474 unsigned long flags);475 476sys_set_mempolicy_home_node set the home node for a VMA policy present in the477task's address range. The system call updates the home node only for the existing478mempolicy range. Other address ranges are ignored. A home node is the NUMA node479closest to which page allocation will come from. Specifying the home node override480the default allocation policy to allocate memory close to the local node for an481executing CPU.482 483 484Memory Policy Command Line Interface485====================================486 487Although not strictly part of the Linux implementation of memory policy,488a command line tool, numactl(8), exists that allows one to:489 490+ set the task policy for a specified program via set_mempolicy(2), fork(2) and491 exec(2)492 493+ set the shared policy for a shared memory segment via mbind(2)494 495The numactl(8) tool is packaged with the run-time version of the library496containing the memory policy system call wrappers. Some distributions497package the headers and compile-time libraries in a separate development498package.499 500.. _mem_pol_and_cpusets:501 502Memory Policies and cpusets503===========================504 505Memory policies work within cpusets as described above. For memory policies506that require a node or set of nodes, the nodes are restricted to the set of507nodes whose memories are allowed by the cpuset constraints. If the nodemask508specified for the policy contains nodes that are not allowed by the cpuset and509MPOL_F_RELATIVE_NODES is not used, the intersection of the set of nodes510specified for the policy and the set of nodes with memory is used. If the511result is the empty set, the policy is considered invalid and cannot be512installed. If MPOL_F_RELATIVE_NODES is used, the policy's nodes are mapped513onto and folded into the task's set of allowed nodes as previously described.514 515The interaction of memory policies and cpusets can be problematic when tasks516in two cpusets share access to a memory region, such as shared memory segments517created by shmget() of mmap() with the MAP_ANONYMOUS and MAP_SHARED flags, and518any of the tasks install shared policy on the region, only nodes whose519memories are allowed in both cpusets may be used in the policies. Obtaining520this information requires "stepping outside" the memory policy APIs to use the521cpuset information and requires that one know in what cpusets other task might522be attaching to the shared region. Furthermore, if the cpusets' allowed523memory sets are disjoint, "local" allocation is the only valid policy.524