596 lines · plain
1=====================2Hugetlbfs Reservation3=====================4 5Overview6========7 8Huge pages as described at Documentation/admin-guide/mm/hugetlbpage.rst are9typically preallocated for application use. These huge pages are instantiated10in a task's address space at page fault time if the VMA indicates huge pages11are to be used. If no huge page exists at page fault time, the task is sent12a SIGBUS and often dies an unhappy death. Shortly after huge page support13was added, it was determined that it would be better to detect a shortage14of huge pages at mmap() time. The idea is that if there were not enough15huge pages to cover the mapping, the mmap() would fail. This was first16done with a simple check in the code at mmap() time to determine if there17were enough free huge pages to cover the mapping. Like most things in the18kernel, the code has evolved over time. However, the basic idea was to19'reserve' huge pages at mmap() time to ensure that huge pages would be20available for page faults in that mapping. The description below attempts to21describe how huge page reserve processing is done in the v4.10 kernel.22 23 24Audience25========26This description is primarily targeted at kernel developers who are modifying27hugetlbfs code.28 29 30The Data Structures31===================32 33resv_huge_pages34 This is a global (per-hstate) count of reserved huge pages. Reserved35 huge pages are only available to the task which reserved them.36 Therefore, the number of huge pages generally available is computed37 as (``free_huge_pages - resv_huge_pages``).38Reserve Map39 A reserve map is described by the structure::40 41 struct resv_map {42 struct kref refs;43 spinlock_t lock;44 struct list_head regions;45 long adds_in_progress;46 struct list_head region_cache;47 long region_cache_count;48 };49 50 There is one reserve map for each huge page mapping in the system.51 The regions list within the resv_map describes the regions within52 the mapping. A region is described as::53 54 struct file_region {55 struct list_head link;56 long from;57 long to;58 };59 60 The 'from' and 'to' fields of the file region structure are huge page61 indices into the mapping. Depending on the type of mapping, a62 region in the reserv_map may indicate reservations exist for the63 range, or reservations do not exist.64Flags for MAP_PRIVATE Reservations65 These are stored in the bottom bits of the reservation map pointer.66 67 ``#define HPAGE_RESV_OWNER (1UL << 0)``68 Indicates this task is the owner of the reservations69 associated with the mapping.70 ``#define HPAGE_RESV_UNMAPPED (1UL << 1)``71 Indicates task originally mapping this range (and creating72 reserves) has unmapped a page from this task (the child)73 due to a failed COW.74Page Flags75 The PagePrivate page flag is used to indicate that a huge page76 reservation must be restored when the huge page is freed. More77 details will be discussed in the "Freeing huge pages" section.78 79 80Reservation Map Location (Private or Shared)81============================================82 83A huge page mapping or segment is either private or shared. If private,84it is typically only available to a single address space (task). If shared,85it can be mapped into multiple address spaces (tasks). The location and86semantics of the reservation map is significantly different for the two types87of mappings. Location differences are:88 89- For private mappings, the reservation map hangs off the VMA structure.90 Specifically, vma->vm_private_data. This reserve map is created at the91 time the mapping (mmap(MAP_PRIVATE)) is created.92- For shared mappings, the reservation map hangs off the inode. Specifically,93 inode->i_mapping->private_data. Since shared mappings are always backed94 by files in the hugetlbfs filesystem, the hugetlbfs code ensures each inode95 contains a reservation map. As a result, the reservation map is allocated96 when the inode is created.97 98 99Creating Reservations100=====================101Reservations are created when a huge page backed shared memory segment is102created (shmget(SHM_HUGETLB)) or a mapping is created via mmap(MAP_HUGETLB).103These operations result in a call to the routine hugetlb_reserve_pages()::104 105 int hugetlb_reserve_pages(struct inode *inode,106 long from, long to,107 struct vm_area_struct *vma,108 vm_flags_t vm_flags)109 110The first thing hugetlb_reserve_pages() does is check if the NORESERVE111flag was specified in either the shmget() or mmap() call. If NORESERVE112was specified, then this routine returns immediately as no reservations113are desired.114 115The arguments 'from' and 'to' are huge page indices into the mapping or116underlying file. For shmget(), 'from' is always 0 and 'to' corresponds to117the length of the segment/mapping. For mmap(), the offset argument could118be used to specify the offset into the underlying file. In such a case,119the 'from' and 'to' arguments have been adjusted by this offset.120 121One of the big differences between PRIVATE and SHARED mappings is the way122in which reservations are represented in the reservation map.123 124- For shared mappings, an entry in the reservation map indicates a reservation125 exists or did exist for the corresponding page. As reservations are126 consumed, the reservation map is not modified.127- For private mappings, the lack of an entry in the reservation map indicates128 a reservation exists for the corresponding page. As reservations are129 consumed, entries are added to the reservation map. Therefore, the130 reservation map can also be used to determine which reservations have131 been consumed.132 133For private mappings, hugetlb_reserve_pages() creates the reservation map and134hangs it off the VMA structure. In addition, the HPAGE_RESV_OWNER flag is set135to indicate this VMA owns the reservations.136 137The reservation map is consulted to determine how many huge page reservations138are needed for the current mapping/segment. For private mappings, this is139always the value (to - from). However, for shared mappings it is possible that140some reservations may already exist within the range (to - from). See the141section :ref:`Reservation Map Modifications <resv_map_modifications>`142for details on how this is accomplished.143 144The mapping may be associated with a subpool. If so, the subpool is consulted145to ensure there is sufficient space for the mapping. It is possible that the146subpool has set aside reservations that can be used for the mapping. See the147section :ref:`Subpool Reservations <sub_pool_resv>` for more details.148 149After consulting the reservation map and subpool, the number of needed new150reservations is known. The routine hugetlb_acct_memory() is called to check151for and take the requested number of reservations. hugetlb_acct_memory()152calls into routines that potentially allocate and adjust surplus page counts.153However, within those routines the code is simply checking to ensure there154are enough free huge pages to accommodate the reservation. If there are,155the global reservation count resv_huge_pages is adjusted something like the156following::157 158 if (resv_needed <= (resv_huge_pages - free_huge_pages))159 resv_huge_pages += resv_needed;160 161Note that the global lock hugetlb_lock is held when checking and adjusting162these counters.163 164If there were enough free huge pages and the global count resv_huge_pages165was adjusted, then the reservation map associated with the mapping is166modified to reflect the reservations. In the case of a shared mapping, a167file_region will exist that includes the range 'from' - 'to'. For private168mappings, no modifications are made to the reservation map as lack of an169entry indicates a reservation exists.170 171If hugetlb_reserve_pages() was successful, the global reservation count and172reservation map associated with the mapping will be modified as required to173ensure reservations exist for the range 'from' - 'to'.174 175.. _consume_resv:176 177Consuming Reservations/Allocating a Huge Page178=============================================179 180Reservations are consumed when huge pages associated with the reservations181are allocated and instantiated in the corresponding mapping. The allocation182is performed within the routine alloc_hugetlb_folio()::183 184 struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,185 unsigned long addr, int avoid_reserve)186 187alloc_hugetlb_folio is passed a VMA pointer and a virtual address, so it can188consult the reservation map to determine if a reservation exists. In addition,189alloc_hugetlb_folio takes the argument avoid_reserve which indicates reserves190should not be used even if it appears they have been set aside for the191specified address. The avoid_reserve argument is most often used in the case192of Copy on Write and Page Migration where additional copies of an existing193page are being allocated.194 195The helper routine vma_needs_reservation() is called to determine if a196reservation exists for the address within the mapping(vma). See the section197:ref:`Reservation Map Helper Routines <resv_map_helpers>` for detailed198information on what this routine does.199The value returned from vma_needs_reservation() is generally2000 or 1. 0 if a reservation exists for the address, 1 if no reservation exists.201If a reservation does not exist, and there is a subpool associated with the202mapping the subpool is consulted to determine if it contains reservations.203If the subpool contains reservations, one can be used for this allocation.204However, in every case the avoid_reserve argument overrides the use of205a reservation for the allocation. After determining whether a reservation206exists and can be used for the allocation, the routine dequeue_huge_page_vma()207is called. This routine takes two arguments related to reservations:208 209- avoid_reserve, this is the same value/argument passed to210 alloc_hugetlb_folio().211- chg, even though this argument is of type long only the values 0 or 1 are212 passed to dequeue_huge_page_vma. If the value is 0, it indicates a213 reservation exists (see the section "Memory Policy and Reservations" for214 possible issues). If the value is 1, it indicates a reservation does not215 exist and the page must be taken from the global free pool if possible.216 217The free lists associated with the memory policy of the VMA are searched for218a free page. If a page is found, the value free_huge_pages is decremented219when the page is removed from the free list. If there was a reservation220associated with the page, the following adjustments are made::221 222 SetPagePrivate(page); /* Indicates allocating this page consumed223 * a reservation, and if an error is224 * encountered such that the page must be225 * freed, the reservation will be restored. */226 resv_huge_pages--; /* Decrement the global reservation count */227 228Note, if no huge page can be found that satisfies the VMA's memory policy229an attempt will be made to allocate one using the buddy allocator. This230brings up the issue of surplus huge pages and overcommit which is beyond231the scope reservations. Even if a surplus page is allocated, the same232reservation based adjustments as above will be made: SetPagePrivate(page) and233resv_huge_pages--.234 235After obtaining a new hugetlb folio, (folio)->_hugetlb_subpool is set to the236value of the subpool associated with the page if it exists. This will be used237for subpool accounting when the folio is freed.238 239The routine vma_commit_reservation() is then called to adjust the reserve240map based on the consumption of the reservation. In general, this involves241ensuring the page is represented within a file_region structure of the region242map. For shared mappings where the reservation was present, an entry243in the reserve map already existed so no change is made. However, if there244was no reservation in a shared mapping or this was a private mapping a new245entry must be created.246 247It is possible that the reserve map could have been changed between the call248to vma_needs_reservation() at the beginning of alloc_hugetlb_folio() and the249call to vma_commit_reservation() after the folio was allocated. This would250be possible if hugetlb_reserve_pages was called for the same page in a shared251mapping. In such cases, the reservation count and subpool free page count252will be off by one. This rare condition can be identified by comparing the253return value from vma_needs_reservation and vma_commit_reservation. If such254a race is detected, the subpool and global reserve counts are adjusted to255compensate. See the section256:ref:`Reservation Map Helper Routines <resv_map_helpers>` for more257information on these routines.258 259 260Instantiate Huge Pages261======================262 263After huge page allocation, the page is typically added to the page tables264of the allocating task. Before this, pages in a shared mapping are added265to the page cache and pages in private mappings are added to an anonymous266reverse mapping. In both cases, the PagePrivate flag is cleared. Therefore,267when a huge page that has been instantiated is freed no adjustment is made268to the global reservation count (resv_huge_pages).269 270 271Freeing Huge Pages272==================273 274Huge pages are freed by free_huge_folio(). It is only passed a pointer275to the folio as it is called from the generic MM code. When a huge page276is freed, reservation accounting may need to be performed. This would277be the case if the page was associated with a subpool that contained278reserves, or the page is being freed on an error path where a global279reserve count must be restored.280 281The page->private field points to any subpool associated with the page.282If the PagePrivate flag is set, it indicates the global reserve count should283be adjusted (see the section284:ref:`Consuming Reservations/Allocating a Huge Page <consume_resv>`285for information on how these are set).286 287The routine first calls hugepage_subpool_put_pages() for the page. If this288routine returns a value of 0 (which does not equal the value passed 1) it289indicates reserves are associated with the subpool, and this newly free page290must be used to keep the number of subpool reserves above the minimum size.291Therefore, the global resv_huge_pages counter is incremented in this case.292 293If the PagePrivate flag was set in the page, the global resv_huge_pages counter294will always be incremented.295 296.. _sub_pool_resv:297 298Subpool Reservations299====================300 301There is a struct hstate associated with each huge page size. The hstate302tracks all huge pages of the specified size. A subpool represents a subset303of pages within a hstate that is associated with a mounted hugetlbfs304filesystem.305 306When a hugetlbfs filesystem is mounted a min_size option can be specified307which indicates the minimum number of huge pages required by the filesystem.308If this option is specified, the number of huge pages corresponding to309min_size are reserved for use by the filesystem. This number is tracked in310the min_hpages field of a struct hugepage_subpool. At mount time,311hugetlb_acct_memory(min_hpages) is called to reserve the specified number of312huge pages. If they can not be reserved, the mount fails.313 314The routines hugepage_subpool_get/put_pages() are called when pages are315obtained from or released back to a subpool. They perform all subpool316accounting, and track any reservations associated with the subpool.317hugepage_subpool_get/put_pages are passed the number of huge pages by which318to adjust the subpool 'used page' count (down for get, up for put). Normally,319they return the same value that was passed or an error if not enough pages320exist in the subpool.321 322However, if reserves are associated with the subpool a return value less323than the passed value may be returned. This return value indicates the324number of additional global pool adjustments which must be made. For example,325suppose a subpool contains 3 reserved huge pages and someone asks for 5.326The 3 reserved pages associated with the subpool can be used to satisfy part327of the request. But, 2 pages must be obtained from the global pools. To328relay this information to the caller, the value 2 is returned. The caller329is then responsible for attempting to obtain the additional two pages from330the global pools.331 332 333COW and Reservations334====================335 336Since shared mappings all point to and use the same underlying pages, the337biggest reservation concern for COW is private mappings. In this case,338two tasks can be pointing at the same previously allocated page. One task339attempts to write to the page, so a new page must be allocated so that each340task points to its own page.341 342When the page was originally allocated, the reservation for that page was343consumed. When an attempt to allocate a new page is made as a result of344COW, it is possible that no free huge pages are free and the allocation345will fail.346 347When the private mapping was originally created, the owner of the mapping348was noted by setting the HPAGE_RESV_OWNER bit in the pointer to the reservation349map of the owner. Since the owner created the mapping, the owner owns all350the reservations associated with the mapping. Therefore, when a write fault351occurs and there is no page available, different action is taken for the owner352and non-owner of the reservation.353 354In the case where the faulting task is not the owner, the fault will fail and355the task will typically receive a SIGBUS.356 357If the owner is the faulting task, we want it to succeed since it owned the358original reservation. To accomplish this, the page is unmapped from the359non-owning task. In this way, the only reference is from the owning task.360In addition, the HPAGE_RESV_UNMAPPED bit is set in the reservation map pointer361of the non-owning task. The non-owning task may receive a SIGBUS if it later362faults on a non-present page. But, the original owner of the363mapping/reservation will behave as expected.364 365 366.. _resv_map_modifications:367 368Reservation Map Modifications369=============================370 371The following low level routines are used to make modifications to a372reservation map. Typically, these routines are not called directly. Rather,373a reservation map helper routine is called which calls one of these low level374routines. These low level routines are fairly well documented in the source375code (mm/hugetlb.c). These routines are::376 377 long region_chg(struct resv_map *resv, long f, long t);378 long region_add(struct resv_map *resv, long f, long t);379 void region_abort(struct resv_map *resv, long f, long t);380 long region_count(struct resv_map *resv, long f, long t);381 382Operations on the reservation map typically involve two operations:383 3841) region_chg() is called to examine the reserve map and determine how385 many pages in the specified range [f, t) are NOT currently represented.386 387 The calling code performs global checks and allocations to determine if388 there are enough huge pages for the operation to succeed.389 3902)391 a) If the operation can succeed, region_add() is called to actually modify392 the reservation map for the same range [f, t) previously passed to393 region_chg().394 b) If the operation can not succeed, region_abort is called for the same395 range [f, t) to abort the operation.396 397Note that this is a two step process where region_add() and region_abort()398are guaranteed to succeed after a prior call to region_chg() for the same399range. region_chg() is responsible for pre-allocating any data structures400necessary to ensure the subsequent operations (specifically region_add()))401will succeed.402 403As mentioned above, region_chg() determines the number of pages in the range404which are NOT currently represented in the map. This number is returned to405the caller. region_add() returns the number of pages in the range added to406the map. In most cases, the return value of region_add() is the same as the407return value of region_chg(). However, in the case of shared mappings it is408possible for changes to the reservation map to be made between the calls to409region_chg() and region_add(). In this case, the return value of region_add()410will not match the return value of region_chg(). It is likely that in such411cases global counts and subpool accounting will be incorrect and in need of412adjustment. It is the responsibility of the caller to check for this condition413and make the appropriate adjustments.414 415The routine region_del() is called to remove regions from a reservation map.416It is typically called in the following situations:417 418- When a file in the hugetlbfs filesystem is being removed, the inode will419 be released and the reservation map freed. Before freeing the reservation420 map, all the individual file_region structures must be freed. In this case421 region_del is passed the range [0, LONG_MAX).422- When a hugetlbfs file is being truncated. In this case, all allocated pages423 after the new file size must be freed. In addition, any file_region entries424 in the reservation map past the new end of file must be deleted. In this425 case, region_del is passed the range [new_end_of_file, LONG_MAX).426- When a hole is being punched in a hugetlbfs file. In this case, huge pages427 are removed from the middle of the file one at a time. As the pages are428 removed, region_del() is called to remove the corresponding entry from the429 reservation map. In this case, region_del is passed the range430 [page_idx, page_idx + 1).431 432In every case, region_del() will return the number of pages removed from the433reservation map. In VERY rare cases, region_del() can fail. This can only434happen in the hole punch case where it has to split an existing file_region435entry and can not allocate a new structure. In this error case, region_del()436will return -ENOMEM. The problem here is that the reservation map will437indicate that there is a reservation for the page. However, the subpool and438global reservation counts will not reflect the reservation. To handle this439situation, the routine hugetlb_fix_reserve_counts() is called to adjust the440counters so that they correspond with the reservation map entry that could441not be deleted.442 443region_count() is called when unmapping a private huge page mapping. In444private mappings, the lack of a entry in the reservation map indicates that445a reservation exists. Therefore, by counting the number of entries in the446reservation map we know how many reservations were consumed and how many are447outstanding (outstanding = (end - start) - region_count(resv, start, end)).448Since the mapping is going away, the subpool and global reservation counts449are decremented by the number of outstanding reservations.450 451.. _resv_map_helpers:452 453Reservation Map Helper Routines454===============================455 456Several helper routines exist to query and modify the reservation maps.457These routines are only interested with reservations for a specific huge458page, so they just pass in an address instead of a range. In addition,459they pass in the associated VMA. From the VMA, the type of mapping (private460or shared) and the location of the reservation map (inode or VMA) can be461determined. These routines simply call the underlying routines described462in the section "Reservation Map Modifications". However, they do take into463account the 'opposite' meaning of reservation map entries for private and464shared mappings and hide this detail from the caller::465 466 long vma_needs_reservation(struct hstate *h,467 struct vm_area_struct *vma,468 unsigned long addr)469 470This routine calls region_chg() for the specified page. If no reservation471exists, 1 is returned. If a reservation exists, 0 is returned::472 473 long vma_commit_reservation(struct hstate *h,474 struct vm_area_struct *vma,475 unsigned long addr)476 477This calls region_add() for the specified page. As in the case of region_chg478and region_add, this routine is to be called after a previous call to479vma_needs_reservation. It will add a reservation entry for the page. It480returns 1 if the reservation was added and 0 if not. The return value should481be compared with the return value of the previous call to482vma_needs_reservation. An unexpected difference indicates the reservation483map was modified between calls::484 485 void vma_end_reservation(struct hstate *h,486 struct vm_area_struct *vma,487 unsigned long addr)488 489This calls region_abort() for the specified page. As in the case of region_chg490and region_abort, this routine is to be called after a previous call to491vma_needs_reservation. It will abort/end the in progress reservation add492operation::493 494 long vma_add_reservation(struct hstate *h,495 struct vm_area_struct *vma,496 unsigned long addr)497 498This is a special wrapper routine to help facilitate reservation cleanup499on error paths. It is only called from the routine restore_reserve_on_error().500This routine is used in conjunction with vma_needs_reservation in an attempt501to add a reservation to the reservation map. It takes into account the502different reservation map semantics for private and shared mappings. Hence,503region_add is called for shared mappings (as an entry present in the map504indicates a reservation), and region_del is called for private mappings (as505the absence of an entry in the map indicates a reservation). See the section506"Reservation cleanup in error paths" for more information on what needs to507be done on error paths.508 509 510Reservation Cleanup in Error Paths511==================================512 513As mentioned in the section514:ref:`Reservation Map Helper Routines <resv_map_helpers>`, reservation515map modifications are performed in two steps. First vma_needs_reservation516is called before a page is allocated. If the allocation is successful,517then vma_commit_reservation is called. If not, vma_end_reservation is called.518Global and subpool reservation counts are adjusted based on success or failure519of the operation and all is well.520 521Additionally, after a huge page is instantiated the PagePrivate flag is522cleared so that accounting when the page is ultimately freed is correct.523 524However, there are several instances where errors are encountered after a huge525page is allocated but before it is instantiated. In this case, the page526allocation has consumed the reservation and made the appropriate subpool,527reservation map and global count adjustments. If the page is freed at this528time (before instantiation and clearing of PagePrivate), then free_huge_folio529will increment the global reservation count. However, the reservation map530indicates the reservation was consumed. This resulting inconsistent state531will cause the 'leak' of a reserved huge page. The global reserve count will532be higher than it should and prevent allocation of a pre-allocated page.533 534The routine restore_reserve_on_error() attempts to handle this situation. It535is fairly well documented. The intention of this routine is to restore536the reservation map to the way it was before the page allocation. In this537way, the state of the reservation map will correspond to the global reservation538count after the page is freed.539 540The routine restore_reserve_on_error itself may encounter errors while541attempting to restore the reservation map entry. In this case, it will542simply clear the PagePrivate flag of the page. In this way, the global543reserve count will not be incremented when the page is freed. However, the544reservation map will continue to look as though the reservation was consumed.545A page can still be allocated for the address, but it will not use a reserved546page as originally intended.547 548There is some code (most notably userfaultfd) which can not call549restore_reserve_on_error. In this case, it simply modifies the PagePrivate550so that a reservation will not be leaked when the huge page is freed.551 552 553Reservations and Memory Policy554==============================555Per-node huge page lists existed in struct hstate when git was first used556to manage Linux code. The concept of reservations was added some time later.557When reservations were added, no attempt was made to take memory policy558into account. While cpusets are not exactly the same as memory policy, this559comment in hugetlb_acct_memory sums up the interaction between reservations560and cpusets/memory policy::561 562 /*563 * When cpuset is configured, it breaks the strict hugetlb page564 * reservation as the accounting is done on a global variable. Such565 * reservation is completely rubbish in the presence of cpuset because566 * the reservation is not checked against page availability for the567 * current cpuset. Application can still potentially OOM'ed by kernel568 * with lack of free htlb page in cpuset that the task is in.569 * Attempt to enforce strict accounting with cpuset is almost570 * impossible (or too ugly) because cpuset is too fluid that571 * task or memory node can be dynamically moved between cpusets.572 *573 * The change of semantics for shared hugetlb mapping with cpuset is574 * undesirable. However, in order to preserve some of the semantics,575 * we fall back to check against current free page availability as576 * a best attempt and hopefully to minimize the impact of changing577 * semantics that cpuset has.578 */579 580Huge page reservations were added to prevent unexpected page allocation581failures (OOM) at page fault time. However, if an application makes use582of cpusets or memory policy there is no guarantee that huge pages will be583available on the required nodes. This is true even if there are a sufficient584number of global reservations.585 586Hugetlbfs regression testing587============================588 589The most complete set of hugetlb tests are in the libhugetlbfs repository.590If you modify any hugetlb related code, use the libhugetlbfs test suite591to check for regressions. In addition, if you add any new hugetlb592functionality, please add appropriate tests to libhugetlbfs.593 594--595Mike Kravetz, 7 April 2017596