140 lines · plain
1==================2HugeTLB Controller3==================4 5HugeTLB controller can be created by first mounting the cgroup filesystem.6 7# mount -t cgroup -o hugetlb none /sys/fs/cgroup8 9With the above step, the initial or the parent HugeTLB group becomes10visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in11the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.12 13New groups can be created under the parent group /sys/fs/cgroup::14 15 # cd /sys/fs/cgroup16 # mkdir g117 # echo $$ > g1/tasks18 19The above steps create a new group g1 and move the current shell20process (bash) into it.21 22Brief summary of control files::23 24 hugetlb.<hugepagesize>.rsvd.limit_in_bytes # set/show limit of "hugepagesize" hugetlb reservations25 hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes # show max "hugepagesize" hugetlb reservations and no-reserve faults26 hugetlb.<hugepagesize>.rsvd.usage_in_bytes # show current reservations and no-reserve faults for "hugepagesize" hugetlb27 hugetlb.<hugepagesize>.rsvd.failcnt # show the number of allocation failure due to HugeTLB reservation limit28 hugetlb.<hugepagesize>.limit_in_bytes # set/show limit of "hugepagesize" hugetlb faults29 hugetlb.<hugepagesize>.max_usage_in_bytes # show max "hugepagesize" hugetlb usage recorded30 hugetlb.<hugepagesize>.usage_in_bytes # show current usage for "hugepagesize" hugetlb31 hugetlb.<hugepagesize>.failcnt # show the number of allocation failure due to HugeTLB usage limit32 hugetlb.<hugepagesize>.numa_stat # show the numa information of the hugetlb memory charged to this cgroup33 34For a system supporting three hugepage sizes (64k, 32M and 1G), the control35files include::36 37 hugetlb.1GB.limit_in_bytes38 hugetlb.1GB.max_usage_in_bytes39 hugetlb.1GB.numa_stat40 hugetlb.1GB.usage_in_bytes41 hugetlb.1GB.failcnt42 hugetlb.1GB.rsvd.limit_in_bytes43 hugetlb.1GB.rsvd.max_usage_in_bytes44 hugetlb.1GB.rsvd.usage_in_bytes45 hugetlb.1GB.rsvd.failcnt46 hugetlb.64KB.limit_in_bytes47 hugetlb.64KB.max_usage_in_bytes48 hugetlb.64KB.numa_stat49 hugetlb.64KB.usage_in_bytes50 hugetlb.64KB.failcnt51 hugetlb.64KB.rsvd.limit_in_bytes52 hugetlb.64KB.rsvd.max_usage_in_bytes53 hugetlb.64KB.rsvd.usage_in_bytes54 hugetlb.64KB.rsvd.failcnt55 hugetlb.32MB.limit_in_bytes56 hugetlb.32MB.max_usage_in_bytes57 hugetlb.32MB.numa_stat58 hugetlb.32MB.usage_in_bytes59 hugetlb.32MB.failcnt60 hugetlb.32MB.rsvd.limit_in_bytes61 hugetlb.32MB.rsvd.max_usage_in_bytes62 hugetlb.32MB.rsvd.usage_in_bytes63 hugetlb.32MB.rsvd.failcnt64 65 661. Page fault accounting67 68::69 70 hugetlb.<hugepagesize>.limit_in_bytes71 hugetlb.<hugepagesize>.max_usage_in_bytes72 hugetlb.<hugepagesize>.usage_in_bytes73 hugetlb.<hugepagesize>.failcnt74 75The HugeTLB controller allows users to limit the HugeTLB usage (page fault) per76control group and enforces the limit during page fault. Since HugeTLB77doesn't support page reclaim, enforcing the limit at page fault time implies78that, the application will get SIGBUS signal if it tries to fault in HugeTLB79pages beyond its limit. Therefore the application needs to know exactly how many80HugeTLB pages it uses before hand, and the sysadmin needs to make sure that81there are enough available on the machine for all the users to avoid processes82getting SIGBUS.83 84 852. Reservation accounting86 87::88 89 hugetlb.<hugepagesize>.rsvd.limit_in_bytes90 hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes91 hugetlb.<hugepagesize>.rsvd.usage_in_bytes92 hugetlb.<hugepagesize>.rsvd.failcnt93 94The HugeTLB controller allows to limit the HugeTLB reservations per control95group and enforces the controller limit at reservation time and at the fault of96HugeTLB memory for which no reservation exists. Since reservation limits are97enforced at reservation time (on mmap or shget), reservation limits never causes98the application to get SIGBUS signal if the memory was reserved before hand. For99MAP_NORESERVE allocations, the reservation limit behaves the same as the fault100limit, enforcing memory usage at fault time and causing the application to101receive a SIGBUS if it's crossing its limit.102 103Reservation limits are superior to page fault limits described above, since104reservation limits are enforced at reservation time (on mmap or shget), and105never causes the application to get SIGBUS signal if the memory was reserved106before hand. This allows for easier fallback to alternatives such as107non-HugeTLB memory for example. In the case of page fault accounting, it's very108hard to avoid processes getting SIGBUS since the sysadmin needs precisely know109the HugeTLB usage of all the tasks in the system and make sure there is enough110pages to satisfy all requests. Avoiding tasks getting SIGBUS on overcommited111systems is practically impossible with page fault accounting.112 113 1143. Caveats with shared memory115 116For shared HugeTLB memory, both HugeTLB reservation and page faults are charged117to the first task that causes the memory to be reserved or faulted, and all118subsequent uses of this reserved or faulted memory is done without charging.119 120Shared HugeTLB memory is only uncharged when it is unreserved or deallocated.121This is usually when the HugeTLB file is deleted, and not when the task that122caused the reservation or fault has exited.123 124 1254. Caveats with HugeTLB cgroup offline.126 127When a HugeTLB cgroup goes offline with some reservations or faults still128charged to it, the behavior is as follows:129 130- The fault charges are charged to the parent HugeTLB cgroup (reparented),131- the reservation charges remain on the offline HugeTLB cgroup.132 133This means that if a HugeTLB cgroup gets offlined while there is still HugeTLB134reservations charged to it, that cgroup persists as a zombie until all HugeTLB135reservations are uncharged. HugeTLB reservations behave in this manner to match136the memory controller whose cgroups also persist as zombie until all charged137memory is uncharged. Also, the tracking of HugeTLB reservations is a bit more138complex compared to the tracking of HugeTLB faults, so it is significantly139harder to reparent reservations at offline time.140