431 lines · plain
1===========2Userfaultfd3===========4 5Objective6=========7 8Userfaults allow the implementation of on-demand paging from userland9and more generally they allow userland to take control of various10memory page faults, something otherwise only the kernel code could do.11 12For example userfaults allows a proper and more optimal implementation13of the ``PROT_NONE+SIGSEGV`` trick.14 15Design16======17 18Userspace creates a new userfaultfd, initializes it, and registers one or more19regions of virtual memory with it. Then, any page faults which occur within the20region(s) result in a message being delivered to the userfaultfd, notifying21userspace of the fault.22 23The ``userfaultfd`` (aside from registering and unregistering virtual24memory ranges) provides two primary functionalities:25 261) ``read/POLLIN`` protocol to notify a userland thread of the faults27 happening28 292) various ``UFFDIO_*`` ioctls that can manage the virtual memory regions30 registered in the ``userfaultfd`` that allows userland to efficiently31 resolve the userfaults it receives via 1) or to manage the virtual32 memory in the background33 34The real advantage of userfaults if compared to regular virtual memory35management of mremap/mprotect is that the userfaults in all their36operations never involve heavyweight structures like vmas (in fact the37``userfaultfd`` runtime load never takes the mmap_lock for writing).38Vmas are not suitable for page- (or hugepage) granular fault tracking39when dealing with virtual address spaces that could span40Terabytes. Too many vmas would be needed for that.41 42The ``userfaultfd``, once created, can also be43passed using unix domain sockets to a manager process, so the same44manager process could handle the userfaults of a multitude of45different processes without them being aware about what is going on46(well of course unless they later try to use the ``userfaultfd``47themselves on the same region the manager is already tracking, which48is a corner case that would currently return ``-EBUSY``).49 50API51===52 53Creating a userfaultfd54----------------------55 56There are two ways to create a new userfaultfd, each of which provide ways to57restrict access to this functionality (since historically userfaultfds which58handle kernel page faults have been a useful tool for exploiting the kernel).59 60The first way, supported since userfaultfd was introduced, is the61userfaultfd(2) syscall. Access to this is controlled in several ways:62 63- Any user can always create a userfaultfd which traps userspace page faults64 only. Such a userfaultfd can be created using the userfaultfd(2) syscall65 with the flag UFFD_USER_MODE_ONLY.66 67- In order to also trap kernel page faults for the address space, either the68 process needs the CAP_SYS_PTRACE capability, or the system must have69 vm.unprivileged_userfaultfd set to 1. By default, vm.unprivileged_userfaultfd70 is set to 0.71 72The second way, added to the kernel more recently, is by opening73/dev/userfaultfd and issuing a USERFAULTFD_IOC_NEW ioctl to it. This method74yields equivalent userfaultfds to the userfaultfd(2) syscall.75 76Unlike userfaultfd(2), access to /dev/userfaultfd is controlled via normal77filesystem permissions (user/group/mode), which gives fine grained access to78userfaultfd specifically, without also granting other unrelated privileges at79the same time (as e.g. granting CAP_SYS_PTRACE would do). Users who have access80to /dev/userfaultfd can always create userfaultfds that trap kernel page faults;81vm.unprivileged_userfaultfd is not considered.82 83Initializing a userfaultfd84--------------------------85 86When first opened the ``userfaultfd`` must be enabled invoking the87``UFFDIO_API`` ioctl specifying a ``uffdio_api.api`` value set to ``UFFD_API`` (or88a later API version) which will specify the ``read/POLLIN`` protocol89userland intends to speak on the ``UFFD`` and the ``uffdio_api.features``90userland requires. The ``UFFDIO_API`` ioctl if successful (i.e. if the91requested ``uffdio_api.api`` is spoken also by the running kernel and the92requested features are going to be enabled) will return into93``uffdio_api.features`` and ``uffdio_api.ioctls`` two 64bit bitmasks of94respectively all the available features of the read(2) protocol and95the generic ioctl available.96 97The ``uffdio_api.features`` bitmask returned by the ``UFFDIO_API`` ioctl98defines what memory types are supported by the ``userfaultfd`` and what99events, except page fault notifications, may be generated:100 101- The ``UFFD_FEATURE_EVENT_*`` flags indicate that various other events102 other than page faults are supported. These events are described in more103 detail below in the `Non-cooperative userfaultfd`_ section.104 105- ``UFFD_FEATURE_MISSING_HUGETLBFS`` and ``UFFD_FEATURE_MISSING_SHMEM``106 indicate that the kernel supports ``UFFDIO_REGISTER_MODE_MISSING``107 registrations for hugetlbfs and shared memory (covering all shmem APIs,108 i.e. tmpfs, ``IPCSHM``, ``/dev/zero``, ``MAP_SHARED``, ``memfd_create``,109 etc) virtual memory areas, respectively.110 111- ``UFFD_FEATURE_MINOR_HUGETLBFS`` indicates that the kernel supports112 ``UFFDIO_REGISTER_MODE_MINOR`` registration for hugetlbfs virtual memory113 areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating114 support for shmem virtual memory areas.115 116- ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an117 existing page contents from userspace.118 119The userland application should set the feature flags it intends to use120when invoking the ``UFFDIO_API`` ioctl, to request that those features be121enabled if supported.122 123Once the ``userfaultfd`` API has been enabled the ``UFFDIO_REGISTER``124ioctl should be invoked (if present in the returned ``uffdio_api.ioctls``125bitmask) to register a memory range in the ``userfaultfd`` by setting the126uffdio_register structure accordingly. The ``uffdio_register.mode``127bitmask will specify to the kernel which kind of faults to track for128the range. The ``UFFDIO_REGISTER`` ioctl will return the129``uffdio_register.ioctls`` bitmask of ioctls that are suitable to resolve130userfaults on the range registered. Not all ioctls will necessarily be131supported for all memory types (e.g. anonymous memory vs. shmem vs.132hugetlbfs), or all types of intercepted faults.133 134Userland can use the ``uffdio_register.ioctls`` to manage the virtual135address space in the background (to add or potentially also remove136memory from the ``userfaultfd`` registered range). This means a userfault137could be triggering just before userland maps in the background the138user-faulted page.139 140Resolving Userfaults141--------------------142 143There are three basic ways to resolve userfaults:144 145- ``UFFDIO_COPY`` atomically copies some existing page contents from146 userspace.147 148- ``UFFDIO_ZEROPAGE`` atomically zeros the new page.149 150- ``UFFDIO_CONTINUE`` maps an existing, previously-populated page.151 152These operations are atomic in the sense that they guarantee nothing can153see a half-populated page, since readers will keep userfaulting until the154operation has finished.155 156By default, these wake up userfaults blocked on the range in question.157They support a ``UFFDIO_*_MODE_DONTWAKE`` ``mode`` flag, which indicates158that waking will be done separately at some later time.159 160Which ioctl to choose depends on the kind of page fault, and what we'd161like to do to resolve it:162 163- For ``UFFDIO_REGISTER_MODE_MISSING`` faults, the fault needs to be164 resolved by either providing a new page (``UFFDIO_COPY``), or mapping165 the zero page (``UFFDIO_ZEROPAGE``). By default, the kernel would map166 the zero page for a missing fault. With userfaultfd, userspace can167 decide what content to provide before the faulting thread continues.168 169- For ``UFFDIO_REGISTER_MODE_MINOR`` faults, there is an existing page (in170 the page cache). Userspace has the option of modifying the page's171 contents before resolving the fault. Once the contents are correct172 (modified or not), userspace asks the kernel to map the page and let the173 faulting thread continue with ``UFFDIO_CONTINUE``.174 175Notes:176 177- You can tell which kind of fault occurred by examining178 ``pagefault.flags`` within the ``uffd_msg``, checking for the179 ``UFFD_PAGEFAULT_FLAG_*`` flags.180 181- None of the page-delivering ioctls default to the range that you182 registered with. You must fill in all fields for the appropriate183 ioctl struct including the range.184 185- You get the address of the access that triggered the missing page186 event out of a struct uffd_msg that you read in the thread from the187 uffd. You can supply as many pages as you want with these IOCTLs.188 Keep in mind that unless you used DONTWAKE then the first of any of189 those IOCTLs wakes up the faulting thread.190 191- Be sure to test for all errors including192 (``pollfd[0].revents & POLLERR``). This can happen, e.g. when ranges193 supplied were incorrect.194 195Write Protect Notifications196---------------------------197 198This is equivalent to (but faster than) using mprotect and a SIGSEGV199signal handler.200 201Firstly you need to register a range with ``UFFDIO_REGISTER_MODE_WP``.202Instead of using mprotect(2) you use203``ioctl(uffd, UFFDIO_WRITEPROTECT, struct *uffdio_writeprotect)``204while ``mode = UFFDIO_WRITEPROTECT_MODE_WP``205in the struct passed in. The range does not default to and does not206have to be identical to the range you registered with. You can write207protect as many ranges as you like (inside the registered range).208Then, in the thread reading from uffd the struct will have209``msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP`` set. Now you send210``ioctl(uffd, UFFDIO_WRITEPROTECT, struct *uffdio_writeprotect)``211again while ``pagefault.mode`` does not have ``UFFDIO_WRITEPROTECT_MODE_WP``212set. This wakes up the thread which will continue to run with writes. This213allows you to do the bookkeeping about the write in the uffd reading214thread before the ioctl.215 216If you registered with both ``UFFDIO_REGISTER_MODE_MISSING`` and217``UFFDIO_REGISTER_MODE_WP`` then you need to think about the sequence in218which you supply a page and undo write protect. Note that there is a219difference between writes into a WP area and into a !WP area. The220former will have ``UFFD_PAGEFAULT_FLAG_WP`` set, the latter221``UFFD_PAGEFAULT_FLAG_WRITE``. The latter did not fail on protection but222you still need to supply a page when ``UFFDIO_REGISTER_MODE_MISSING`` was223used.224 225Userfaultfd write-protect mode currently behave differently on none ptes226(when e.g. page is missing) over different types of memories.227 228For anonymous memory, ``ioctl(UFFDIO_WRITEPROTECT)`` will ignore none ptes229(e.g. when pages are missing and not populated). For file-backed memories230like shmem and hugetlbfs, none ptes will be write protected just like a231present pte. In other words, there will be a userfaultfd write fault232message generated when writing to a missing page on file typed memories,233as long as the page range was write-protected before. Such a message will234not be generated on anonymous memories by default.235 236If the application wants to be able to write protect none ptes on anonymous237memory, one can pre-populate the memory with e.g. MADV_POPULATE_READ. On238newer kernels, one can also detect the feature UFFD_FEATURE_WP_UNPOPULATED239and set the feature bit in advance to make sure none ptes will also be240write protected even upon anonymous memory.241 242When using ``UFFDIO_REGISTER_MODE_WP`` in combination with either243``UFFDIO_REGISTER_MODE_MISSING`` or ``UFFDIO_REGISTER_MODE_MINOR``, when244resolving missing / minor faults with ``UFFDIO_COPY`` or ``UFFDIO_CONTINUE``245respectively, it may be desirable for the new page / mapping to be246write-protected (so future writes will also result in a WP fault). These ioctls247support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP``248respectively) to configure the mapping this way.249 250If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set,251any vma registered with write-protection will work in async mode rather252than the default sync mode.253 254In async mode, there will be no message generated when a write operation255happens, meanwhile the write-protection will be resolved automatically by256the kernel. It can be seen as a more accurate version of soft-dirty257tracking and it can be different in a few ways:258 259 - The dirty result will not be affected by vma changes (e.g. vma260 merging) because the dirty is only tracked by the pte.261 262 - It supports range operations by default, so one can enable tracking on263 any range of memory as long as page aligned.264 265 - Dirty information will not get lost if the pte was zapped due to266 various reasons (e.g. during split of a shmem transparent huge page).267 268 - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit269 set; dirty when uffd-wp bit cleared), it has different semantics on270 some of the memory operations. For example: ``MADV_DONTNEED`` on271 anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as272 dirtying of memory by dropping uffd-wp bit during the procedure.273 274The user app can collect the "written/dirty" status by looking up the275uffd-wp bit for the pages being interested in /proc/pagemap.276 277The page will not be under track of uffd-wp async mode until the page is278explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode279flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault280that was tracked by async mode userfaultfd-wp is invalid.281 282When userfaultfd-wp async mode is used alone, it can be applied to all283kinds of memory.284 285Memory Poisioning Emulation286---------------------------287 288In response to a fault (either missing or minor), an action userspace can289take to "resolve" it is to issue a ``UFFDIO_POISON``. This will cause any290future faulters to either get a SIGBUS, or in KVM's case the guest will291receive an MCE as if there were hardware memory poisoning.292 293This is used to emulate hardware memory poisoning. Imagine a VM running on a294machine which experiences a real hardware memory error. Later, we live migrate295the VM to another physical machine. Since we want the migration to be296transparent to the guest, we want that same address range to act as if it was297still poisoned, even though it's on a new physical host which ostensibly298doesn't have a memory error in the exact same spot.299 300QEMU/KVM301========302 303QEMU/KVM is using the ``userfaultfd`` syscall to implement postcopy live304migration. Postcopy live migration is one form of memory305externalization consisting of a virtual machine running with part or306all of its memory residing on a different node in the cloud. The307``userfaultfd`` abstraction is generic enough that not a single line of308KVM kernel code had to be modified in order to add postcopy live309migration to QEMU.310 311Guest async page faults, ``FOLL_NOWAIT`` and all other ``GUP*`` features work312just fine in combination with userfaults. Userfaults trigger async313page faults in the guest scheduler so those guest processes that314aren't waiting for userfaults (i.e. network bound) can keep running in315the guest vcpus.316 317It is generally beneficial to run one pass of precopy live migration318just before starting postcopy live migration, in order to avoid319generating userfaults for readonly guest regions.320 321The implementation of postcopy live migration currently uses one322single bidirectional socket but in the future two different sockets323will be used (to reduce the latency of the userfaults to the minimum324possible without having to decrease ``/proc/sys/net/ipv4/tcp_wmem``).325 326The QEMU in the source node writes all pages that it knows are missing327in the destination node, into the socket, and the migration thread of328the QEMU running in the destination node runs ``UFFDIO_COPY|ZEROPAGE``329ioctls on the ``userfaultfd`` in order to map the received pages into the330guest (``UFFDIO_ZEROCOPY`` is used if the source page was a zero page).331 332A different postcopy thread in the destination node listens with333poll() to the ``userfaultfd`` in parallel. When a ``POLLIN`` event is334generated after a userfault triggers, the postcopy thread read() from335the ``userfaultfd`` and receives the fault address (or ``-EAGAIN`` in case the336userfault was already resolved and waken by a ``UFFDIO_COPY|ZEROPAGE`` run337by the parallel QEMU migration thread).338 339After the QEMU postcopy thread (running in the destination node) gets340the userfault address it writes the information about the missing page341into the socket. The QEMU source node receives the information and342roughly "seeks" to that page address and continues sending all343remaining missing pages from that new page offset. Soon after that344(just the time to flush the tcp_wmem queue through the network) the345migration thread in the QEMU running in the destination node will346receive the page that triggered the userfault and it'll map it as347usual with the ``UFFDIO_COPY|ZEROPAGE`` (without actually knowing if it348was spontaneously sent by the source or if it was an urgent page349requested through a userfault).350 351By the time the userfaults start, the QEMU in the destination node352doesn't need to keep any per-page state bitmap relative to the live353migration around and a single per-page bitmap has to be maintained in354the QEMU running in the source node to know which pages are still355missing in the destination node. The bitmap in the source node is356checked to find which missing pages to send in round robin and we seek357over it when receiving incoming userfaults. After sending each page of358course the bitmap is updated accordingly. It's also useful to avoid359sending the same page twice (in case the userfault is read by the360postcopy thread just before ``UFFDIO_COPY|ZEROPAGE`` runs in the migration361thread).362 363Non-cooperative userfaultfd364===========================365 366When the ``userfaultfd`` is monitored by an external manager, the manager367must be able to track changes in the process virtual memory368layout. Userfaultfd can notify the manager about such changes using369the same read(2) protocol as for the page fault notifications. The370manager has to explicitly enable these events by setting appropriate371bits in ``uffdio_api.features`` passed to ``UFFDIO_API`` ioctl:372 373``UFFD_FEATURE_EVENT_FORK``374 enable ``userfaultfd`` hooks for fork(). When this feature is375 enabled, the ``userfaultfd`` context of the parent process is376 duplicated into the newly created process. The manager377 receives ``UFFD_EVENT_FORK`` with file descriptor of the new378 ``userfaultfd`` context in the ``uffd_msg.fork``.379 380``UFFD_FEATURE_EVENT_REMAP``381 enable notifications about mremap() calls. When the382 non-cooperative process moves a virtual memory area to a383 different location, the manager will receive384 ``UFFD_EVENT_REMAP``. The ``uffd_msg.remap`` will contain the old and385 new addresses of the area and its original length.386 387``UFFD_FEATURE_EVENT_REMOVE``388 enable notifications about madvise(MADV_REMOVE) and389 madvise(MADV_DONTNEED) calls. The event ``UFFD_EVENT_REMOVE`` will390 be generated upon these calls to madvise(). The ``uffd_msg.remove``391 will contain start and end addresses of the removed area.392 393``UFFD_FEATURE_EVENT_UNMAP``394 enable notifications about memory unmapping. The manager will395 get ``UFFD_EVENT_UNMAP`` with ``uffd_msg.remove`` containing start and396 end addresses of the unmapped area.397 398Although the ``UFFD_FEATURE_EVENT_REMOVE`` and ``UFFD_FEATURE_EVENT_UNMAP``399are pretty similar, they quite differ in the action expected from the400``userfaultfd`` manager. In the former case, the virtual memory is401removed, but the area is not, the area remains monitored by the402``userfaultfd``, and if a page fault occurs in that area it will be403delivered to the manager. The proper resolution for such page fault is404to zeromap the faulting address. However, in the latter case, when an405area is unmapped, either explicitly (with munmap() system call), or406implicitly (e.g. during mremap()), the area is removed and in turn the407``userfaultfd`` context for such area disappears too and the manager will408not get further userland page faults from the removed area. Still, the409notification is required in order to prevent manager from using410``UFFDIO_COPY`` on the unmapped area.411 412Unlike userland page faults which have to be synchronous and require413explicit or implicit wakeup, all the events are delivered414asynchronously and the non-cooperative process resumes execution as415soon as manager executes read(). The ``userfaultfd`` manager should416carefully synchronize calls to ``UFFDIO_COPY`` with the events417processing. To aid the synchronization, the ``UFFDIO_COPY`` ioctl will418return ``-ENOSPC`` when the monitored process exits at the time of419``UFFDIO_COPY``, and ``-ENOENT``, when the non-cooperative process has changed420its virtual memory layout simultaneously with outstanding ``UFFDIO_COPY``421operation.422 423The current asynchronous model of the event delivery is optimal for424single threaded non-cooperative ``userfaultfd`` manager implementations. A425synchronous event delivery model can be added later as a new426``userfaultfd`` feature to facilitate multithreading enhancements of the427non cooperative manager, for example to allow ``UFFDIO_COPY`` ioctls to428run in parallel to the event reception. Single threaded429implementations should continue to use the current async event430delivery model instead.431