556 lines · c
1/* SPDX-License-Identifier: MIT */2/*3 * Copyright © 2022 Intel Corporation4 */5 6#ifndef _XE_VM_DOC_H_7#define _XE_VM_DOC_H_8 9/**10 * DOC: XE VM (user address space)11 *12 * VM creation13 * ===========14 *15 * Allocate a physical page for root of the page table structure, create default16 * bind engine, and return a handle to the user.17 *18 * Scratch page19 * ------------20 *21 * If the VM is created with the flag, DRM_XE_VM_CREATE_FLAG_SCRATCH_PAGE, set the22 * entire page table structure defaults pointing to blank page allocated by the23 * VM. Invalid memory access rather than fault just read / write to this page.24 *25 * VM bind (create GPU mapping for a BO or userptr)26 * ================================================27 *28 * Creates GPU mappings for a BO or userptr within a VM. VM binds uses the same29 * in / out fence interface (struct drm_xe_sync) as execs which allows users to30 * think of binds and execs as more or less the same operation.31 *32 * Operations33 * ----------34 *35 * DRM_XE_VM_BIND_OP_MAP - Create mapping for a BO36 * DRM_XE_VM_BIND_OP_UNMAP - Destroy mapping for a BO / userptr37 * DRM_XE_VM_BIND_OP_MAP_USERPTR - Create mapping for userptr38 *39 * Implementation details40 * ~~~~~~~~~~~~~~~~~~~~~~41 *42 * All bind operations are implemented via a hybrid approach of using the CPU43 * and GPU to modify page tables. If a new physical page is allocated in the44 * page table structure we populate that page via the CPU and insert that new45 * page into the existing page table structure via a GPU job. Also any existing46 * pages in the page table structure that need to be modified also are updated47 * via the GPU job. As the root physical page is prealloced on VM creation our48 * GPU job will always have at least 1 update. The in / out fences are passed to49 * this job so again this is conceptually the same as an exec.50 *51 * Very simple example of few binds on an empty VM with 48 bits of address space52 * and the resulting operations:53 *54 * .. code-block::55 *56 * bind BO0 0x0-0x100057 * alloc page level 3a, program PTE[0] to BO0 phys address (CPU)58 * alloc page level 2, program PDE[0] page level 3a phys address (CPU)59 * alloc page level 1, program PDE[0] page level 2 phys address (CPU)60 * update root PDE[0] to page level 1 phys address (GPU)61 *62 * bind BO1 0x201000-0x20200063 * alloc page level 3b, program PTE[1] to BO1 phys address (CPU)64 * update page level 2 PDE[1] to page level 3b phys address (GPU)65 *66 * bind BO2 0x1ff000-0x20100067 * update page level 3a PTE[511] to BO2 phys addres (GPU)68 * update page level 3b PTE[0] to BO2 phys addres + 0x1000 (GPU)69 *70 * GPU bypass71 * ~~~~~~~~~~72 *73 * In the above example the steps using the GPU can be converted to CPU if the74 * bind can be done immediately (all in-fences satisfied, VM dma-resv kernel75 * slot is idle).76 *77 * Address space78 * -------------79 *80 * Depending on platform either 48 or 57 bits of address space is supported.81 *82 * Page sizes83 * ----------84 *85 * The minimum page size is either 4k or 64k depending on platform and memory86 * placement (sysmem vs. VRAM). We enforce that binds must be aligned to the87 * minimum page size.88 *89 * Larger pages (2M or 1GB) can be used for BOs in VRAM, the BO physical address90 * is aligned to the larger pages size, and VA is aligned to the larger page91 * size. Larger pages for userptrs / BOs in sysmem should be possible but is not92 * yet implemented.93 *94 * Sync error handling mode95 * ------------------------96 *97 * In both modes during the bind IOCTL the user input is validated. In sync98 * error handling mode the newly bound BO is validated (potentially moved back99 * to a region of memory where is can be used), page tables are updated by the100 * CPU and the job to do the GPU binds is created in the IOCTL itself. This step101 * can fail due to memory pressure. The user can recover by freeing memory and102 * trying this operation again.103 *104 * Async error handling mode105 * -------------------------106 *107 * In async error handling the step of validating the BO, updating page tables,108 * and generating a job are deferred to an async worker. As this step can now109 * fail after the IOCTL has reported success we need an error handling flow for110 * which the user can recover from.111 *112 * The solution is for a user to register a user address with the VM which the113 * VM uses to report errors to. The ufence wait interface can be used to wait on114 * a VM going into an error state. Once an error is reported the VM's async115 * worker is paused. While the VM's async worker is paused sync,116 * DRM_XE_VM_BIND_OP_UNMAP operations are allowed (this can free memory). Once the117 * uses believe the error state is fixed, the async worker can be resumed via118 * XE_VM_BIND_OP_RESTART operation. When VM async bind work is restarted, the119 * first operation processed is the operation that caused the original error.120 *121 * Bind queues / engines122 * ---------------------123 *124 * Think of the case where we have two bind operations A + B and are submitted125 * in that order. A has in fences while B has none. If using a single bind126 * queue, B is now blocked on A's in fences even though it is ready to run. This127 * example is a real use case for VK sparse binding. We work around this128 * limitation by implementing bind engines.129 *130 * In the bind IOCTL the user can optionally pass in an engine ID which must map131 * to an engine which is of the special class DRM_XE_ENGINE_CLASS_VM_BIND.132 * Underneath this is a really virtual engine that can run on any of the copy133 * hardware engines. The job(s) created each IOCTL are inserted into this134 * engine's ring. In the example above if A and B have different bind engines B135 * is free to pass A. If the engine ID field is omitted, the default bind queue136 * for the VM is used.137 *138 * TODO: Explain race in issue 41 and how we solve it139 *140 * Array of bind operations141 * ------------------------142 *143 * The uAPI allows multiple binds operations to be passed in via a user array,144 * of struct drm_xe_vm_bind_op, in a single VM bind IOCTL. This interface145 * matches the VK sparse binding API. The implementation is rather simple, parse146 * the array into a list of operations, pass the in fences to the first operation,147 * and pass the out fences to the last operation. The ordered nature of a bind148 * engine makes this possible.149 *150 * Munmap semantics for unbinds151 * ----------------------------152 *153 * Munmap allows things like:154 *155 * .. code-block::156 *157 * 0x0000-0x2000 and 0x3000-0x5000 have mappings158 * Munmap 0x1000-0x4000, results in mappings 0x0000-0x1000 and 0x4000-0x5000159 *160 * To support this semantic in the above example we decompose the above example161 * into 4 operations:162 *163 * .. code-block::164 *165 * unbind 0x0000-0x2000166 * unbind 0x3000-0x5000167 * rebind 0x0000-0x1000168 * rebind 0x4000-0x5000169 *170 * Why not just do a partial unbind of 0x1000-0x2000 and 0x3000-0x4000? This171 * falls apart when using large pages at the edges and the unbind forces us to172 * use a smaller page size. For simplity we always issue a set of unbinds173 * unmapping anything in the range and at most 2 rebinds on the edges.174 *175 * Similar to an array of binds, in fences are passed to the first operation and176 * out fences are signaled on the last operation.177 *178 * In this example there is a window of time where 0x0000-0x1000 and179 * 0x4000-0x5000 are invalid but the user didn't ask for these addresses to be180 * removed from the mapping. To work around this we treat any munmap style181 * unbinds which require a rebind as a kernel operations (BO eviction or userptr182 * invalidation). The first operation waits on the VM's183 * DMA_RESV_USAGE_PREEMPT_FENCE slots (waits for all pending jobs on VM to184 * complete / triggers preempt fences) and the last operation is installed in185 * the VM's DMA_RESV_USAGE_KERNEL slot (blocks future jobs / resume compute mode186 * VM). The caveat is all dma-resv slots must be updated atomically with respect187 * to execs and compute mode rebind worker. To accomplish this, hold the188 * vm->lock in write mode from the first operation until the last.189 *190 * Deferred binds in fault mode191 * ----------------------------192 *193 * If a VM is in fault mode (TODO: link to fault mode), new bind operations that194 * create mappings are by default deferred to the page fault handler (first195 * use). This behavior can be overriden by setting the flag196 * DRM_XE_VM_BIND_FLAG_IMMEDIATE which indicates to creating the mapping197 * immediately.198 *199 * User pointer200 * ============201 *202 * User pointers are user allocated memory (malloc'd, mmap'd, etc..) for which the203 * user wants to create a GPU mapping. Typically in other DRM drivers a dummy BO204 * was created and then a binding was created. We bypass creating a dummy BO in205 * XE and simply create a binding directly from the userptr.206 *207 * Invalidation208 * ------------209 *210 * Since this a core kernel managed memory the kernel can move this memory211 * whenever it wants. We register an invalidation MMU notifier to alert XE when212 * a user poiter is about to move. The invalidation notifier needs to block213 * until all pending users (jobs or compute mode engines) of the userptr are214 * idle to ensure no faults. This done by waiting on all of VM's dma-resv slots.215 *216 * Rebinds217 * -------218 *219 * Either the next exec (non-compute) or rebind worker (compute mode) will220 * rebind the userptr. The invalidation MMU notifier kicks the rebind worker221 * after the VM dma-resv wait if the VM is in compute mode.222 *223 * Compute mode224 * ============225 *226 * A VM in compute mode enables long running workloads and ultra low latency227 * submission (ULLS). ULLS is implemented via a continuously running batch +228 * semaphores. This enables the user to insert jump to new batch commands229 * into the continuously running batch. In both cases these batches exceed the230 * time a dma fence is allowed to exist for before signaling, as such dma fences231 * are not used when a VM is in compute mode. User fences (TODO: link user fence232 * doc) are used instead to signal operation's completion.233 *234 * Preempt fences235 * --------------236 *237 * If the kernel decides to move memory around (either userptr invalidate, BO238 * eviction, or mumap style unbind which results in a rebind) and a batch is239 * running on an engine, that batch can fault or cause a memory corruption as240 * page tables for the moved memory are no longer valid. To work around this we241 * introduce the concept of preempt fences. When sw signaling is enabled on a242 * preempt fence it tells the submission backend to kick that engine off the243 * hardware and the preempt fence signals when the engine is off the hardware.244 * Once all preempt fences are signaled for a VM the kernel can safely move the245 * memory and kick the rebind worker which resumes all the engines execution.246 *247 * A preempt fence, for every engine using the VM, is installed into the VM's248 * dma-resv DMA_RESV_USAGE_PREEMPT_FENCE slot. The same preempt fence, for every249 * engine using the VM, is also installed into the same dma-resv slot of every250 * external BO mapped in the VM.251 *252 * Rebind worker253 * -------------254 *255 * The rebind worker is very similar to an exec. It is resposible for rebinding256 * evicted BOs or userptrs, waiting on those operations, installing new preempt257 * fences, and finally resuming executing of engines in the VM.258 *259 * Flow260 * ~~~~261 *262 * .. code-block::263 *264 * <----------------------------------------------------------------------|265 * Check if VM is closed, if so bail out |266 * Lock VM global lock in read mode |267 * Pin userptrs (also finds userptr invalidated since last rebind worker) |268 * Lock VM dma-resv and external BOs dma-resv |269 * Validate BOs that have been evicted |270 * Wait on and allocate new preempt fences for every engine using the VM |271 * Rebind invalidated userptrs + evicted BOs |272 * Wait on last rebind fence |273 * Wait VM's DMA_RESV_USAGE_KERNEL dma-resv slot |274 * Install preeempt fences and issue resume for every engine using the VM |275 * Check if any userptrs invalidated since pin |276 * Squash resume for all engines |277 * Unlock all |278 * Wait all VM's dma-resv slots |279 * Retry ----------------------------------------------------------280 * Release all engines waiting to resume281 * Unlock all282 *283 * Timeslicing284 * -----------285 *286 * In order to prevent an engine from continuously being kicked off the hardware287 * and making no forward progress an engine has a period of time it allowed to288 * run after resume before it can be kicked off again. This effectively gives289 * each engine a timeslice.290 *291 * Handling multiple GTs292 * =====================293 *294 * If a GT has slower access to some regions and the page table structure are in295 * the slow region, the performance on that GT could adversely be affected. To296 * work around this we allow a VM page tables to be shadowed in multiple GTs.297 * When VM is created, a default bind engine and PT table structure are created298 * on each GT.299 *300 * Binds can optionally pass in a mask of GTs where a mapping should be created,301 * if this mask is zero then default to all the GTs where the VM has page302 * tables.303 *304 * The implementation for this breaks down into a bunch for_each_gt loops in305 * various places plus exporting a composite fence for multi-GT binds to the306 * user.307 *308 * Fault mode (unified shared memory)309 * ==================================310 *311 * A VM in fault mode can be enabled on devices that support page faults. If312 * page faults are enabled, using dma fences can potentially induce a deadlock:313 * A pending page fault can hold up the GPU work which holds up the dma fence314 * signaling, and memory allocation is usually required to resolve a page315 * fault, but memory allocation is not allowed to gate dma fence signaling. As316 * such, dma fences are not allowed when VM is in fault mode. Because dma-fences317 * are not allowed, only long running workloads and ULLS are enabled on a faulting318 * VM.319 *320 * Defered VM binds321 * ----------------322 *323 * By default, on a faulting VM binds just allocate the VMA and the actual324 * updating of the page tables is defered to the page fault handler. This325 * behavior can be overridden by setting the flag DRM_XE_VM_BIND_FLAG_IMMEDIATE in326 * the VM bind which will then do the bind immediately.327 *328 * Page fault handler329 * ------------------330 *331 * Page faults are received in the G2H worker under the CT lock which is in the332 * path of dma fences (no memory allocations are allowed, faults require memory333 * allocations) thus we cannot process faults under the CT lock. Another issue334 * is faults issue TLB invalidations which require G2H credits and we cannot335 * allocate G2H credits in the G2H handlers without deadlocking. Lastly, we do336 * not want the CT lock to be an outer lock of the VM global lock (VM global337 * lock required to fault processing).338 *339 * To work around the above issue with processing faults in the G2H worker, we340 * sink faults to a buffer which is large enough to sink all possible faults on341 * the GT (1 per hardware engine) and kick a worker to process the faults. Since342 * the page faults G2H are already received in a worker, kicking another worker343 * adds more latency to a critical performance path. We add a fast path in the344 * G2H irq handler which looks at first G2H and if it is a page fault we sink345 * the fault to the buffer and kick the worker to process the fault. TLB346 * invalidation responses are also in the critical path so these can also be347 * processed in this fast path.348 *349 * Multiple buffers and workers are used and hashed over based on the ASID so350 * faults from different VMs can be processed in parallel.351 *352 * The page fault handler itself is rather simple, flow is below.353 *354 * .. code-block::355 *356 * Lookup VM from ASID in page fault G2H357 * Lock VM global lock in read mode358 * Lookup VMA from address in page fault G2H359 * Check if VMA is valid, if not bail360 * Check if VMA's BO has backing store, if not allocate361 * <----------------------------------------------------------------------|362 * If userptr, pin pages |363 * Lock VM & BO dma-resv locks |364 * If atomic fault, migrate to VRAM, else validate BO location |365 * Issue rebind |366 * Wait on rebind to complete |367 * Check if userptr invalidated since pin |368 * Drop VM & BO dma-resv locks |369 * Retry ----------------------------------------------------------370 * Unlock all371 * Issue blocking TLB invalidation |372 * Send page fault response to GuC373 *374 * Access counters375 * ---------------376 *377 * Access counters can be configured to trigger a G2H indicating the device is378 * accessing VMAs in system memory frequently as hint to migrate those VMAs to379 * VRAM.380 *381 * Same as the page fault handler, access counters G2H cannot be processed the382 * G2H worker under the CT lock. Again we use a buffer to sink access counter383 * G2H. Unlike page faults there is no upper bound so if the buffer is full we384 * simply drop the G2H. Access counters are a best case optimization and it is385 * safe to drop these unlike page faults.386 *387 * The access counter handler itself is rather simple flow is below.388 *389 * .. code-block::390 *391 * Lookup VM from ASID in access counter G2H392 * Lock VM global lock in read mode393 * Lookup VMA from address in access counter G2H394 * If userptr, bail nothing to do395 * Lock VM & BO dma-resv locks396 * Issue migration to VRAM397 * Unlock all398 *399 * Notice no rebind is issued in the access counter handler as the rebind will400 * be issued on next page fault.401 *402 * Caveats with eviction / user pointer invalidation403 * -------------------------------------------------404 *405 * In the case of eviction and user pointer invalidation on a faulting VM, there406 * is no need to issue a rebind rather we just need to blow away the page tables407 * for the VMAs and the page fault handler will rebind the VMAs when they fault.408 * The caveat is to update / read the page table structure the VM global lock is409 * needed. In both the case of eviction and user pointer invalidation locks are410 * held which make acquiring the VM global lock impossible. To work around this411 * every VMA maintains a list of leaf page table entries which should be written412 * to zero to blow away the VMA's page tables. After writing zero to these413 * entries a blocking TLB invalidate is issued. At this point it is safe for the414 * kernel to move the VMA's memory around. This is a necessary lockless415 * algorithm and is safe as leafs cannot be changed while either an eviction or416 * userptr invalidation is occurring.417 *418 * Locking419 * =======420 *421 * VM locking protects all of the core data paths (bind operations, execs,422 * evictions, and compute mode rebind worker) in XE.423 *424 * Locks425 * -----426 *427 * VM global lock (vm->lock) - rw semaphore lock. Outer most lock which protects428 * the list of userptrs mapped in the VM, the list of engines using this VM, and429 * the array of external BOs mapped in the VM. When adding or removing any of the430 * aforementioned state from the VM should acquire this lock in write mode. The VM431 * bind path also acquires this lock in write while the exec / compute mode432 * rebind worker acquires this lock in read mode.433 *434 * VM dma-resv lock (vm->ttm.base.resv->lock) - WW lock. Protects VM dma-resv435 * slots which is shared with any private BO in the VM. Expected to be acquired436 * during VM binds, execs, and compute mode rebind worker. This lock is also437 * held when private BOs are being evicted.438 *439 * external BO dma-resv lock (bo->ttm.base.resv->lock) - WW lock. Protects440 * external BO dma-resv slots. Expected to be acquired during VM binds (in441 * addition to the VM dma-resv lock). All external BO dma-locks within a VM are442 * expected to be acquired (in addition to the VM dma-resv lock) during execs443 * and the compute mode rebind worker. This lock is also held when an external444 * BO is being evicted.445 *446 * Putting it all together447 * -----------------------448 *449 * 1. An exec and bind operation with the same VM can't be executing at the same450 * time (vm->lock).451 *452 * 2. A compute mode rebind worker and bind operation with the same VM can't be453 * executing at the same time (vm->lock).454 *455 * 3. We can't add / remove userptrs or external BOs to a VM while an exec with456 * the same VM is executing (vm->lock).457 *458 * 4. We can't add / remove userptrs, external BOs, or engines to a VM while a459 * compute mode rebind worker with the same VM is executing (vm->lock).460 *461 * 5. Evictions within a VM can't be happen while an exec with the same VM is462 * executing (dma-resv locks).463 *464 * 6. Evictions within a VM can't be happen while a compute mode rebind worker465 * with the same VM is executing (dma-resv locks).466 *467 * dma-resv usage468 * ==============469 *470 * As previously stated to enforce the ordering of kernel ops (eviction, userptr471 * invalidation, munmap style unbinds which result in a rebind), rebinds during472 * execs, execs, and resumes in the rebind worker we use both the VMs and473 * external BOs dma-resv slots. Let try to make this as clear as possible.474 *475 * Slot installation476 * -----------------477 *478 * 1. Jobs from kernel ops install themselves into the DMA_RESV_USAGE_KERNEL479 * slot of either an external BO or VM (depends on if kernel op is operating on480 * an external or private BO)481 *482 * 2. In non-compute mode, jobs from execs install themselves into the483 * DMA_RESV_USAGE_BOOKKEEP slot of the VM484 *485 * 3. In non-compute mode, jobs from execs install themselves into the486 * DMA_RESV_USAGE_WRITE slot of all external BOs in the VM487 *488 * 4. Jobs from binds install themselves into the DMA_RESV_USAGE_BOOKKEEP slot489 * of the VM490 *491 * 5. Jobs from binds install themselves into the DMA_RESV_USAGE_BOOKKEEP slot492 * of the external BO (if the bind is to an external BO, this is addition to #4)493 *494 * 6. Every engine using a compute mode VM has a preempt fence in installed into495 * the DMA_RESV_USAGE_PREEMPT_FENCE slot of the VM496 *497 * 7. Every engine using a compute mode VM has a preempt fence in installed into498 * the DMA_RESV_USAGE_PREEMPT_FENCE slot of all the external BOs in the VM499 *500 * Slot waiting501 * ------------502 *503 * 1. The exection of all jobs from kernel ops shall wait on all slots504 * (DMA_RESV_USAGE_PREEMPT_FENCE) of either an external BO or VM (depends on if505 * kernel op is operating on external or private BO)506 *507 * 2. In non-compute mode, the exection of all jobs from rebinds in execs shall508 * wait on the DMA_RESV_USAGE_KERNEL slot of either an external BO or VM509 * (depends on if the rebind is operatiing on an external or private BO)510 *511 * 3. In non-compute mode, the exection of all jobs from execs shall wait on the512 * last rebind job513 *514 * 4. In compute mode, the exection of all jobs from rebinds in the rebind515 * worker shall wait on the DMA_RESV_USAGE_KERNEL slot of either an external BO516 * or VM (depends on if rebind is operating on external or private BO)517 *518 * 5. In compute mode, resumes in rebind worker shall wait on last rebind fence519 *520 * 6. In compute mode, resumes in rebind worker shall wait on the521 * DMA_RESV_USAGE_KERNEL slot of the VM522 *523 * Putting it all together524 * -----------------------525 *526 * 1. New jobs from kernel ops are blocked behind any existing jobs from527 * non-compute mode execs528 *529 * 2. New jobs from non-compute mode execs are blocked behind any existing jobs530 * from kernel ops and rebinds531 *532 * 3. New jobs from kernel ops are blocked behind all preempt fences signaling in533 * compute mode534 *535 * 4. Compute mode engine resumes are blocked behind any existing jobs from536 * kernel ops and rebinds537 *538 * Future work539 * ===========540 *541 * Support large pages for sysmem and userptr.542 *543 * Update page faults to handle BOs are page level grainularity (e.g. part of BO544 * could be in system memory while another part could be in VRAM).545 *546 * Page fault handler likely we be optimized a bit more (e.g. Rebinds always547 * wait on the dma-resv kernel slots of VM or BO, technically we only have to548 * wait the BO moving. If using a job to do the rebind, we could not block in549 * the page fault handler rather attach a callback to fence of the rebind job to550 * signal page fault complete. Our handling of short circuting for atomic faults551 * for bound VMAs could be better. etc...). We can tune all of this once we have552 * benchmarks / performance number from workloads up and running.553 */554 555#endif556