287 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3===========================================4Shared Virtual Addressing (SVA) with ENQCMD5===========================================6 7Background8==========9 10Shared Virtual Addressing (SVA) allows the processor and device to use the11same virtual addresses avoiding the need for software to translate virtual12addresses to physical addresses. SVA is what PCIe calls Shared Virtual13Memory (SVM).14 15In addition to the convenience of using application virtual addresses16by the device, it also doesn't require pinning pages for DMA.17PCIe Address Translation Services (ATS) along with Page Request Interface18(PRI) allow devices to function much the same way as the CPU handling19application page-faults. For more information please refer to the PCIe20specification Chapter 10: ATS Specification.21 22Use of SVA requires IOMMU support in the platform. IOMMU is also23required to support the PCIe features ATS and PRI. ATS allows devices24to cache translations for virtual addresses. The IOMMU driver uses the25mmu_notifier() support to keep the device TLB cache and the CPU cache in26sync. When an ATS lookup fails for a virtual address, the device should27use the PRI in order to request the virtual address to be paged into the28CPU page tables. The device must use ATS again in order the fetch the29translation before use.30 31Shared Hardware Workqueues32==========================33 34Unlike Single Root I/O Virtualization (SR-IOV), Scalable IOV (SIOV) permits35the use of Shared Work Queues (SWQ) by both applications and Virtual36Machines (VM's). This allows better hardware utilization vs. hard37partitioning resources that could result in under utilization. In order to38allow the hardware to distinguish the context for which work is being39executed in the hardware by SWQ interface, SIOV uses Process Address Space40ID (PASID), which is a 20-bit number defined by the PCIe SIG.41 42PASID value is encoded in all transactions from the device. This allows the43IOMMU to track I/O on a per-PASID granularity in addition to using the PCIe44Resource Identifier (RID) which is the Bus/Device/Function.45 46 47ENQCMD48======49 50ENQCMD is a new instruction on Intel platforms that atomically submits a51work descriptor to a device. The descriptor includes the operation to be52performed, virtual addresses of all parameters, virtual address of a completion53record, and the PASID (process address space ID) of the current process.54 55ENQCMD works with non-posted semantics and carries a status back if the56command was accepted by hardware. This allows the submitter to know if the57submission needs to be retried or other device specific mechanisms to58implement fairness or ensure forward progress should be provided.59 60ENQCMD is the glue that ensures applications can directly submit commands61to the hardware and also permits hardware to be aware of application context62to perform I/O operations via use of PASID.63 64Process Address Space Tagging65=============================66 67A new thread-scoped MSR (IA32_PASID) provides the connection between68user processes and the rest of the hardware. When an application first69accesses an SVA-capable device, this MSR is initialized with a newly70allocated PASID. The driver for the device calls an IOMMU-specific API71that sets up the routing for DMA and page-requests.72 73For example, the Intel Data Streaming Accelerator (DSA) uses74iommu_sva_bind_device(), which will do the following:75 76- Allocate the PASID, and program the process page-table (%cr3 register) in the77 PASID context entries.78- Register for mmu_notifier() to track any page-table invalidations to keep79 the device TLB in sync. For example, when a page-table entry is invalidated,80 the IOMMU propagates the invalidation to the device TLB. This will force any81 future access by the device to this virtual address to participate in82 ATS. If the IOMMU responds with proper response that a page is not83 present, the device would request the page to be paged in via the PCIe PRI84 protocol before performing I/O.85 86This MSR is managed with the XSAVE feature set as "supervisor state" to87ensure the MSR is updated during context switch.88 89PASID Management90================91 92The kernel must allocate a PASID on behalf of each process which will use93ENQCMD and program it into the new MSR to communicate the process identity to94platform hardware. ENQCMD uses the PASID stored in this MSR to tag requests95from this process. When a user submits a work descriptor to a device using the96ENQCMD instruction, the PASID field in the descriptor is auto-filled with the97value from MSR_IA32_PASID. Requests for DMA from the device are also tagged98with the same PASID. The platform IOMMU uses the PASID in the transaction to99perform address translation. The IOMMU APIs setup the corresponding PASID100entry in IOMMU with the process address used by the CPU (e.g. %cr3 register in101x86).102 103The MSR must be configured on each logical CPU before any application104thread can interact with a device. Threads that belong to the same105process share the same page tables, thus the same MSR value.106 107PASID Life Cycle Management108===========================109 110PASID is initialized as IOMMU_PASID_INVALID (-1) when a process is created.111 112Only processes that access SVA-capable devices need to have a PASID113allocated. This allocation happens when a process opens/binds an SVA-capable114device but finds no PASID for this process. Subsequent binds of the same, or115other devices will share the same PASID.116 117Although the PASID is allocated to the process by opening a device,118it is not active in any of the threads of that process. It's loaded to the119IA32_PASID MSR lazily when a thread tries to submit a work descriptor120to a device using the ENQCMD.121 122That first access will trigger a #GP fault because the IA32_PASID MSR123has not been initialized with the PASID value assigned to the process124when the device was opened. The Linux #GP handler notes that a PASID has125been allocated for the process, and so initializes the IA32_PASID MSR126and returns so that the ENQCMD instruction is re-executed.127 128On fork(2) or exec(2) the PASID is removed from the process as it no129longer has the same address space that it had when the device was opened.130 131On clone(2) the new task shares the same address space, so will be132able to use the PASID allocated to the process. The IA32_PASID is not133preemptively initialized as the PASID value might not be allocated yet or134the kernel does not know whether this thread is going to access the device135and the cleared IA32_PASID MSR reduces context switch overhead by xstate136init optimization. Since #GP faults have to be handled on any threads that137were created before the PASID was assigned to the mm of the process, newly138created threads might as well be treated in a consistent way.139 140Due to complexity of freeing the PASID and clearing all IA32_PASID MSRs in141all threads in unbind, free the PASID lazily only on mm exit.142 143If a process does a close(2) of the device file descriptor and munmap(2)144of the device MMIO portal, then the driver will unbind the device. The145PASID is still marked VALID in the PASID_MSR for any threads in the146process that accessed the device. But this is harmless as without the147MMIO portal they cannot submit new work to the device.148 149Relationships150=============151 152 * Each process has many threads, but only one PASID.153 * Devices have a limited number (~10's to 1000's) of hardware workqueues.154 The device driver manages allocating hardware workqueues.155 * A single mmap() maps a single hardware workqueue as a "portal" and156 each portal maps down to a single workqueue.157 * For each device with which a process interacts, there must be158 one or more mmap()'d portals.159 * Many threads within a process can share a single portal to access160 a single device.161 * Multiple processes can separately mmap() the same portal, in162 which case they still share one device hardware workqueue.163 * The single process-wide PASID is used by all threads to interact164 with all devices. There is not, for instance, a PASID for each165 thread or each thread<->device pair.166 167FAQ168===169 170* What is SVA/SVM?171 172Shared Virtual Addressing (SVA) permits I/O hardware and the processor to173work in the same address space, i.e., to share it. Some call it Shared174Virtual Memory (SVM), but Linux community wanted to avoid confusing it with175POSIX Shared Memory and Secure Virtual Machines which were terms already in176circulation.177 178* What is a PASID?179 180A Process Address Space ID (PASID) is a PCIe-defined Transaction Layer Packet181(TLP) prefix. A PASID is a 20-bit number allocated and managed by the OS.182PASID is included in all transactions between the platform and the device.183 184* How are shared workqueues different?185 186Traditionally, in order for userspace applications to interact with hardware,187there is a separate hardware instance required per process. For example,188consider doorbells as a mechanism of informing hardware about work to process.189Each doorbell is required to be spaced 4k (or page-size) apart for process190isolation. This requires hardware to provision that space and reserve it in191MMIO. This doesn't scale as the number of threads becomes quite large. The192hardware also manages the queue depth for Shared Work Queues (SWQ), and193consumers don't need to track queue depth. If there is no space to accept194a command, the device will return an error indicating retry.195 196A user should check Deferrable Memory Write (DMWr) capability on the device197and only submits ENQCMD when the device supports it. In the new DMWr PCIe198terminology, devices need to support DMWr completer capability. In addition,199it requires all switch ports to support DMWr routing and must be enabled by200the PCIe subsystem, much like how PCIe atomic operations are managed for201instance.202 203SWQ allows hardware to provision just a single address in the device. When204used with ENQCMD to submit work, the device can distinguish the process205submitting the work since it will include the PASID assigned to that206process. This helps the device scale to a large number of processes.207 208* Is this the same as a user space device driver?209 210Communicating with the device via the shared workqueue is much simpler211than a full blown user space driver. The kernel driver does all the212initialization of the hardware. User space only needs to worry about213submitting work and processing completions.214 215* Is this the same as SR-IOV?216 217Single Root I/O Virtualization (SR-IOV) focuses on providing independent218hardware interfaces for virtualizing hardware. Hence, it's required to be219almost fully functional interface to software supporting the traditional220BARs, space for interrupts via MSI-X, its own register layout.221Virtual Functions (VFs) are assisted by the Physical Function (PF)222driver.223 224Scalable I/O Virtualization builds on the PASID concept to create device225instances for virtualization. SIOV requires host software to assist in226creating virtual devices; each virtual device is represented by a PASID227along with the bus/device/function of the device. This allows device228hardware to optimize device resource creation and can grow dynamically on229demand. SR-IOV creation and management is very static in nature. Consult230references below for more details.231 232* Why not just create a virtual function for each app?233 234Creating PCIe SR-IOV type Virtual Functions (VF) is expensive. VFs require235duplicated hardware for PCI config space and interrupts such as MSI-X.236Resources such as interrupts have to be hard partitioned between VFs at237creation time, and cannot scale dynamically on demand. The VFs are not238completely independent from the Physical Function (PF). Most VFs require239some communication and assistance from the PF driver. SIOV, in contrast,240creates a software-defined device where all the configuration and control241aspects are mediated via the slow path. The work submission and completion242happen without any mediation.243 244* Does this support virtualization?245 246ENQCMD can be used from within a guest VM. In these cases, the VMM helps247with setting up a translation table to translate from Guest PASID to Host248PASID. Please consult the ENQCMD instruction set reference for more249details.250 251* Does memory need to be pinned?252 253When devices support SVA along with platform hardware such as IOMMU254supporting such devices, there is no need to pin memory for DMA purposes.255Devices that support SVA also support other PCIe features that remove the256pinning requirement for memory.257 258Device TLB support - Device requests the IOMMU to lookup an address before259use via Address Translation Service (ATS) requests. If the mapping exists260but there is no page allocated by the OS, IOMMU hardware returns that no261mapping exists.262 263Device requests the virtual address to be mapped via Page Request264Interface (PRI). Once the OS has successfully completed the mapping, it265returns the response back to the device. The device requests again for266a translation and continues.267 268IOMMU works with the OS in managing consistency of page-tables with the269device. When removing pages, it interacts with the device to remove any270device TLB entry that might have been cached before removing the mappings from271the OS.272 273References274==========275 276VT-D:277https://01.org/blogs/ashokraj/2018/recent-enhancements-intel-virtualization-technology-directed-i/o-intel-vt-d278 279SIOV:280https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux281 282ENQCMD in ISE:283https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf284 285DSA spec:286https://software.intel.com/sites/default/files/341204-intel-data-streaming-accelerator-spec.pdf287