274 lines · plain
1.. SPDX-License-Identifier: GPL-2.0-only2.. include:: <isonum.txt>3 4=====================5VFIO Mediated devices6=====================7 8:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.9:Author: Neo Jia <cjia@nvidia.com>10:Author: Kirti Wankhede <kwankhede@nvidia.com>11 12 13 14Virtual Function I/O (VFIO) Mediated devices[1]15===============================================16 17The number of use cases for virtualizing DMA devices that do not have built-in18SR_IOV capability is increasing. Previously, to virtualize such devices,19developers had to create their own management interfaces and APIs, and then20integrate them with user space software. To simplify integration with user space21software, we have identified common requirements and a unified management22interface for such devices.23 24The VFIO driver framework provides unified APIs for direct device access. It is25an IOMMU/device-agnostic framework for exposing direct device access to user26space in a secure, IOMMU-protected environment. This framework is used for27multiple devices, such as GPUs, network adapters, and compute accelerators. With28direct device access, virtual machines or user space applications have direct29access to the physical device. This framework is reused for mediated devices.30 31The mediated core driver provides a common interface for mediated device32management that can be used by drivers of different devices. This module33provides a generic interface to perform these operations:34 35* Create and destroy a mediated device36* Add a mediated device to and remove it from a mediated bus driver37* Add a mediated device to and remove it from an IOMMU group38 39The mediated core driver also provides an interface to register a bus driver.40For example, the mediated VFIO mdev driver is designed for mediated devices and41supports VFIO APIs. The mediated bus driver adds a mediated device to and42removes it from a VFIO group.43 44The following high-level block diagram shows the main components and interfaces45in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM46devices as examples, as these devices are the first devices to use this module::47 48 +---------------+49 | |50 | +-----------+ | mdev_register_driver() +--------------+51 | | | +<------------------------+ |52 | | mdev | | | |53 | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user54 | | driver | | probe()/remove() | | APIs55 | | | | +--------------+56 | +-----------+ |57 | |58 | MDEV CORE |59 | MODULE |60 | mdev.ko |61 | +-----------+ | mdev_register_parent() +--------------+62 | | | +<------------------------+ |63 | | | | | ccw_device.ko|<-> physical64 | | | +------------------------>+ | device65 | | | | callbacks +--------------+66 | | Physical | |67 | | device | | mdev_register_parent() +--------------+68 | | interface | |<------------------------+ |69 | | | | | i915.ko |<-> physical70 | | | +------------------------>+ | device71 | | | | callbacks +--------------+72 | +-----------+ |73 +---------------+74 75 76Registration Interfaces77=======================78 79The mediated core driver provides the following types of registration80interfaces:81 82* Registration interface for a mediated bus driver83* Physical device driver interface84 85Registration Interface for a Mediated Bus Driver86------------------------------------------------87 88The registration interface for a mediated device driver provides the following89structure to represent a mediated device's driver::90 91 /*92 * struct mdev_driver [2] - Mediated device's driver93 * @probe: called when new device created94 * @remove: called when device removed95 * @driver: device driver structure96 */97 struct mdev_driver {98 int (*probe) (struct mdev_device *dev);99 void (*remove) (struct mdev_device *dev);100 unsigned int (*get_available)(struct mdev_type *mtype);101 ssize_t (*show_description)(struct mdev_type *mtype, char *buf);102 struct device_driver driver;103 };104 105A mediated bus driver for mdev should use this structure in the function calls106to register and unregister itself with the core driver:107 108* Register::109 110 int mdev_register_driver(struct mdev_driver *drv);111 112* Unregister::113 114 void mdev_unregister_driver(struct mdev_driver *drv);115 116The mediated bus driver's probe function should create a vfio_device on top of117the mdev_device and connect it to an appropriate implementation of118vfio_device_ops.119 120When a driver wants to add the GUID creation sysfs to an existing device it has121probe'd to then it should call::122 123 int mdev_register_parent(struct mdev_parent *parent, struct device *dev,124 struct mdev_driver *mdev_driver);125 126This will provide the 'mdev_supported_types/XX/create' files which can then be127used to trigger the creation of a mdev_device. The created mdev_device will be128attached to the specified driver.129 130When the driver needs to remove itself it calls::131 132 void mdev_unregister_parent(struct mdev_parent *parent);133 134Which will unbind and destroy all the created mdevs and remove the sysfs files.135 136Mediated Device Management Interface Through sysfs137==================================================138 139The management interface through sysfs enables user space software, such as140libvirt, to query and configure mediated devices in a hardware-agnostic fashion.141This management interface provides flexibility to the underlying physical142device's driver to support features such as:143 144* Mediated device hot plug145* Multiple mediated devices in a single virtual machine146* Multiple mediated devices from different physical devices147 148Links in the mdev_bus Class Directory149-------------------------------------150The /sys/class/mdev_bus/ directory contains links to devices that are registered151with the mdev core driver.152 153Directories and files under the sysfs for Each Physical Device154--------------------------------------------------------------155 156::157 158 |- [parent physical device]159 |--- Vendor-specific-attributes [optional]160 |--- [mdev_supported_types]161 | |--- [<type-id>]162 | | |--- create163 | | |--- name164 | | |--- available_instances165 | | |--- device_api166 | | |--- description167 | | |--- [devices]168 | |--- [<type-id>]169 | | |--- create170 | | |--- name171 | | |--- available_instances172 | | |--- device_api173 | | |--- description174 | | |--- [devices]175 | |--- [<type-id>]176 | |--- create177 | |--- name178 | |--- available_instances179 | |--- device_api180 | |--- description181 | |--- [devices]182 183* [mdev_supported_types]184 185 The list of currently supported mediated device types and their details.186 187 [<type-id>], device_api, and available_instances are mandatory attributes188 that should be provided by vendor driver.189 190* [<type-id>]191 192 The [<type-id>] name is created by adding the device driver string as a prefix193 to the string provided by the vendor driver. This format of this name is as194 follows::195 196 sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);197 198* device_api199 200 This attribute shows which device API is being created, for example,201 "vfio-pci" for a PCI device.202 203* available_instances204 205 This attribute shows the number of devices of type <type-id> that can be206 created.207 208* [device]209 210 This directory contains links to the devices of type <type-id> that have been211 created.212 213* name214 215 This attribute shows a human readable name.216 217* description218 219 This attribute can show brief features/description of the type. This is an220 optional attribute.221 222Directories and Files Under the sysfs for Each mdev Device223----------------------------------------------------------224 225::226 227 |- [parent phy device]228 |--- [$MDEV_UUID]229 |--- remove230 |--- mdev_type {link to its type}231 |--- vendor-specific-attributes [optional]232 233* remove (write only)234 235Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can236fail the remove() callback if that device is active and the vendor driver237doesn't support hot unplug.238 239Example::240 241 # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove242 243Mediated device Hot plug244------------------------245 246Mediated devices can be created and assigned at runtime. The procedure to hot247plug a mediated device is the same as the procedure to hot plug a PCI device.248 249Translation APIs for Mediated Devices250=====================================251 252The following APIs are provided for translating user pfn to host pfn in a VFIO253driver::254 255 int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,256 int npage, int prot, struct page **pages);257 258 void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,259 int npage);260 261These functions call back into the back-end IOMMU module by using the pin_pages262and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently263these callbacks are supported in the TYPE1 IOMMU module. To enable them for264other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide265these two callback functions.266 267References268==========269 2701. See Documentation/driver-api/vfio.rst for more information on VFIO.2712. struct mdev_driver in include/linux/mdev.h2723. struct mdev_parent_ops in include/linux/mdev.h2734. struct vfio_iommu_driver_ops in include/linux/vfio.h274