1130 lines · plain
1===============================2Adjunct Processor (AP) facility3===============================4 5 6Introduction7============8The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised9of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.10The AP devices provide cryptographic functions to all CPUs assigned to a11linux system running in an IBM Z system LPAR.12 13The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap14is to make AP cards available to KVM guests using the VFIO mediated device15framework. This implementation relies considerably on the s390 virtualization16facilities which do most of the hard work of providing direct access to AP17devices.18 19AP Architectural Overview20=========================21To facilitate the comprehension of the design, let's start with some22definitions:23 24* AP adapter25 26 An AP adapter is an IBM Z adapter card that can perform cryptographic27 functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters28 assigned to the LPAR in which a linux host is running will be available to29 the linux host. Each adapter is identified by a number from 0 to 255; however,30 the maximum adapter number is determined by machine model and/or adapter type.31 When installed, an AP adapter is accessed by AP instructions executed by any32 CPU.33 34 The AP adapter cards are assigned to a given LPAR via the system's Activation35 Profile which can be edited via the HMC. When the linux host system is IPL'd36 in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and37 creates a sysfs device for each assigned adapter. For example, if AP adapters38 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following39 sysfs device entries::40 41 /sys/devices/ap/card0442 /sys/devices/ap/card0a43 44 Symbolic links to these devices will also be created in the AP bus devices45 sub-directory::46 47 /sys/bus/ap/devices/[card04]48 /sys/bus/ap/devices/[card04]49 50* AP domain51 52 An adapter is partitioned into domains. An adapter can hold up to 256 domains53 depending upon the adapter type and hardware configuration. A domain is54 identified by a number from 0 to 255; however, the maximum domain number is55 determined by machine model and/or adapter type.. A domain can be thought of56 as a set of hardware registers and memory used for processing AP commands. A57 domain can be configured with a secure private key used for clear key58 encryption. A domain is classified in one of two ways depending upon how it59 may be accessed:60 61 * Usage domains are domains that are targeted by an AP instruction to62 process an AP command.63 64 * Control domains are domains that are changed by an AP command sent to a65 usage domain; for example, to set the secure private key for the control66 domain.67 68 The AP usage and control domains are assigned to a given LPAR via the system's69 Activation Profile which can be edited via the HMC. When a linux host system70 is IPL'd in the LPAR, the AP bus module detects the AP usage and control71 domains assigned to the LPAR. The domain number of each usage domain and72 adapter number of each AP adapter are combined to create AP queue devices73 (see AP Queue section below). The domain number of each control domain will be74 represented in a bitmask and stored in a sysfs file75 /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least76 significant bit, correspond to domains 0-255.77 78* AP Queue79 80 An AP queue is the means by which an AP command is sent to a usage domain81 inside a specific adapter. An AP queue is identified by a tuple82 comprised of an AP adapter ID (APID) and an AP queue index (APQI). The83 APQI corresponds to a given usage domain number within the adapter. This tuple84 forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP85 instructions include a field containing the APQN to identify the AP queue to86 which the AP command is to be sent for processing.87 88 The AP bus will create a sysfs device for each APQN that can be derived from89 the cross product of the AP adapter and usage domain numbers detected when the90 AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage91 domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the92 following sysfs entries::93 94 /sys/devices/ap/card04/04.000695 /sys/devices/ap/card04/04.004796 /sys/devices/ap/card0a/0a.000697 /sys/devices/ap/card0a/0a.004798 99 The following symbolic links to these devices will be created in the AP bus100 devices subdirectory::101 102 /sys/bus/ap/devices/[04.0006]103 /sys/bus/ap/devices/[04.0047]104 /sys/bus/ap/devices/[0a.0006]105 /sys/bus/ap/devices/[0a.0047]106 107* AP Instructions:108 109 There are three AP instructions:110 111 * NQAP: to enqueue an AP command-request message to a queue112 * DQAP: to dequeue an AP command-reply message from a queue113 * PQAP: to administer the queues114 115 AP instructions identify the domain that is targeted to process the AP116 command; this must be one of the usage domains. An AP command may modify a117 domain that is not one of the usage domains, but the modified domain118 must be one of the control domains.119 120AP and SIE121==========122Let's now take a look at how AP instructions executed on a guest are interpreted123by the hardware.124 125A satellite control block called the Crypto Control Block (CRYCB) is attached to126our main hardware virtualization control block. The CRYCB contains an AP Control127Block (APCB) that has three fields to identify the adapters, usage domains and128control domains assigned to the KVM guest:129 130* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned131 to the KVM guest. Each bit in the mask, from left to right, corresponds to132 an APID from 0-255. If a bit is set, the corresponding adapter is valid for133 use by the KVM guest.134 135* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains136 assigned to the KVM guest. Each bit in the mask, from left to right,137 corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the138 corresponding queue is valid for use by the KVM guest.139 140* The AP Domain Mask field is a bit mask that identifies the AP control domains141 assigned to the KVM guest. The ADM bit mask controls which domains can be142 changed by an AP command-request message sent to a usage domain from the143 guest. Each bit in the mask, from left to right, corresponds to a domain from144 0-255. If a bit is set, the corresponding domain can be modified by an AP145 command-request message sent to a usage domain.146 147If you recall from the description of an AP Queue, AP instructions include148an APQN to identify the AP queue to which an AP command-request message is to be149sent (NQAP and PQAP instructions), or from which a command-reply message is to150be received (DQAP instruction). The validity of an APQN is defined by the matrix151calculated from the APM and AQM; it is the Cartesian product of all assigned152adapter numbers (APM) with all assigned queue indexes (AQM). For example, if153adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs154(1,5), (1,6), (2,5) and (2,6) will be valid for the guest.155 156The APQNs can provide secure key functionality - i.e., a private key is stored157on the adapter card for each of its domains - so each APQN must be assigned to158at most one guest or to the linux host::159 160 Example 1: Valid configuration:161 ------------------------------162 Guest1: adapters 1,2 domains 5,6163 Guest2: adapter 1,2 domain 7164 165 This is valid because both guests have a unique set of APQNs:166 Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);167 Guest2 has APQNs (1,7), (2,7)168 169 Example 2: Valid configuration:170 ------------------------------171 Guest1: adapters 1,2 domains 5,6172 Guest2: adapters 3,4 domains 5,6173 174 This is also valid because both guests have a unique set of APQNs:175 Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);176 Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)177 178 Example 3: Invalid configuration:179 --------------------------------180 Guest1: adapters 1,2 domains 5,6181 Guest2: adapter 1 domains 6,7182 183 This is an invalid configuration because both guests have access to184 APQN (1,6).185 186The Design187==========188The design introduces three new objects:189 1901. AP matrix device1912. VFIO AP device driver (vfio_ap.ko)1923. VFIO AP mediated pass-through device193 194The VFIO AP device driver195-------------------------196The VFIO AP (vfio_ap) device driver serves the following purposes:197 1981. Provides the interfaces to secure APQNs for exclusive use of KVM guests.199 2002. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediated201 device and creates the sysfs interfaces for assigning adapters, usage202 domains, and control domains comprising the matrix for a KVM guest.203 2043. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referenced205 by a KVM guest's SIE state description to grant the guest access to a matrix206 of AP devices207 208Reserve APQNs for exclusive use of KVM guests209---------------------------------------------210The following block diagram illustrates the mechanism by which APQNs are211reserved::212 213 +------------------+214 7 remove | |215 +--------------------> cex4queue driver |216 | | |217 | +------------------+218 |219 |220 | +------------------+ +----------------+221 | 5 register driver | | 3 create | |222 | +----------------> Device core +----------> matrix device |223 | | | | | |224 | | +--------^---------+ +----------------+225 | | |226 | | +-------------------+227 | | +-----------------------------------+ |228 | | | 4 register AP driver | | 2 register device229 | | | | |230 +--------+---+-v---+ +--------+-------+-+231 | | | |232 | ap_bus +--------------------- > vfio_ap driver |233 | | 8 probe | |234 +--------^---------+ +--^--^------------+235 6 edit | | |236 apmask | +-----------------------------+ | 11 mdev create237 aqmask | | 1 modprobe |238 +--------+-----+---+ +----------------+-+ +----------------+239 | | | |10 create| mediated |240 | admin | | VFIO device core |---------> matrix |241 | + | | | device |242 +------+-+---------+ +--------^---------+ +--------^-------+243 | | | |244 | | 9 create vfio_ap-passthrough | |245 | +------------------------------+ |246 +-------------------------------------------------------------+247 12 assign adapter/domain/control domain248 249The process for reserving an AP queue for use by a KVM guest is:250 2511. The administrator loads the vfio_ap device driver2522. The vfio-ap driver during its initialization will register a single 'matrix'253 device with the device core. This will serve as the parent device for254 all vfio_ap mediated devices used to configure an AP matrix for a guest.2553. The /sys/devices/vfio_ap/matrix device is created by the device core2564. The vfio_ap device driver will register with the AP bus for AP queue devices257 of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap258 driver's probe and remove callback interfaces. Devices older than CEX4 queues259 are not supported to simplify the implementation by not needlessly260 complicating the design by supporting older devices that will go out of261 service in the relatively near future, and for which there are few older262 systems around on which to test.2635. The AP bus registers the vfio_ap device driver with the device core2646. The administrator edits the AP adapter and queue masks to reserve AP queues265 for use by the vfio_ap device driver.2667. The AP bus removes the AP queues reserved for the vfio_ap driver from the267 default zcrypt cex4queue driver.2688. The AP bus probes the vfio_ap device driver to bind the queues reserved for269 it.2709. The administrator creates a passthrough type vfio_ap mediated device to be271 used by a guest27210. The administrator assigns the adapters, usage domains and control domains273 to be exclusively used by a guest.274 275Set up the VFIO mediated device interfaces276------------------------------------------277The VFIO AP device driver utilizes the common interfaces of the VFIO mediated278device core driver to:279 280* Register an AP mediated bus driver to add a vfio_ap mediated device to and281 remove it from a VFIO group.282* Create and destroy a vfio_ap mediated device283* Add a vfio_ap mediated device to and remove it from the AP mediated bus driver284* Add a vfio_ap mediated device to and remove it from an IOMMU group285 286The following high-level block diagram shows the main components and interfaces287of the VFIO AP mediated device driver::288 289 +-------------+290 | |291 | +---------+ | mdev_register_driver() +--------------+292 | | Mdev | +<-----------------------+ |293 | | bus | | | vfio_mdev.ko |294 | | driver | +----------------------->+ |<-> VFIO user295 | +---------+ | probe()/remove() +--------------+ APIs296 | |297 | MDEV CORE |298 | MODULE |299 | mdev.ko |300 | +---------+ | mdev_register_parent() +--------------+301 | |Physical | +<-----------------------+ |302 | | device | | | vfio_ap.ko |<-> matrix303 | |interface| +----------------------->+ | device304 | +---------+ | callback +--------------+305 +-------------+306 307During initialization of the vfio_ap module, the matrix device is registered308with an 'mdev_parent_ops' structure that provides the sysfs attribute309structures, mdev functions and callback interfaces for managing the mediated310matrix device.311 312* sysfs attribute structures:313 314 supported_type_groups315 The VFIO mediated device framework supports creation of user-defined316 mediated device types. These mediated device types are specified317 via the 'supported_type_groups' structure when a device is registered318 with the mediated device framework. The registration process creates the319 sysfs structures for each mediated device type specified in the320 'mdev_supported_types' sub-directory of the device being registered. Along321 with the device type, the sysfs attributes of the mediated device type are322 provided.323 324 The VFIO AP device driver will register one mediated device type for325 passthrough devices:326 327 /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough328 329 Only the read-only attributes required by the VFIO mdev framework will330 be provided::331 332 ... name333 ... device_api334 ... available_instances335 ... device_api336 337 Where:338 339 * name:340 specifies the name of the mediated device type341 * device_api:342 the mediated device type's API343 * available_instances:344 the number of vfio_ap mediated passthrough devices345 that can be created346 * device_api:347 specifies the VFIO API348 mdev_attr_groups349 This attribute group identifies the user-defined sysfs attributes of the350 mediated device. When a device is registered with the VFIO mediated device351 framework, the sysfs attribute files identified in the 'mdev_attr_groups'352 structure will be created in the vfio_ap mediated device's directory. The353 sysfs attributes for a vfio_ap mediated device are:354 355 assign_adapter / unassign_adapter:356 Write-only attributes for assigning/unassigning an AP adapter to/from the357 vfio_ap mediated device. To assign/unassign an adapter, the APID of the358 adapter is echoed into the respective attribute file.359 assign_domain / unassign_domain:360 Write-only attributes for assigning/unassigning an AP usage domain to/from361 the vfio_ap mediated device. To assign/unassign a domain, the domain362 number of the usage domain is echoed into the respective attribute363 file.364 matrix:365 A read-only file for displaying the APQNs derived from the Cartesian366 product of the adapter and domain numbers assigned to the vfio_ap mediated367 device.368 guest_matrix:369 A read-only file for displaying the APQNs derived from the Cartesian370 product of the adapter and domain numbers assigned to the APM and AQM371 fields respectively of the KVM guest's CRYCB. This may differ from the372 the APQNs assigned to the vfio_ap mediated device if any APQN does not373 reference a queue device bound to the vfio_ap device driver (i.e., the374 queue is not in the host's AP configuration).375 assign_control_domain / unassign_control_domain:376 Write-only attributes for assigning/unassigning an AP control domain377 to/from the vfio_ap mediated device. To assign/unassign a control domain,378 the ID of the domain to be assigned/unassigned is echoed into the379 respective attribute file.380 control_domains:381 A read-only file for displaying the control domain numbers assigned to the382 vfio_ap mediated device.383 ap_config:384 A read/write file that, when written to, allows all three of the385 vfio_ap mediated device's ap matrix masks to be replaced in one shot.386 Three masks are given, one for adapters, one for domains, and one for387 control domains. If the given state cannot be set then no changes are388 made to the vfio-ap mediated device.389 390 The format of the data written to ap_config is as follows:391 {amask},{dmask},{cmask}\n392 393 \n is a newline character.394 395 amask, dmask, and cmask are masks identifying which adapters, domains,396 and control domains should be assigned to the mediated device.397 398 The format of a mask is as follows:399 0xNN..NN400 401 Where NN..NN is 64 hexadecimal characters representing a 256-bit value.402 The leftmost (highest order) bit represents adapter/domain 0.403 404 For an example set of masks that represent your mdev's current405 configuration, simply cat ap_config.406 407 Setting an adapter or domain number greater than the maximum allowed for408 the system will result in an error.409 410 This attribute is intended to be used by automation. End users would be411 better served using the respective assign/unassign attributes for412 adapters, domains, and control domains.413 414* functions:415 416 create:417 allocates the ap_matrix_mdev structure used by the vfio_ap driver to:418 419 * Store the reference to the KVM structure for the guest using the mdev420 * Store the AP matrix configuration for the adapters, domains, and control421 domains assigned via the corresponding sysfs attributes files422 * Store the AP matrix configuration for the adapters, domains and control423 domains available to a guest. A guest may not be provided access to APQNs424 referencing queue devices that do not exist, or are not bound to the425 vfio_ap device driver.426 427 remove:428 deallocates the vfio_ap mediated device's ap_matrix_mdev structure.429 This will be allowed only if a running guest is not using the mdev.430 431* callback interfaces432 433 open_device:434 The vfio_ap driver uses this callback to register a435 VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the matrix mdev436 devices. The open_device callback is invoked by userspace to connect the437 VFIO iommu group for the matrix mdev device to the MDEV bus. Access to the438 KVM structure used to configure the KVM guest is provided via this callback.439 The KVM structure, is used to configure the guest's access to the AP matrix440 defined via the vfio_ap mediated device's sysfs attribute files.441 442 close_device:443 unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the444 matrix mdev device and deconfigures the guest's AP matrix.445 446 ioctl:447 this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctls448 defined by the vfio framework.449 450Configure the guest's AP resources451----------------------------------452Configuring the AP resources for a KVM guest will be performed when the453VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier454function is called when userspace connects to KVM. The guest's AP resources are455configured via its APCB by:456 457* Setting the bits in the APM corresponding to the APIDs assigned to the458 vfio_ap mediated device via its 'assign_adapter' interface.459* Setting the bits in the AQM corresponding to the domains assigned to the460 vfio_ap mediated device via its 'assign_domain' interface.461* Setting the bits in the ADM corresponding to the domain dIDs assigned to the462 vfio_ap mediated device via its 'assign_control_domains' interface.463 464The linux device model precludes passing a device through to a KVM guest that465is not bound to the device driver facilitating its pass-through. Consequently,466an APQN that does not reference a queue device bound to the vfio_ap device467driver will not be assigned to a KVM guest's matrix. The AP architecture,468however, does not provide a means to filter individual APQNs from the guest's469matrix, so the adapters, domains and control domains assigned to vfio_ap470mediated device via its sysfs 'assign_adapter', 'assign_domain' and471'assign_control_domain' interfaces will be filtered before providing the AP472configuration to a guest:473 474* The APIDs of the adapters, the APQIs of the domains and the domain numbers of475 the control domains assigned to the matrix mdev that are not also assigned to476 the host's AP configuration will be filtered.477 478* Each APQN derived from the Cartesian product of the APIDs and APQIs assigned479 to the vfio_ap mdev is examined and if any one of them does not reference a480 queue device bound to the vfio_ap device driver, the adapter will not be481 plugged into the guest (i.e., the bit corresponding to its APID will not be482 set in the APM of the guest's APCB).483 484The CPU model features for AP485-----------------------------486The AP stack relies on the presence of the AP instructions as well as three487facilities: The AP Facilities Test (APFT) facility; the AP Query488Configuration Information (QCI) facility; and the AP Queue Interruption Control489facility. These features/facilities are made available to a KVM guest via the490following CPU model features:491 4921. ap: Indicates whether the AP instructions are installed on the guest. This493 feature will be enabled by KVM only if the AP instructions are installed494 on the host.495 4962. apft: Indicates the APFT facility is available on the guest. This facility497 can be made available to the guest only if it is available on the host (i.e.,498 facility bit 15 is set).499 5003. apqci: Indicates the AP QCI facility is available on the guest. This facility501 can be made available to the guest only if it is available on the host (i.e.,502 facility bit 12 is set).503 5044. apqi: Indicates AP Queue Interruption Control faclity is available on the505 guest. This facility can be made available to the guest only if it is506 available on the host (i.e., facility bit 65 is set).507 508Note: If the user chooses to specify a CPU model different than the 'host'509model to QEMU, the CPU model features and facilities need to be turned on510explicitly; for example::511 512 /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on513 514A guest can be precluded from using AP features/facilities by turning them off515explicitly; for example::516 517 /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off518 519Note: If the APFT facility is turned off (apft=off) for the guest, the guest520will not see any AP devices. The zcrypt device drivers on the guest that521register for type 10 and newer AP devices - i.e., the cex4card and cex4queue522device drivers - need the APFT facility to ascertain the facilities installed on523a given AP device. If the APFT facility is not installed on the guest, then no524adapter or domain devices will get created by the AP bus running on the525guest because only type 10 and newer devices can be configured for guest use.526 527Example528=======529Let's now provide an example to illustrate how KVM guests may be given530access to AP facilities. For this example, we will show how to configure531three guests such that executing the lszcrypt command on the guests would532look like this:533 534Guest1535------536=========== ===== ============537CARD.DOMAIN TYPE MODE538=========== ===== ============53905 CEX5C CCA-Coproc54005.0004 CEX5C CCA-Coproc54105.00ab CEX5C CCA-Coproc54206 CEX5A Accelerator54306.0004 CEX5A Accelerator54406.00ab CEX5A Accelerator545=========== ===== ============546 547Guest2548------549=========== ===== ============550CARD.DOMAIN TYPE MODE551=========== ===== ============55205 CEX5C CCA-Coproc55305.0047 CEX5C CCA-Coproc55405.00ff CEX5C CCA-Coproc555=========== ===== ============556 557Guest3558------559=========== ===== ============560CARD.DOMAIN TYPE MODE561=========== ===== ============56206 CEX5A Accelerator56306.0047 CEX5A Accelerator56406.00ff CEX5A Accelerator565=========== ===== ============566 567These are the steps:568 5691. Install the vfio_ap module on the linux host. The dependency chain for the570 vfio_ap module is:571 * iommu572 * s390573 * zcrypt574 * vfio575 * vfio_mdev576 * vfio_mdev_device577 * KVM578 579 To build the vfio_ap module, the kernel build must be configured with the580 following Kconfig elements selected:581 * IOMMU_SUPPORT582 * S390583 * AP584 * VFIO585 * KVM586 587 If using make menuconfig select the following to build the vfio_ap module::588 589 -> Device Drivers590 -> IOMMU Hardware Support591 select S390 AP IOMMU Support592 -> VFIO Non-Privileged userspace driver framework593 -> Mediated device driver frramework594 -> VFIO driver for Mediated devices595 -> I/O subsystem596 -> VFIO support for AP devices597 5982. Secure the AP queues to be used by the three guests so that the host can not599 access them. To secure them, there are two sysfs files that specify600 bitmasks marking a subset of the APQN range as usable only by the default AP601 queue device drivers. All remaining APQNs are available for use by602 any other device driver. The vfio_ap device driver is currently the only603 non-default device driver. The location of the sysfs files containing the604 masks are::605 606 /sys/bus/ap/apmask607 /sys/bus/ap/aqmask608 609 The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs610 (APID). Each bit in the mask, from left to right, corresponds to an APID from611 0-255. If a bit is set, the APID belongs to the subset of APQNs marked as612 available only to the default AP queue device drivers.613 614 The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes615 (APQI). Each bit in the mask, from left to right, corresponds to an APQI from616 0-255. If a bit is set, the APQI belongs to the subset of APQNs marked as617 available only to the default AP queue device drivers.618 619 The Cartesian product of the APIDs corresponding to the bits set in the620 apmask and the APQIs corresponding to the bits set in the aqmask comprise621 the subset of APQNs that can be used only by the host default device drivers.622 All other APQNs are available to the non-default device drivers such as the623 vfio_ap driver.624 625 Take, for example, the following masks::626 627 apmask:628 0x7d00000000000000000000000000000000000000000000000000000000000000629 630 aqmask:631 0x8000000000000000000000000000000000000000000000000000000000000000632 633 The masks indicate:634 635 * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default636 device drivers.637 638 * Domain 0 is available for use by the host default device drivers639 640 * The subset of APQNs available for use only by the default host device641 drivers are:642 643 (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)644 645 * All other APQNs are available for use by the non-default device drivers.646 647 The APQN of each AP queue device assigned to the linux host is checked by the648 AP bus against the set of APQNs derived from the Cartesian product of APIDs649 and APQIs marked as available to the default AP queue device drivers. If a650 match is detected, only the default AP queue device drivers will be probed;651 otherwise, the vfio_ap device driver will be probed.652 653 By default, the two masks are set to reserve all APQNs for use by the default654 AP queue device drivers. There are two ways the default masks can be changed:655 656 1. The sysfs mask files can be edited by echoing a string into the657 respective sysfs mask file in one of two formats:658 659 * An absolute hex string starting with 0x - like "0x12345678" - sets660 the mask. If the given string is shorter than the mask, it is padded661 with 0s on the right; for example, specifying a mask value of 0x41 is662 the same as specifying::663 664 0x4100000000000000000000000000000000000000000000000000000000000000665 666 Keep in mind that the mask reads from left to right, so the mask667 above identifies device numbers 1 and 7 (01000001).668 669 If the string is longer than the mask, the operation is terminated with670 an error (EINVAL).671 672 * Individual bits in the mask can be switched on and off by specifying673 each bit number to be switched in a comma separated list. Each bit674 number string must be prepended with a ('+') or minus ('-') to indicate675 the corresponding bit is to be switched on ('+') or off ('-'). Some676 valid values are:677 678 - "+0" switches bit 0 on679 - "-13" switches bit 13 off680 - "+0x41" switches bit 65 on681 - "-0xff" switches bit 255 off682 683 The following example:684 685 +0,-6,+0x47,-0xf0686 687 Switches bits 0 and 71 (0x47) on688 689 Switches bits 6 and 240 (0xf0) off690 691 Note that the bits not specified in the list remain as they were before692 the operation.693 694 2. The masks can also be changed at boot time via parameters on the kernel695 command line like this:696 697 ap.apmask=0xffff ap.aqmask=0x40698 699 This would create the following masks::700 701 apmask:702 0xffff000000000000000000000000000000000000000000000000000000000000703 704 aqmask:705 0x4000000000000000000000000000000000000000000000000000000000000000706 707 Resulting in these two pools::708 709 default drivers pool: adapter 0-15, domain 1710 alternate drivers pool: adapter 16-255, domains 0, 2-255711 712 **Note:**713 Changing a mask such that one or more APQNs will be taken from a vfio_ap714 mediated device (see below) will fail with an error (EBUSY). A message715 is logged to the kernel ring buffer which can be viewed with the 'dmesg'716 command. The output identifies each APQN flagged as 'in use' and identifies717 the vfio_ap mediated device to which it is assigned; for example:718 719 Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804720 Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4721 722Securing the APQNs for our example723----------------------------------724 To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,725 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding726 APQNs can be removed from the default masks using either of the following727 commands::728 729 echo -5,-6 > /sys/bus/ap/apmask730 731 echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask732 733 Or the masks can be set as follows::734 735 echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \736 > apmask737 738 echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \739 > aqmask740 741 This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,742 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The743 sysfs directory for the vfio_ap device driver will now contain symbolic links744 to the AP queue devices bound to it::745 746 /sys/bus/ap747 ... [drivers]748 ...... [vfio_ap]749 ......... [05.0004]750 ......... [05.0047]751 ......... [05.00ab]752 ......... [05.00ff]753 ......... [06.0004]754 ......... [06.0047]755 ......... [06.00ab]756 ......... [06.00ff]757 758 Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)759 can be bound to the vfio_ap device driver. The reason for this is to760 simplify the implementation by not needlessly complicating the design by761 supporting older devices that will go out of service in the relatively near762 future and for which there are few older systems on which to test.763 764 The administrator, therefore, must take care to secure only AP queues that765 can be bound to the vfio_ap device driver. The device type for a given AP766 queue device can be read from the parent card's sysfs directory. For example,767 to see the hardware type of the queue 05.0004:768 769 cat /sys/bus/ap/devices/card05/hwtype770 771 The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the772 vfio_ap device driver.773 7743. Create the mediated devices needed to configure the AP matrixes for the775 three guests and to provide an interface to the vfio_ap driver for776 use by the guests::777 778 /sys/devices/vfio_ap/matrix/779 --- [mdev_supported_types]780 ------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type)781 --------- create782 --------- [devices]783 784 To create the mediated devices for the three guests::785 786 uuidgen > create787 uuidgen > create788 uuidgen > create789 790 or791 792 echo $uuid1 > create793 echo $uuid2 > create794 echo $uuid3 > create795 796 This will create three mediated devices in the [devices] subdirectory named797 after the UUID written to the create attribute file. We call them $uuid1,798 $uuid2 and $uuid3 and this is the sysfs directory structure after creation::799 800 /sys/devices/vfio_ap/matrix/801 --- [mdev_supported_types]802 ------ [vfio_ap-passthrough]803 --------- [devices]804 ------------ [$uuid1]805 --------------- assign_adapter806 --------------- assign_control_domain807 --------------- assign_domain808 --------------- matrix809 --------------- unassign_adapter810 --------------- unassign_control_domain811 --------------- unassign_domain812 813 ------------ [$uuid2]814 --------------- assign_adapter815 --------------- assign_control_domain816 --------------- assign_domain817 --------------- matrix818 --------------- unassign_adapter819 ----------------unassign_control_domain820 ----------------unassign_domain821 822 ------------ [$uuid3]823 --------------- assign_adapter824 --------------- assign_control_domain825 --------------- assign_domain826 --------------- matrix827 --------------- unassign_adapter828 ----------------unassign_control_domain829 ----------------unassign_domain830 831 Note *****: The vfio_ap mdevs do not persist across reboots unless the832 mdevctl tool is used to create and persist them.833 8344. The administrator now needs to configure the matrixes for the mediated835 devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).836 837 This is how the matrix is configured for Guest1::838 839 echo 5 > assign_adapter840 echo 6 > assign_adapter841 echo 4 > assign_domain842 echo 0xab > assign_domain843 844 Control domains can similarly be assigned using the assign_control_domain845 sysfs file.846 847 If a mistake is made configuring an adapter, domain or control domain,848 you can use the unassign_xxx files to unassign the adapter, domain or849 control domain.850 851 To display the matrix configuration for Guest1::852 853 cat matrix854 855 To display the matrix that is or will be assigned to Guest1::856 857 cat guest_matrix858 859 This is how the matrix is configured for Guest2::860 861 echo 5 > assign_adapter862 echo 0x47 > assign_domain863 echo 0xff > assign_domain864 865 This is how the matrix is configured for Guest3::866 867 echo 6 > assign_adapter868 echo 0x47 > assign_domain869 echo 0xff > assign_domain870 871 In order to successfully assign an adapter:872 873 * The adapter number specified must represent a value from 0 up to the874 maximum adapter number configured for the system. If an adapter number875 higher than the maximum is specified, the operation will terminate with876 an error (ENODEV).877 878 Note: The maximum adapter number can be obtained via the sysfs879 /sys/bus/ap/ap_max_adapter_id attribute file.880 881 * Each APQN derived from the Cartesian product of the APID of the adapter882 being assigned and the APQIs of the domains previously assigned:883 884 - Must only be available to the vfio_ap device driver as specified in the885 sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even886 one APQN is reserved for use by the host device driver, the operation887 will terminate with an error (EADDRNOTAVAIL).888 889 - Must NOT be assigned to another vfio_ap mediated device. If even one APQN890 is assigned to another vfio_ap mediated device, the operation will891 terminate with an error (EBUSY).892 893 - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and894 sys/bus/ap/aqmask attribute files are being edited or the operation may895 terminate with an error (EBUSY).896 897 In order to successfully assign a domain:898 899 * The domain number specified must represent a value from 0 up to the900 maximum domain number configured for the system. If a domain number901 higher than the maximum is specified, the operation will terminate with902 an error (ENODEV).903 904 Note: The maximum domain number can be obtained via the sysfs905 /sys/bus/ap/ap_max_domain_id attribute file.906 907 * Each APQN derived from the Cartesian product of the APQI of the domain908 being assigned and the APIDs of the adapters previously assigned:909 910 - Must only be available to the vfio_ap device driver as specified in the911 sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even912 one APQN is reserved for use by the host device driver, the operation913 will terminate with an error (EADDRNOTAVAIL).914 915 - Must NOT be assigned to another vfio_ap mediated device. If even one APQN916 is assigned to another vfio_ap mediated device, the operation will917 terminate with an error (EBUSY).918 919 - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and920 sys/bus/ap/aqmask attribute files are being edited or the operation may921 terminate with an error (EBUSY).922 923 In order to successfully assign a control domain:924 925 * The domain number specified must represent a value from 0 up to the maximum926 domain number configured for the system. If a control domain number higher927 than the maximum is specified, the operation will terminate with an928 error (ENODEV).929 9305. Start Guest1::931 932 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \933 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...934 9357. Start Guest2::936 937 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \938 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...939 9407. Start Guest3::941 942 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \943 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...944 945When the guest is shut down, the vfio_ap mediated devices may be removed.946 947Using our example again, to remove the vfio_ap mediated device $uuid1::948 949 /sys/devices/vfio_ap/matrix/950 --- [mdev_supported_types]951 ------ [vfio_ap-passthrough]952 --------- [devices]953 ------------ [$uuid1]954 --------------- remove955 956::957 958 echo 1 > remove959 960This will remove all of the matrix mdev device's sysfs structures including961the mdev device itself. To recreate and reconfigure the matrix mdev device,962all of the steps starting with step 3 will have to be performed again. Note963that the remove will fail if a guest using the vfio_ap mdev is still running.964 965It is not necessary to remove a vfio_ap mdev, but one may want to966remove it if no guest will use it during the remaining lifetime of the linux967host. If the vfio_ap mdev is removed, one may want to also reconfigure968the pool of adapters and queues reserved for use by the default drivers.969 970Hot plug/unplug support:971========================972An adapter, domain or control domain may be hot plugged into a running KVM973guest by assigning it to the vfio_ap mediated device being used by the guest if974the following conditions are met:975 976* The adapter, domain or control domain must also be assigned to the host's977 AP configuration.978 979* Each APQN derived from the Cartesian product comprised of the APID of the980 adapter being assigned and the APQIs of the domains assigned must reference a981 queue device bound to the vfio_ap device driver.982 983* To hot plug a domain, each APQN derived from the Cartesian product984 comprised of the APQI of the domain being assigned and the APIDs of the985 adapters assigned must reference a queue device bound to the vfio_ap device986 driver.987 988An adapter, domain or control domain may be hot unplugged from a running KVM989guest by unassigning it from the vfio_ap mediated device being used by the990guest.991 992Over-provisioning of AP queues for a KVM guest:993===============================================994Over-provisioning is defined herein as the assignment of adapters or domains to995a vfio_ap mediated device that do not reference AP devices in the host's AP996configuration. The idea here is that when the adapter or domain becomes997available, it will be automatically hot-plugged into the KVM guest using998the vfio_ap mediated device to which it is assigned as long as each new APQN999resulting from plugging it in references a queue device bound to the vfio_ap1000device driver.1001 1002Driver Features1003===============1004The vfio_ap driver exposes a sysfs file containing supported features.1005This exists so third party tools (like Libvirt and mdevctl) can query the1006availability of specific features.1007 1008The features list can be found here: /sys/bus/matrix/devices/matrix/features1009 1010Entries are space delimited. Each entry consists of a combination of1011alphanumeric and underscore characters.1012 1013Example:1014cat /sys/bus/matrix/devices/matrix/features1015guest_matrix dyn ap_config1016 1017the following features are advertised:1018 1019---------------+---------------------------------------------------------------+1020| Flag | Description |1021+==============+===============================================================+1022| guest_matrix | guest_matrix attribute exists. It reports the matrix of |1023| | adapters and domains that are or will be passed through to a |1024| | guest when the mdev is attached to it. |1025+--------------+---------------------------------------------------------------+1026| dyn | Indicates hot plug/unplug of AP adapters, domains and control |1027| | domains for a guest to which the mdev is attached. |1028+------------+-----------------------------------------------------------------+1029| ap_config | ap_config interface for one-shot modifications to mdev config |1030+--------------+---------------------------------------------------------------+1031 1032Limitations1033===========1034Live guest migration is not supported for guests using AP devices without1035intervention by a system administrator. Before a KVM guest can be migrated,1036the vfio_ap mediated device must be removed. Unfortunately, it can not be1037removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while1038the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,1039its mdev can be hot unplugged from the guest in one of two ways:1040 10411. If the KVM guest was started with libvirt, you can hot unplug the mdev via1042 the following commands:1043 1044 virsh detach-device <guestname> <path-to-device-xml>1045 1046 For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from1047 the guest named 'my-guest':1048 1049 virsh detach-device my-guest ~/config/my-guest-hostdev.xml1050 1051 The contents of my-guest-hostdev.xml:1052 1053.. code-block:: xml1054 1055 <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>1056 <source>1057 <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>1058 </source>1059 </hostdev>1060 1061 1062 virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"1063 1064 For example, to hot unplug the vfio_ap mediated device identified on the1065 qemu command line with 'id=hostdev0' from the guest named 'my-guest':1066 1067.. code-block:: sh1068 1069 virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"1070 10712. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor1072 to the guest and using the following qemu monitor command:1073 1074 (QEMU) device-del id=<device-id>1075 1076 For example, to hot unplug the vfio_ap mediated device that was specified1077 on the qemu command line with 'id=hostdev0' when the guest was started:1078 1079 (QEMU) device-del id=hostdev01080 1081After live migration of the KVM guest completes, an AP configuration can be1082restored to the KVM guest by hot plugging a vfio_ap mediated device on the target1083system into the guest in one of two ways:1084 10851. If the KVM guest was started with libvirt, you can hot plug a matrix mediated1086 device into the guest via the following virsh commands:1087 1088 virsh attach-device <guestname> <path-to-device-xml>1089 1090 For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into1091 the guest named 'my-guest':1092 1093 virsh attach-device my-guest ~/config/my-guest-hostdev.xml1094 1095 The contents of my-guest-hostdev.xml:1096 1097.. code-block:: xml1098 1099 <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>1100 <source>1101 <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>1102 </source>1103 </hostdev>1104 1105 1106 virsh qemu-monitor-command <guest-name> --hmp \1107 "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"1108 1109 For example, to hot plug the vfio_ap mediated device1110 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with1111 device-id hostdev0:1112 1113 virsh qemu-monitor-command my-guest --hmp \1114 "device_add vfio-ap,\1115 sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\1116 id=hostdev0"1117 11182. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor1119 to the guest and using the following qemu monitor command:1120 1121 (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"1122 1123 For example, to plug the vfio_ap mediated device1124 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id1125 hostdev0:1126 1127 (QEMU) device-add "vfio-ap,\1128 sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\1129 id=hostdev0"1130