248 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3===========================================================4POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)5===========================================================6 7Device types supported:8 - KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 19 10This device acts as a VM interrupt controller. It provides the KVM11interface to configure the interrupt sources of a VM in the underlying12POWER9 XIVE interrupt controller.13 14Only one XIVE instance may be instantiated. A guest XIVE device15requires a POWER9 host and the guest OS should have support for the16XIVE native exploitation interrupt mode. If not, it should run using17the legacy interrupt mode, referred as XICS (POWER7/8).18 19* Device Mappings20 21 The KVM device exposes different MMIO ranges of the XIVE HW which22 are required for interrupt management. These are exposed to the23 guest in VMAs populated with a custom VM fault handler.24 25 1. Thread Interrupt Management Area (TIMA)26 27 Each thread has an associated Thread Interrupt Management context28 composed of a set of registers. These registers let the thread29 handle priority management and interrupt acknowledgment. The most30 important are :31 32 - Interrupt Pending Buffer (IPB)33 - Current Processor Priority (CPPR)34 - Notification Source Register (NSR)35 36 They are exposed to software in four different pages each proposing37 a view with a different privilege. The first page is for the38 physical thread context and the second for the hypervisor. Only the39 third (operating system) and the fourth (user level) are exposed the40 guest.41 42 2. Event State Buffer (ESB)43 44 Each source is associated with an Event State Buffer (ESB) with45 either a pair of even/odd pair of pages which provides commands to46 manage the source: to trigger, to EOI, to turn off the source for47 instance.48 49 3. Device pass-through50 51 When a device is passed-through into the guest, the source52 interrupts are from a different HW controller (PHB4) and the ESB53 pages exposed to the guest should accommodate this change.54 55 The passthru_irq helpers, kvmppc_xive_set_mapped() and56 kvmppc_xive_clr_mapped() are called when the device HW irqs are57 mapped into or unmapped from the guest IRQ number space. The KVM58 device extends these helpers to clear the ESB pages of the guest IRQ59 number being mapped and then lets the VM fault handler repopulate.60 The handler will insert the ESB page corresponding to the HW61 interrupt of the device being passed-through or the initial IPI ESB62 page if the device has being removed.63 64 The ESB remapping is fully transparent to the guest and the OS65 device driver. All handling is done within VFIO and the above66 helpers in KVM-PPC.67 68* Groups:69 701. KVM_DEV_XIVE_GRP_CTRL71 Provides global controls on the device72 73 Attributes:74 1.1 KVM_DEV_XIVE_RESET (write only)75 Resets the interrupt controller configuration for sources and event76 queues. To be used by kexec and kdump.77 78 Errors: none79 80 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)81 Sync all the sources and queues and mark the EQ pages dirty. This82 to make sure that a consistent memory state is captured when83 migrating the VM.84 85 Errors: none86 87 1.3 KVM_DEV_XIVE_NR_SERVERS (write only)88 The kvm_device_attr.addr points to a __u32 value which is the number of89 interrupt server numbers (ie, highest possible vcpu id plus one).90 91 Errors:92 93 ======= ==========================================94 -EINVAL Value greater than KVM_MAX_VCPU_IDS.95 -EFAULT Invalid user pointer for attr->addr.96 -EBUSY A vCPU is already connected to the device.97 ======= ==========================================98 992. KVM_DEV_XIVE_GRP_SOURCE (write only)100 Initializes a new source in the XIVE device and mask it.101 102 Attributes:103 Interrupt source number (64-bit)104 105 The kvm_device_attr.addr points to a __u64 value::106 107 bits: | 63 .... 2 | 1 | 0108 values: | unused | level | type109 110 - type: 0:MSI 1:LSI111 - level: assertion level in case of an LSI.112 113 Errors:114 115 ======= ==========================================116 -E2BIG Interrupt source number is out of range117 -ENOMEM Could not create a new source block118 -EFAULT Invalid user pointer for attr->addr.119 -ENXIO Could not allocate underlying HW interrupt120 ======= ==========================================121 1223. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)123 Configures source targeting124 125 Attributes:126 Interrupt source number (64-bit)127 128 The kvm_device_attr.addr points to a __u64 value::129 130 bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0131 values: | eisn | mask | server | priority132 133 - priority: 0-7 interrupt priority level134 - server: CPU number chosen to handle the interrupt135 - mask: mask flag (unused)136 - eisn: Effective Interrupt Source Number137 138 Errors:139 140 ======= =======================================================141 -ENOENT Unknown source number142 -EINVAL Not initialized source number143 -EINVAL Invalid priority144 -EINVAL Invalid CPU number.145 -EFAULT Invalid user pointer for attr->addr.146 -ENXIO CPU event queues not configured or configuration of the147 underlying HW interrupt failed148 -EBUSY No CPU available to serve interrupt149 ======= =======================================================150 1514. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)152 Configures an event queue of a CPU153 154 Attributes:155 EQ descriptor identifier (64-bit)156 157 The EQ descriptor identifier is a tuple (server, priority)::158 159 bits: | 63 .... 32 | 31 .. 3 | 2 .. 0160 values: | unused | server | priority161 162 The kvm_device_attr.addr points to::163 164 struct kvm_ppc_xive_eq {165 __u32 flags;166 __u32 qshift;167 __u64 qaddr;168 __u32 qtoggle;169 __u32 qindex;170 __u8 pad[40];171 };172 173 - flags: queue flags174 KVM_XIVE_EQ_ALWAYS_NOTIFY (required)175 forces notification without using the coalescing mechanism176 provided by the XIVE END ESBs.177 - qshift: queue size (power of 2)178 - qaddr: real address of queue179 - qtoggle: current queue toggle bit180 - qindex: current queue index181 - pad: reserved for future use182 183 Errors:184 185 ======= =========================================186 -ENOENT Invalid CPU number187 -EINVAL Invalid priority188 -EINVAL Invalid flags189 -EINVAL Invalid queue size190 -EINVAL Invalid queue address191 -EFAULT Invalid user pointer for attr->addr.192 -EIO Configuration of the underlying HW failed193 ======= =========================================194 1955. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)196 Synchronize the source to flush event notifications197 198 Attributes:199 Interrupt source number (64-bit)200 201 Errors:202 203 ======= =============================204 -ENOENT Unknown source number205 -EINVAL Not initialized source number206 ======= =============================207 208* VCPU state209 210 The XIVE IC maintains VP interrupt state in an internal structure211 called the NVT. When a VP is not dispatched on a HW processor212 thread, this structure can be updated by HW if the VP is the target213 of an event notification.214 215 It is important for migration to capture the cached IPB from the NVT216 as it synthesizes the priorities of the pending interrupts. We217 capture a bit more to report debug information.218 219 KVM_REG_PPC_VP_STATE (2 * 64bits)::220 221 bits: | 63 .... 32 | 31 .... 0 |222 values: | TIMA word0 | TIMA word1 |223 bits: | 127 .......... 64 |224 values: | unused |225 226* Migration:227 228 Saving the state of a VM using the XIVE native exploitation mode229 should follow a specific sequence. When the VM is stopped :230 231 1. Mask all sources (PQ=01) to stop the flow of events.232 233 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to234 flush any in-flight event notification and to stabilize the EQs. At235 this stage, the EQ pages are marked dirty to make sure they are236 transferred in the migration sequence.237 238 3. Capture the state of the source targeting, the EQs configuration239 and the state of thread interrupt context registers.240 241 Restore is similar:242 243 1. Restore the EQ configuration. As targeting depends on it.244 2. Restore targeting245 3. Restore the thread interrupt contexts246 4. Restore the source states247 5. Let the vCPU run248