327 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3VMBus4=====5VMBus is a software construct provided by Hyper-V to guest VMs. It6consists of a control path and common facilities used by synthetic7devices that Hyper-V presents to guest VMs. The control path is8used to offer synthetic devices to the guest VM and, in some cases,9to rescind those devices. The common facilities include software10channels for communicating between the device driver in the guest VM11and the synthetic device implementation that is part of Hyper-V, and12signaling primitives to allow Hyper-V and the guest to interrupt13each other.14 15VMBus is modeled in Linux as a bus, with the expected /sys/bus/vmbus16entry in a running Linux guest. The VMBus driver (drivers/hv/vmbus_drv.c)17establishes the VMBus control path with the Hyper-V host, then18registers itself as a Linux bus driver. It implements the standard19bus functions for adding and removing devices to/from the bus.20 21Most synthetic devices offered by Hyper-V have a corresponding Linux22device driver. These devices include:23 24* SCSI controller25* NIC26* Graphics frame buffer27* Keyboard28* Mouse29* PCI device pass-thru30* Heartbeat31* Time Sync32* Shutdown33* Memory balloon34* Key/Value Pair (KVP) exchange with Hyper-V35* Hyper-V online backup (a.k.a. VSS)36 37Guest VMs may have multiple instances of the synthetic SCSI38controller, synthetic NIC, and PCI pass-thru devices. Other39synthetic devices are limited to a single instance per VM. Not40listed above are a small number of synthetic devices offered by41Hyper-V that are used only by Windows guests and for which Linux42does not have a driver.43 44Hyper-V uses the terms "VSP" and "VSC" in describing synthetic45devices. "VSP" refers to the Hyper-V code that implements a46particular synthetic device, while "VSC" refers to the driver for47the device in the guest VM. For example, the Linux driver for the48synthetic NIC is referred to as "netvsc" and the Linux driver for49the synthetic SCSI controller is "storvsc". These drivers contain50functions with names like "storvsc_connect_to_vsp".51 52VMBus channels53--------------54An instance of a synthetic device uses VMBus channels to communicate55between the VSP and the VSC. Channels are bi-directional and used56for passing messages. Most synthetic devices use a single channel,57but the synthetic SCSI controller and synthetic NIC may use multiple58channels to achieve higher performance and greater parallelism.59 60Each channel consists of two ring buffers. These are classic ring61buffers from a university data structures textbook. If the read62and writes pointers are equal, the ring buffer is considered to be63empty, so a full ring buffer always has at least one byte unused.64The "in" ring buffer is for messages from the Hyper-V host to the65guest, and the "out" ring buffer is for messages from the guest to66the Hyper-V host. In Linux, the "in" and "out" designations are as67viewed by the guest side. The ring buffers are memory that is68shared between the guest and the host, and they follow the standard69paradigm where the memory is allocated by the guest, with the list70of GPAs that make up the ring buffer communicated to the host. Each71ring buffer consists of a header page (4 Kbytes) with the read and72write indices and some control flags, followed by the memory for the73actual ring. The size of the ring is determined by the VSC in the74guest and is specific to each synthetic device. The list of GPAs75making up the ring is communicated to the Hyper-V host over the76VMBus control path as a GPA Descriptor List (GPADL). See function77vmbus_establish_gpadl().78 79Each ring buffer is mapped into contiguous Linux kernel virtual80space in three parts: 1) the 4 Kbyte header page, 2) the memory81that makes up the ring itself, and 3) a second mapping of the memory82that makes up the ring itself. Because (2) and (3) are contiguous83in kernel virtual space, the code that copies data to and from the84ring buffer need not be concerned with ring buffer wrap-around.85Once a copy operation has completed, the read or write index may86need to be reset to point back into the first mapping, but the87actual data copy does not need to be broken into two parts. This88approach also allows complex data structures to be easily accessed89directly in the ring without handling wrap-around.90 91On arm64 with page sizes > 4 Kbytes, the header page must still be92passed to Hyper-V as a 4 Kbyte area. But the memory for the actual93ring must be aligned to PAGE_SIZE and have a size that is a multiple94of PAGE_SIZE so that the duplicate mapping trick can be done. Hence95a portion of the header page is unused and not communicated to96Hyper-V. This case is handled by vmbus_establish_gpadl().97 98Hyper-V enforces a limit on the aggregate amount of guest memory99that can be shared with the host via GPADLs. This limit ensures100that a rogue guest can't force the consumption of excessive host101resources. For Windows Server 2019 and later, this limit is102approximately 1280 Mbytes. For versions prior to Windows Server1032019, the limit is approximately 384 Mbytes.104 105VMBus channel messages106----------------------107All messages sent in a VMBus channel have a standard header that includes108the message length, the offset of the message payload, some flags, and a109transactionID. The portion of the message after the header is110unique to each VSP/VSC pair.111 112Messages follow one of two patterns:113 114* Unidirectional: Either side sends a message and does not115 expect a response message116* Request/response: One side (usually the guest) sends a message117 and expects a response118 119The transactionID (a.k.a. "requestID") is for matching requests &120responses. Some synthetic devices allow multiple requests to be in-121flight simultaneously, so the guest specifies a transactionID when122sending a request. Hyper-V sends back the same transactionID in the123matching response.124 125Messages passed between the VSP and VSC are control messages. For126example, a message sent from the storvsc driver might be "execute127this SCSI command". If a message also implies some data transfer128between the guest and the Hyper-V host, the actual data to be129transferred may be embedded with the control message, or it may be130specified as a separate data buffer that the Hyper-V host will131access as a DMA operation. The former case is used when the size of132the data is small and the cost of copying the data to and from the133ring buffer is minimal. For example, time sync messages from the134Hyper-V host to the guest contain the actual time value. When the135data is larger, a separate data buffer is used. In this case, the136control message contains a list of GPAs that describe the data137buffer. For example, the storvsc driver uses this approach to138specify the data buffers to/from which disk I/O is done.139 140Three functions exist to send VMBus channel messages:141 1421. vmbus_sendpacket(): Control-only messages and messages with143 embedded data -- no GPAs1442. vmbus_sendpacket_pagebuffer(): Message with list of GPAs145 identifying data to transfer. An offset and length is146 associated with each GPA so that multiple discontinuous areas147 of guest memory can be targeted.1483. vmbus_sendpacket_mpb_desc(): Message with list of GPAs149 identifying data to transfer. A single offset and length is150 associated with a list of GPAs. The GPAs must describe a151 single logical area of guest memory to be targeted.152 153Historically, Linux guests have trusted Hyper-V to send well-formed154and valid messages, and Linux drivers for synthetic devices did not155fully validate messages. With the introduction of processor156technologies that fully encrypt guest memory and that allow the157guest to not trust the hypervisor (AMD SEV-SNP, Intel TDX), trusting158the Hyper-V host is no longer a valid assumption. The drivers for159VMBus synthetic devices are being updated to fully validate any160values read from memory that is shared with Hyper-V, which includes161messages from VMBus devices. To facilitate such validation,162messages read by the guest from the "in" ring buffer are copied to a163temporary buffer that is not shared with Hyper-V. Validation is164performed in this temporary buffer without the risk of Hyper-V165maliciously modifying the message after it is validated but before166it is used.167 168Synthetic Interrupt Controller (synic)169--------------------------------------170Hyper-V provides each guest CPU with a synthetic interrupt controller171that is used by VMBus for host-guest communication. While each synic172defines 16 synthetic interrupts (SINT), Linux uses only one of the 16173(VMBUS_MESSAGE_SINT). All interrupts related to communication between174the Hyper-V host and a guest CPU use that SINT.175 176The SINT is mapped to a single per-CPU architectural interrupt (i.e,177an 8-bit x86/x64 interrupt vector, or an arm64 PPI INTID). Because178each CPU in the guest has a synic and may receive VMBus interrupts,179they are best modeled in Linux as per-CPU interrupts. This model works180well on arm64 where a single per-CPU Linux IRQ is allocated for181VMBUS_MESSAGE_SINT. This IRQ appears in /proc/interrupts as an IRQ labelled182"Hyper-V VMbus". Since x86/x64 lacks support for per-CPU IRQs, an x86183interrupt vector is statically allocated (HYPERVISOR_CALLBACK_VECTOR)184across all CPUs and explicitly coded to call vmbus_isr(). In this case,185there's no Linux IRQ, and the interrupts are visible in aggregate in186/proc/interrupts on the "HYP" line.187 188The synic provides the means to demultiplex the architectural interrupt into189one or more logical interrupts and route the logical interrupt to the proper190VMBus handler in Linux. This demultiplexing is done by vmbus_isr() and191related functions that access synic data structures.192 193The synic is not modeled in Linux as an irq chip or irq domain,194and the demultiplexed logical interrupts are not Linux IRQs. As such,195they don't appear in /proc/interrupts or /proc/irq. The CPU196affinity for one of these logical interrupts is controlled via an197entry under /sys/bus/vmbus as described below.198 199VMBus interrupts200----------------201VMBus provides a mechanism for the guest to interrupt the host when202the guest has queued new messages in a ring buffer. The host203expects that the guest will send an interrupt only when an "out"204ring buffer transitions from empty to non-empty. If the guest sends205interrupts at other times, the host deems such interrupts to be206unnecessary. If a guest sends an excessive number of unnecessary207interrupts, the host may throttle that guest by suspending its208execution for a few seconds to prevent a denial-of-service attack.209 210Similarly, the host will interrupt the guest via the synic when211it sends a new message on the VMBus control path, or when a VMBus212channel "in" ring buffer transitions from empty to non-empty due to213the host inserting a new VMBus channel message. The control message stream214and each VMBus channel "in" ring buffer are separate logical interrupts215that are demultiplexed by vmbus_isr(). It demultiplexes by first checking216for channel interrupts by calling vmbus_chan_sched(), which looks at a synic217bitmap to determine which channels have pending interrupts on this CPU.218If multiple channels have pending interrupts for this CPU, they are219processed sequentially. When all channel interrupts have been processed,220vmbus_isr() checks for and processes any messages received on the VMBus221control path.222 223The guest CPU that a VMBus channel will interrupt is selected by the224guest when the channel is created, and the host is informed of that225selection. VMBus devices are broadly grouped into two categories:226 2271. "Slow" devices that need only one VMBus channel. The devices228 (such as keyboard, mouse, heartbeat, and timesync) generate229 relatively few interrupts. Their VMBus channels are all230 assigned to interrupt the VMBUS_CONNECT_CPU, which is always231 CPU 0.232 2332. "High speed" devices that may use multiple VMBus channels for234 higher parallelism and performance. These devices include the235 synthetic SCSI controller and synthetic NIC. Their VMBus236 channels interrupts are assigned to CPUs that are spread out237 among the available CPUs in the VM so that interrupts on238 multiple channels can be processed in parallel.239 240The assignment of VMBus channel interrupts to CPUs is done in the241function init_vp_index(). This assignment is done outside of the242normal Linux interrupt affinity mechanism, so the interrupts are243neither "unmanaged" nor "managed" interrupts.244 245The CPU that a VMBus channel will interrupt can be seen in246/sys/bus/vmbus/devices/<deviceGUID>/ channels/<channelRelID>/cpu.247When running on later versions of Hyper-V, the CPU can be changed248by writing a new value to this sysfs entry. Because VMBus channel249interrupts are not Linux IRQs, there are no entries in /proc/interrupts250or /proc/irq corresponding to individual VMBus channel interrupts.251 252An online CPU in a Linux guest may not be taken offline if it has253VMBus channel interrupts assigned to it. Any such channel254interrupts must first be manually reassigned to another CPU as255described above. When no channel interrupts are assigned to the256CPU, it can be taken offline.257 258The VMBus channel interrupt handling code is designed to work259correctly even if an interrupt is received on a CPU other than the260CPU assigned to the channel. Specifically, the code does not use261CPU-based exclusion for correctness. In normal operation, Hyper-V262will interrupt the assigned CPU. But when the CPU assigned to a263channel is being changed via sysfs, the guest doesn't know exactly264when Hyper-V will make the transition. The code must work correctly265even if there is a time lag before Hyper-V starts interrupting the266new CPU. See comments in target_cpu_store().267 268VMBus device creation/deletion269------------------------------270Hyper-V and the Linux guest have a separate message-passing path271that is used for synthetic device creation and deletion. This272path does not use a VMBus channel. See vmbus_post_msg() and273vmbus_on_msg_dpc().274 275The first step is for the guest to connect to the generic276Hyper-V VMBus mechanism. As part of establishing this connection,277the guest and Hyper-V agree on a VMBus protocol version they will278use. This negotiation allows newer Linux kernels to run on older279Hyper-V versions, and vice versa.280 281The guest then tells Hyper-V to "send offers". Hyper-V sends an282offer message to the guest for each synthetic device that the VM283is configured to have. Each VMBus device type has a fixed GUID284known as the "class ID", and each VMBus device instance is also285identified by a GUID. The offer message from Hyper-V contains286both GUIDs to uniquely (within the VM) identify the device.287There is one offer message for each device instance, so a VM with288two synthetic NICs will get two offers messages with the NIC289class ID. The ordering of offer messages can vary from boot-to-boot290and must not be assumed to be consistent in Linux code. Offer291messages may also arrive long after Linux has initially booted292because Hyper-V supports adding devices, such as synthetic NICs,293to running VMs. A new offer message is processed by294vmbus_process_offer(), which indirectly invokes vmbus_add_channel_work().295 296Upon receipt of an offer message, the guest identifies the device297type based on the class ID, and invokes the correct driver to set up298the device. Driver/device matching is performed using the standard299Linux mechanism.300 301The device driver probe function opens the primary VMBus channel to302the corresponding VSP. It allocates guest memory for the channel303ring buffers and shares the ring buffer with the Hyper-V host by304giving the host a list of GPAs for the ring buffer memory. See305vmbus_establish_gpadl().306 307Once the ring buffer is set up, the device driver and VSP exchange308setup messages via the primary channel. These messages may include309negotiating the device protocol version to be used between the Linux310VSC and the VSP on the Hyper-V host. The setup messages may also311include creating additional VMBus channels, which are somewhat312mis-named as "sub-channels" since they are functionally313equivalent to the primary channel once they are created.314 315Finally, the device driver may create entries in /dev as with316any device driver.317 318The Hyper-V host can send a "rescind" message to the guest to319remove a device that was previously offered. Linux drivers must320handle such a rescind message at any time. Rescinding a device321invokes the device driver "remove" function to cleanly shut322down the device and remove it. Once a synthetic device is323rescinded, neither Hyper-V nor Linux retains any state about324its previous existence. Such a device might be re-added later,325in which case it is treated as an entirely new device. See326vmbus_onoffer_rescind().327