208 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3Overview4========5The Linux kernel contains a variety of code for running as a fully6enlightened guest on Microsoft's Hyper-V hypervisor. Hyper-V7consists primarily of a bare-metal hypervisor plus a virtual machine8management service running in the parent partition (roughly9equivalent to KVM and QEMU, for example). Guest VMs run in child10partitions. In this documentation, references to Hyper-V usually11encompass both the hypervisor and the VMM service without making a12distinction about which functionality is provided by which13component.14 15Hyper-V runs on x86/x64 and arm64 architectures, and Linux guests16are supported on both. The functionality and behavior of Hyper-V is17generally the same on both architectures unless noted otherwise.18 19Linux Guest Communication with Hyper-V20--------------------------------------21Linux guests communicate with Hyper-V in four different ways:22 23* Implicit traps: As defined by the x86/x64 or arm64 architecture,24 some guest actions trap to Hyper-V. Hyper-V emulates the action and25 returns control to the guest. This behavior is generally invisible26 to the Linux kernel.27 28* Explicit hypercalls: Linux makes an explicit function call to29 Hyper-V, passing parameters. Hyper-V performs the requested action30 and returns control to the caller. Parameters are passed in31 processor registers or in memory shared between the Linux guest and32 Hyper-V. On x86/x64, hypercalls use a Hyper-V specific calling33 sequence. On arm64, hypercalls use the ARM standard SMCCC calling34 sequence.35 36* Synthetic register access: Hyper-V implements a variety of37 synthetic registers. On x86/x64 these registers appear as MSRs in38 the guest, and the Linux kernel can read or write these MSRs using39 the normal mechanisms defined by the x86/x64 architecture. On40 arm64, these synthetic registers must be accessed using explicit41 hypercalls.42 43* VMBus: VMBus is a higher-level software construct that is built on44 the other 3 mechanisms. It is a message passing interface between45 the Hyper-V host and the Linux guest. It uses memory that is shared46 between Hyper-V and the guest, along with various signaling47 mechanisms.48 49The first three communication mechanisms are documented in the50`Hyper-V Top Level Functional Spec (TLFS)`_. The TLFS describes51general Hyper-V functionality and provides details on the hypercalls52and synthetic registers. The TLFS is currently written for the53x86/x64 architecture only.54 55.. _Hyper-V Top Level Functional Spec (TLFS): https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs56 57VMBus is not documented. This documentation provides a high-level58overview of VMBus and how it works, but the details can be discerned59only from the code.60 61Sharing Memory62--------------63Many aspects are communication between Hyper-V and Linux are based64on sharing memory. Such sharing is generally accomplished as65follows:66 67* Linux allocates memory from its physical address space using68 standard Linux mechanisms.69 70* Linux tells Hyper-V the guest physical address (GPA) of the71 allocated memory. Many shared areas are kept to 1 page so that a72 single GPA is sufficient. Larger shared areas require a list of73 GPAs, which usually do not need to be contiguous in the guest74 physical address space. How Hyper-V is told about the GPA or list75 of GPAs varies. In some cases, a single GPA is written to a76 synthetic register. In other cases, a GPA or list of GPAs is sent77 in a VMBus message.78 79* Hyper-V translates the GPAs into "real" physical memory addresses,80 and creates a virtual mapping that it can use to access the memory.81 82* Linux can later revoke sharing it has previously established by83 telling Hyper-V to set the shared GPA to zero.84 85Hyper-V operates with a page size of 4 Kbytes. GPAs communicated to86Hyper-V may be in the form of page numbers, and always describe a87range of 4 Kbytes. Since the Linux guest page size on x86/x64 is88also 4 Kbytes, the mapping from guest page to Hyper-V page is 1-to-1.89On arm64, Hyper-V supports guests with 4/16/64 Kbyte pages as90defined by the arm64 architecture. If Linux is using 16 or 6491Kbyte pages, Linux code must be careful to communicate with Hyper-V92only in terms of 4 Kbyte pages. HV_HYP_PAGE_SIZE and related macros93are used in code that communicates with Hyper-V so that it works94correctly in all configurations.95 96As described in the TLFS, a few memory pages shared between Hyper-V97and the Linux guest are "overlay" pages. With overlay pages, Linux98uses the usual approach of allocating guest memory and telling99Hyper-V the GPA of the allocated memory. But Hyper-V then replaces100that physical memory page with a page it has allocated, and the101original physical memory page is no longer accessible in the guest102VM. Linux may access the memory normally as if it were the memory103that it originally allocated. The "overlay" behavior is visible104only because the contents of the page (as seen by Linux) change at105the time that Linux originally establishes the sharing and the106overlay page is inserted. Similarly, the contents change if Linux107revokes the sharing, in which case Hyper-V removes the overlay page,108and the guest page originally allocated by Linux becomes visible109again.110 111Before Linux does a kexec to a kdump kernel or any other kernel,112memory shared with Hyper-V should be revoked. Hyper-V could modify113a shared page or remove an overlay page after the new kernel is114using the page for a different purpose, corrupting the new kernel.115Hyper-V does not provide a single "set everything" operation to116guest VMs, so Linux code must individually revoke all sharing before117doing kexec. See hv_kexec_handler() and hv_crash_handler(). But118the crash/panic path still has holes in cleanup because some shared119pages are set using per-CPU synthetic registers and there's no120mechanism to revoke the shared pages for CPUs other than the CPU121running the panic path.122 123CPU Management124--------------125Hyper-V does not have a ability to hot-add or hot-remove a CPU126from a running VM. However, Windows Server 2019 Hyper-V and127earlier versions may provide guests with ACPI tables that indicate128more CPUs than are actually present in the VM. As is normal, Linux129treats these additional CPUs as potential hot-add CPUs, and reports130them as such even though Hyper-V will never actually hot-add them.131Starting in Windows Server 2022 Hyper-V, the ACPI tables reflect132only the CPUs actually present in the VM, so Linux does not report133any hot-add CPUs.134 135A Linux guest CPU may be taken offline using the normal Linux136mechanisms, provided no VMBus channel interrupts are assigned to137the CPU. See the section on VMBus Interrupts for more details138on how VMBus channel interrupts can be re-assigned to permit139taking a CPU offline.140 14132-bit and 64-bit142-----------------143On x86/x64, Hyper-V supports 32-bit and 64-bit guests, and Linux144will build and run in either version. While the 32-bit version is145expected to work, it is used rarely and may suffer from undetected146regressions.147 148On arm64, Hyper-V supports only 64-bit guests.149 150Endian-ness151-----------152All communication between Hyper-V and guest VMs uses Little-Endian153format on both x86/x64 and arm64. Big-endian format on arm64 is not154supported by Hyper-V, and Linux code does not use endian-ness macros155when accessing data shared with Hyper-V.156 157Versioning158----------159Current Linux kernels operate correctly with older versions of160Hyper-V back to Windows Server 2012 Hyper-V. Support for running161on the original Hyper-V release in Windows Server 2008/2008 R2162has been removed.163 164A Linux guest on Hyper-V outputs in dmesg the version of Hyper-V165it is running on. This version is in the form of a Windows build166number and is for display purposes only. Linux code does not167test this version number at runtime to determine available features168and functionality. Hyper-V indicates feature/function availability169via flags in synthetic MSRs that Hyper-V provides to the guest,170and the guest code tests these flags.171 172VMBus has its own protocol version that is negotiated during the173initial VMBus connection from the guest to Hyper-V. This version174number is also output to dmesg during boot. This version number175is checked in a few places in the code to determine if specific176functionality is present.177 178Furthermore, each synthetic device on VMBus also has a protocol179version that is separate from the VMBus protocol version. Device180drivers for these synthetic devices typically negotiate the device181protocol version, and may test that protocol version to determine182if specific device functionality is present.183 184Code Packaging185--------------186Hyper-V related code appears in the Linux kernel code tree in three187main areas:188 1891. drivers/hv190 1912. arch/x86/hyperv and arch/arm64/hyperv192 1933. individual device driver areas such as drivers/scsi, drivers/net,194 drivers/clocksource, etc.195 196A few miscellaneous files appear elsewhere. See the full list under197"Hyper-V/Azure CORE AND DRIVERS" and "DRM DRIVER FOR HYPERV198SYNTHETIC VIDEO DEVICE" in the MAINTAINERS file.199 200The code in #1 and #2 is built only when CONFIG_HYPERV is set.201Similarly, the code for most Hyper-V related drivers is built only202when CONFIG_HYPERV is set.203 204Most Hyper-V related code in #1 and #3 can be built as a module.205The architecture specific code in #2 must be built-in. Also,206drivers/hv/hv_common.c is low-level code that is common across207architectures and must be built-in.208