228 lines · plain
1.. SPDX-License-Identifier: GPL-2.0+2 3======================================================4IBM Virtual Management Channel Kernel Driver (IBMVMC)5======================================================6 7:Authors:8 Dave Engebretsen <engebret@us.ibm.com>,9 Adam Reznechek <adreznec@linux.vnet.ibm.com>,10 Steven Royer <seroyer@linux.vnet.ibm.com>,11 Bryant G. Ly <bryantly@linux.vnet.ibm.com>,12 13Introduction14============15 16Note: Knowledge of virtualization technology is required to understand17this document.18 19A good reference document would be:20 21https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf22 23The Virtual Management Channel (VMC) is a logical device which provides an24interface between the hypervisor and a management partition. This interface25is like a message passing interface. This management partition is intended26to provide an alternative to systems that use a Hardware Management27Console (HMC) - based system management.28 29The primary hardware management solution that is developed by IBM relies30on an appliance server named the Hardware Management Console (HMC),31packaged as an external tower or rack-mounted personal computer. In a32Power Systems environment, a single HMC can manage multiple POWER33processor-based systems.34 35Management Application36----------------------37 38In the management partition, a management application exists which enables39a system administrator to configure the system’s partitioning40characteristics via a command line interface (CLI) or Representational41State Transfer Application (REST API's).42 43The management application runs on a Linux logical partition on a44POWER8 or newer processor-based server that is virtualized by PowerVM.45System configuration, maintenance, and control functions which46traditionally require an HMC can be implemented in the management47application using a combination of HMC to hypervisor interfaces and48existing operating system methods. This tool provides a subset of the49functions implemented by the HMC and enables basic partition configuration.50The set of HMC to hypervisor messages supported by the management51application component are passed to the hypervisor over a VMC interface,52which is defined below.53 54The VMC enables the management partition to provide basic partitioning55functions:56 57- Logical Partitioning Configuration58- Start, and stop actions for individual partitions59- Display of partition status60- Management of virtual Ethernet61- Management of virtual Storage62- Basic system management63 64Virtual Management Channel (VMC)65--------------------------------66 67A logical device, called the Virtual Management Channel (VMC), is defined68for communicating between the management application and the hypervisor. It69basically creates the pipes that enable virtualization management70software. This device is presented to a designated management partition as71a virtual device.72 73This communication device uses Command/Response Queue (CRQ) and the74Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is75defined that must take place to establish that both the hypervisor and76management partition sides of the channel are running prior to77sending/receiving any of the protocol messages.78 79This driver also utilizes Transport Event CRQs. CRQ messages are sent80when the hypervisor detects one of the peer partitions has abnormally81terminated, or one side has called H_FREE_CRQ to close their CRQ.82Two new classes of CRQ messages are introduced for the VMC device. VMC83Administrative messages are used for each partition using the VMC to84communicate capabilities to their partner. HMC Interface messages are used85for the actual flow of HMC messages between the management partition and86the hypervisor. As most HMC messages far exceed the size of a CRQ buffer,87a virtual DMA (RMDA) of the HMC message data is done prior to each HMC88Interface CRQ message. Only the management partition drives RDMA89operations; hypervisors never directly cause the movement of message data.90 91 92Terminology93-----------94RDMA95 Remote Direct Memory Access is DMA transfer from the server to its96 client or from the server to its partner partition. DMA refers97 to both physical I/O to and from memory operations and to memory98 to memory move operations.99CRQ100 Command/Response Queue a facility which is used to communicate101 between partner partitions. Transport events which are signaled102 from the hypervisor to partition are also reported in this queue.103 104Example Management Partition VMC Driver Interface105=================================================106 107This section provides an example for the management application108implementation where a device driver is used to interface to the VMC109device. This driver consists of a new device, for example /dev/ibmvmc,110which provides interfaces to open, close, read, write, and perform111ioctl’s against the VMC device.112 113VMC Interface Initialization114----------------------------115 116The device driver is responsible for initializing the VMC when the driver117is loaded. It first creates and initializes the CRQ. Next, an exchange of118VMC capabilities is performed to indicate the code version and number of119resources available in both the management partition and the hypervisor.120Finally, the hypervisor requests that the management partition create an121initial pool of VMC buffers, one buffer for each possible HMC connection,122which will be used for management application session initialization.123Prior to completion of this initialization sequence, the device returns124EBUSY to open() calls. EIO is returned for all open() failures.125 126::127 128 Management Partition Hypervisor129 CRQ INIT130 ---------------------------------------->131 CRQ INIT COMPLETE132 <----------------------------------------133 CAPABILITIES134 ---------------------------------------->135 CAPABILITIES RESPONSE136 <----------------------------------------137 ADD BUFFER (HMC IDX=0,1,..) _138 <---------------------------------------- |139 ADD BUFFER RESPONSE | - Perform # HMCs Iterations140 ----------------------------------------> -141 142VMC Interface Open143------------------144 145After the basic VMC channel has been initialized, an HMC session level146connection can be established. The application layer performs an open() to147the VMC device and executes an ioctl() against it, indicating the HMC ID148(32 bytes of data) for this session. If the VMC device is in an invalid149state, EIO will be returned for the ioctl(). The device driver creates a150new HMC session value (ranging from 1 to 255) and HMC index value (starting151at index 0 and ranging to 254) for this HMC ID. The driver then does an152RDMA of the HMC ID to the hypervisor, and then sends an Interface Open153message to the hypervisor to establish the session over the VMC. After the154hypervisor receives this information, it sends Add Buffer messages to the155management partition to seed an initial pool of buffers for the new HMC156connection. Finally, the hypervisor sends an Interface Open Response157message, to indicate that it is ready for normal runtime messaging. The158following illustrates this VMC flow:159 160::161 162 Management Partition Hypervisor163 RDMA HMC ID164 ---------------------------------------->165 Interface Open166 ---------------------------------------->167 Add Buffer _168 <---------------------------------------- |169 Add Buffer Response | - Perform N Iterations170 ----------------------------------------> -171 Interface Open Response172 <----------------------------------------173 174VMC Interface Runtime175---------------------176 177During normal runtime, the management application and the hypervisor178exchange HMC messages via the Signal VMC message and RDMA operations. When179sending data to the hypervisor, the management application performs a180write() to the VMC device, and the driver RDMA’s the data to the hypervisor181and then sends a Signal Message. If a write() is attempted before VMC182device buffers have been made available by the hypervisor, or no buffers183are currently available, EBUSY is returned in response to the write(). A184write() will return EIO for all other errors, such as an invalid device185state. When the hypervisor sends a message to the management, the data is186put into a VMC buffer and an Signal Message is sent to the VMC driver in187the management partition. The driver RDMA’s the buffer into the partition188and passes the data up to the appropriate management application via a189read() to the VMC device. The read() request blocks if there is no buffer190available to read. The management application may use select() to wait for191the VMC device to become ready with data to read.192 193::194 195 Management Partition Hypervisor196 MSG RDMA197 ---------------------------------------->198 SIGNAL MSG199 ---------------------------------------->200 SIGNAL MSG201 <----------------------------------------202 MSG RDMA203 <----------------------------------------204 205VMC Interface Close206-------------------207 208HMC session level connections are closed by the management partition when209the application layer performs a close() against the device. This action210results in an Interface Close message flowing to the hypervisor, which211causes the session to be terminated. The device driver must free any212storage allocated for buffers for this HMC connection.213 214::215 216 Management Partition Hypervisor217 INTERFACE CLOSE218 ---------------------------------------->219 INTERFACE CLOSE RESPONSE220 <----------------------------------------221 222Additional Information223======================224 225For more information on the documentation for CRQ Messages, VMC Messages,226HMC interface Buffers, and signal messages please refer to the Linux on227Power Architecture Platform Reference. Section F.228