176 lines · plain
1.. SPDX-License-Identifier: GPL-2.0+2 3==============================================================4Linux kernel driver for Compute Engine Virtual Ethernet (gve):5==============================================================6 7Supported Hardware8===================9The GVE driver binds to a single PCI device id used by the virtual10Ethernet device found in some Compute Engine VMs.11 12+--------------+----------+---------+13|Field | Value | Comments|14+==============+==========+=========+15|Vendor ID | `0x1AE0` | Google |16+--------------+----------+---------+17|Device ID | `0x0042` | |18+--------------+----------+---------+19|Sub-vendor ID | `0x1AE0` | Google |20+--------------+----------+---------+21|Sub-device ID | `0x0058` | |22+--------------+----------+---------+23|Revision ID | `0x0` | |24+--------------+----------+---------+25|Device Class | `0x200` | Ethernet|26+--------------+----------+---------+27 28PCI Bars29========30The gVNIC PCI device exposes three 32-bit memory BARS:31- Bar0 - Device configuration and status registers.32- Bar1 - MSI-X vector table33- Bar2 - IRQ, RX and TX doorbells34 35Device Interactions36===================37The driver interacts with the device in the following ways:38 - Registers39 - A block of MMIO registers40 - See gve_register.h for more detail41 - Admin Queue42 - See description below43 - Reset44 - At any time the device can be reset45 - Interrupts46 - See supported interrupts below47 - Transmit and Receive Queues48 - See description below49 50Descriptor Formats51------------------52GVE supports two descriptor formats: GQI and DQO. These two formats have53entirely different descriptors, which will be described below.54 55Addressing Mode56------------------57GVE supports two addressing modes: QPL and RDA.58QPL ("queue-page-list") mode communicates data through a set of59pre-registered pages.60 61For RDA ("raw DMA addressing") mode, the set of pages is dynamic.62Therefore, the packet buffers can be anywhere in guest memory.63 64Registers65---------66All registers are MMIO.67 68The registers are used for initializing and configuring the device as well as69querying device status in response to management interrupts.70 71Endianness72----------73- Admin Queue messages and registers are all Big Endian.74- GQI descriptors and datapath registers are Big Endian.75- DQO descriptors and datapath registers are Little Endian.76 77Admin Queue (AQ)78----------------79The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ80commands, used by the driver to issue commands to the device and set up81resources.The driver and the device maintain a count of how many commands82have been submitted and executed. To issue AQ commands, the driver must do83the following (with proper locking):84 851) Copy new commands into next available slots in the AQ array862) Increment its counter by he number of new commands873) Write the counter into the GVE_ADMIN_QUEUE_DOORBELL register884) Poll the ADMIN_QUEUE_EVENT_COUNTER register until it equals89 the value written to the doorbell, or until a timeout.90 91The device will update the status field in each AQ command reported as92executed through the ADMIN_QUEUE_EVENT_COUNTER register.93 94Device Resets95-------------96A device reset is triggered by writing 0x0 to the AQ PFN register.97This causes the device to release all resources allocated by the98driver, including the AQ itself.99 100Interrupts101----------102The following interrupts are supported by the driver:103 104Management Interrupt105~~~~~~~~~~~~~~~~~~~~106The management interrupt is used by the device to tell the driver to107look at the GVE_DEVICE_STATUS register.108 109The handler for the management irq simply queues the service task in110the workqueue to check the register and acks the irq.111 112Notification Block Interrupts113~~~~~~~~~~~~~~~~~~~~~~~~~~~~~114The notification block interrupts are used to tell the driver to poll115the queues associated with that interrupt.116 117The handler for these irqs schedule the napi for that block to run118and poll the queues.119 120GQI Traffic Queues121------------------122GQI queues are composed of a descriptor ring and a buffer and are assigned to a123notification block.124 125The descriptor rings are power-of-two-sized ring buffers consisting of126fixed-size descriptors. They advance their head pointer using a __be32127doorbell located in Bar2. The tail pointers are advanced by consuming128descriptors in-order and updating a __be32 counter. Both the doorbell129and the counter overflow to zero.130 131Each queue's buffers must be registered in advance with the device as a132queue page list, and packet data can only be put in those pages.133 134Transmit135~~~~~~~~136gve maps the buffers for transmit rings into a FIFO and copies the packets137into the FIFO before sending them to the NIC.138 139Receive140~~~~~~~141The buffers for receive rings are put into a data ring that is the same142length as the descriptor ring and the head and tail pointers advance over143the rings together.144 145DQO Traffic Queues146------------------147- Every TX and RX queue is assigned a notification block.148 149- TX and RX buffers queues, which send descriptors to the device, use MMIO150 doorbells to notify the device of new descriptors.151 152- RX and TX completion queues, which receive descriptors from the device, use a153 "generation bit" to know when a descriptor was populated by the device. The154 driver initializes all bits with the "current generation". The device will155 populate received descriptors with the "next generation" which is inverted156 from the current generation. When the ring wraps, the current/next generation157 are swapped.158 159- It's the driver's responsibility to ensure that the RX and TX completion160 queues are not overrun. This can be accomplished by limiting the number of161 descriptors posted to HW.162 163- TX packets have a 16 bit completion_tag and RX buffers have a 16 bit164 buffer_id. These will be returned on the TX completion and RX queues165 respectively to let the driver know which packet/buffer was completed.166 167Transmit168~~~~~~~~169A packet's buffers are DMA mapped for the device to access before transmission.170After the packet was successfully transmitted, the buffers are unmapped.171 172Receive173~~~~~~~174The driver posts fixed sized buffers to HW on the RX buffer queue. The packet175received on the associated RX queue may span multiple descriptors.176