177 lines · plain
1========================================================2OpenCAPI (Open Coherent Accelerator Processor Interface)3========================================================4 5OpenCAPI is an interface between processors and accelerators. It aims6at being low-latency and high-bandwidth. The specification is7developed by the `OpenCAPI Consortium <http://opencapi.org/>`_.8 9It allows an accelerator (which could be an FPGA, ASICs, ...) to access10the host memory coherently, using virtual addresses. An OpenCAPI11device can also host its own memory, that can be accessed from the12host.13 14OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic15evolution of 'cxl' (the driver for the IBM CAPI interface for16powerpc), which was named that way to avoid confusion with the ISDN17CAPI subsystem.18 19 20High-level view21===============22 23OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to24be implemented on top of a physical link. Any processor or device25implementing the DL and TL can start sharing memory.26 27::28 29 +-----------+ +-------------+30 | | | |31 | | | Accelerated |32 | Processor | | Function |33 | | +--------+ | Unit | +--------+34 | |--| Memory | | (AFU) |--| Memory |35 | | +--------+ | | +--------+36 +-----------+ +-------------+37 | |38 +-----------+ +-------------+39 | TL | | TLX |40 +-----------+ +-------------+41 | |42 +-----------+ +-------------+43 | DL | | DLX |44 +-----------+ +-------------+45 | |46 | PHY |47 +---------------------------------------+48 49 50 51Device discovery52================53 54OpenCAPI relies on a PCI-like configuration space, implemented on the55device. So the host can discover AFUs by querying the config space.56 57OpenCAPI devices in Linux are treated like PCI devices (with a few58caveats). The firmware is expected to abstract the hardware as if it59was a PCI link. A lot of the existing PCI infrastructure is reused:60devices are scanned and BARs are assigned during the standard PCI61enumeration. Commands like 'lspci' can therefore be used to see what62devices are available.63 64The configuration space defines the AFU(s) that can be found on the65physical adapter, such as its name, how many memory contexts it can66work with, the size of its MMIO areas, ...67 68 69 70MMIO71====72 73OpenCAPI defines two MMIO areas for each AFU:74 75* the global MMIO area, with registers pertinent to the whole AFU.76* a per-process MMIO area, which has a fixed size for each context.77 78 79 80AFU interrupts81==============82 83OpenCAPI includes the possibility for an AFU to send an interrupt to a84host process. It is done through a 'intrp_req' defined in the85Transaction Layer, specifying a 64-bit object handle which defines the86interrupt.87 88The driver allows a process to allocate an interrupt and obtain its8964-bit object handle, that can be passed to the AFU.90 91 92 93char devices94============95 96The driver creates one char device per AFU found on the physical97device. A physical device may have multiple functions and each98function can have multiple AFUs. At the time of this writing though,99it has only been tested with devices exporting only one AFU.100 101Char devices can be found in /dev/ocxl/ and are named as:102/dev/ocxl/<AFU name>.<location>.<index>103 104where <AFU name> is a max 20-character long name, as found in the105config space of the AFU.106<location> is added by the driver and can help distinguish devices107when a system has more than one instance of the same OpenCAPI device.108<index> is also to help distinguish AFUs in the unlikely case where a109device carries multiple copies of the same AFU.110 111 112 113Sysfs class114===========115 116An ocxl class is added for the devices representing the AFUs. See117/sys/class/ocxl. The layout is described in118Documentation/ABI/testing/sysfs-class-ocxl119 120 121 122User API123========124 125open126----127 128Based on the AFU definition found in the config space, an AFU may129support working with more than one memory context, in which case the130associated char device may be opened multiple times by different131processes.132 133 134ioctl135-----136 137OCXL_IOCTL_ATTACH:138 139 Attach the memory context of the calling process to the AFU so that140 the AFU can access its memory.141 142OCXL_IOCTL_IRQ_ALLOC:143 144 Allocate an AFU interrupt and return an identifier.145 146OCXL_IOCTL_IRQ_FREE:147 148 Free a previously allocated AFU interrupt.149 150OCXL_IOCTL_IRQ_SET_FD:151 152 Associate an event fd to an AFU interrupt so that the user process153 can be notified when the AFU sends an interrupt.154 155OCXL_IOCTL_GET_METADATA:156 157 Obtains configuration information from the card, such at the size of158 MMIO areas, the AFU version, and the PASID for the current context.159 160OCXL_IOCTL_ENABLE_P9_WAIT:161 162 Allows the AFU to wake a userspace thread executing 'wait'. Returns163 information to userspace to allow it to configure the AFU. Note that164 this is only available on POWER9.165 166OCXL_IOCTL_GET_FEATURES:167 168 Reports on which CPU features that affect OpenCAPI are usable from169 userspace.170 171 172mmap173----174 175A process can mmap the per-process MMIO area for interactions with the176AFU.177