76 lines · plain
1======================2Userspace verbs access3======================4 5 The ib_uverbs module, built by enabling CONFIG_INFINIBAND_USER_VERBS,6 enables direct userspace access to IB hardware via "verbs," as7 described in chapter 11 of the InfiniBand Architecture Specification.8 9 To use the verbs, the libibverbs library, available from10 https://github.com/linux-rdma/rdma-core, is required. libibverbs contains a11 device-independent API for using the ib_uverbs interface.12 libibverbs also requires appropriate device-dependent kernel and13 userspace driver for your InfiniBand hardware. For example, to use14 a Mellanox HCA, you will need the ib_mthca kernel module and the15 libmthca userspace driver be installed.16 17User-kernel communication18=========================19 20 Userspace communicates with the kernel for slow path, resource21 management operations via the /dev/infiniband/uverbsN character22 devices. Fast path operations are typically performed by writing23 directly to hardware registers mmap()ed into userspace, with no24 system call or context switch into the kernel.25 26 Commands are sent to the kernel via write()s on these device files.27 The ABI is defined in drivers/infiniband/include/ib_user_verbs.h.28 The structs for commands that require a response from the kernel29 contain a 64-bit field used to pass a pointer to an output buffer.30 Status is returned to userspace as the return value of the write()31 system call.32 33Resource management34===================35 36 Since creation and destruction of all IB resources is done by37 commands passed through a file descriptor, the kernel can keep track38 of which resources are attached to a given userspace context. The39 ib_uverbs module maintains idr tables that are used to translate40 between kernel pointers and opaque userspace handles, so that kernel41 pointers are never exposed to userspace and userspace cannot trick42 the kernel into following a bogus pointer.43 44 This also allows the kernel to clean up when a process exits and45 prevent one process from touching another process's resources.46 47Memory pinning48==============49 50 Direct userspace I/O requires that memory regions that are potential51 I/O targets be kept resident at the same physical address. The52 ib_uverbs module manages pinning and unpinning memory regions via53 get_user_pages() and put_page() calls. It also accounts for the54 amount of memory pinned in the process's pinned_vm, and checks that55 unprivileged processes do not exceed their RLIMIT_MEMLOCK limit.56 57 Pages that are pinned multiple times are counted each time they are58 pinned, so the value of pinned_vm may be an overestimate of the59 number of pages pinned by a process.60 61/dev files62==========63 64 To create the appropriate character device files automatically with65 udev, a rule like::66 67 KERNEL=="uverbs*", NAME="infiniband/%k"68 69 can be used. This will create device nodes named::70 71 /dev/infiniband/uverbs072 73 and so on. Since the InfiniBand userspace verbs should be safe for74 use by non-privileged processes, it may be useful to add an75 appropriate MODE or GROUP to the udev rule.76