207 lines · plain
1===============2BPF ring buffer3===============4 5This document describes BPF ring buffer design, API, and implementation details.6 7.. contents::8 :local:9 :depth: 210 11Motivation12----------13 14There are two distinctive motivators for this work, which are not satisfied by15existing perf buffer, which prompted creation of a new ring buffer16implementation.17 18- more efficient memory utilization by sharing ring buffer across CPUs;19- preserving ordering of events that happen sequentially in time, even across20 multiple CPUs (e.g., fork/exec/exit events for a task).21 22These two problems are independent, but perf buffer fails to satisfy both.23Both are a result of a choice to have per-CPU perf ring buffer. Both can be24also solved by having an MPSC implementation of ring buffer. The ordering25problem could technically be solved for perf buffer with some in-kernel26counting, but given the first one requires an MPSC buffer, the same solution27would solve the second problem automatically.28 29Semantics and APIs30------------------31 32Single ring buffer is presented to BPF programs as an instance of BPF map of33type ``BPF_MAP_TYPE_RINGBUF``. Two other alternatives considered, but34ultimately rejected.35 36One way would be to, similar to ``BPF_MAP_TYPE_PERF_EVENT_ARRAY``, make37``BPF_MAP_TYPE_RINGBUF`` could represent an array of ring buffers, but not38enforce "same CPU only" rule. This would be more familiar interface compatible39with existing perf buffer use in BPF, but would fail if application needed more40advanced logic to lookup ring buffer by arbitrary key.41``BPF_MAP_TYPE_HASH_OF_MAPS`` addresses this with current approach.42Additionally, given the performance of BPF ringbuf, many use cases would just43opt into a simple single ring buffer shared among all CPUs, for which current44approach would be an overkill.45 46Another approach could introduce a new concept, alongside BPF map, to represent47generic "container" object, which doesn't necessarily have key/value interface48with lookup/update/delete operations. This approach would add a lot of extra49infrastructure that has to be built for observability and verifier support. It50would also add another concept that BPF developers would have to familiarize51themselves with, new syntax in libbpf, etc. But then would really provide no52additional benefits over the approach of using a map. ``BPF_MAP_TYPE_RINGBUF``53doesn't support lookup/update/delete operations, but so doesn't few other map54types (e.g., queue and stack; array doesn't support delete, etc).55 56The approach chosen has an advantage of re-using existing BPF map57infrastructure (introspection APIs in kernel, libbpf support, etc), being58familiar concept (no need to teach users a new type of object in BPF program),59and utilizing existing tooling (bpftool). For common scenario of using a single60ring buffer for all CPUs, it's as simple and straightforward, as would be with61a dedicated "container" object. On the other hand, by being a map, it can be62combined with ``ARRAY_OF_MAPS`` and ``HASH_OF_MAPS`` map-in-maps to implement63a wide variety of topologies, from one ring buffer for each CPU (e.g., as64a replacement for perf buffer use cases), to a complicated application65hashing/sharding of ring buffers (e.g., having a small pool of ring buffers66with hashed task's tgid being a look up key to preserve order, but reduce67contention).68 69Key and value sizes are enforced to be zero. ``max_entries`` is used to specify70the size of ring buffer and has to be a power of 2 value.71 72There are a bunch of similarities between perf buffer73(``BPF_MAP_TYPE_PERF_EVENT_ARRAY``) and new BPF ring buffer semantics:74 75- variable-length records;76- if there is no more space left in ring buffer, reservation fails, no77 blocking;78- memory-mappable data area for user-space applications for ease of79 consumption and high performance;80- epoll notifications for new incoming data;81- but still the ability to do busy polling for new data to achieve the82 lowest latency, if necessary.83 84BPF ringbuf provides two sets of APIs to BPF programs:85 86- ``bpf_ringbuf_output()`` allows to *copy* data from one place to a ring87 buffer, similarly to ``bpf_perf_event_output()``;88- ``bpf_ringbuf_reserve()``/``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()``89 APIs split the whole process into two steps. First, a fixed amount of space90 is reserved. If successful, a pointer to a data inside ring buffer data91 area is returned, which BPF programs can use similarly to a data inside92 array/hash maps. Once ready, this piece of memory is either committed or93 discarded. Discard is similar to commit, but makes consumer ignore the94 record.95 96``bpf_ringbuf_output()`` has disadvantage of incurring extra memory copy,97because record has to be prepared in some other place first. But it allows to98submit records of the length that's not known to verifier beforehand. It also99closely matches ``bpf_perf_event_output()``, so will simplify migration100significantly.101 102``bpf_ringbuf_reserve()`` avoids the extra copy of memory by providing a memory103pointer directly to ring buffer memory. In a lot of cases records are larger104than BPF stack space allows, so many programs have use extra per-CPU array as105a temporary heap for preparing sample. bpf_ringbuf_reserve() avoid this needs106completely. But in exchange, it only allows a known constant size of memory to107be reserved, such that verifier can verify that BPF program can't access memory108outside its reserved record space. bpf_ringbuf_output(), while slightly slower109due to extra memory copy, covers some use cases that are not suitable for110``bpf_ringbuf_reserve()``.111 112The difference between commit and discard is very small. Discard just marks113a record as discarded, and such records are supposed to be ignored by consumer114code. Discard is useful for some advanced use-cases, such as ensuring115all-or-nothing multi-record submission, or emulating temporary116``malloc()``/``free()`` within single BPF program invocation.117 118Each reserved record is tracked by verifier through existing119reference-tracking logic, similar to socket ref-tracking. It is thus120impossible to reserve a record, but forget to submit (or discard) it.121 122``bpf_ringbuf_query()`` helper allows to query various properties of ring123buffer. Currently 4 are supported:124 125- ``BPF_RB_AVAIL_DATA`` returns amount of unconsumed data in ring buffer;126- ``BPF_RB_RING_SIZE`` returns the size of ring buffer;127- ``BPF_RB_CONS_POS``/``BPF_RB_PROD_POS`` returns current logical position128 of consumer/producer, respectively.129 130Returned values are momentarily snapshots of ring buffer state and could be131off by the time helper returns, so this should be used only for132debugging/reporting reasons or for implementing various heuristics, that take133into account highly-changeable nature of some of those characteristics.134 135One such heuristic might involve more fine-grained control over poll/epoll136notifications about new data availability in ring buffer. Together with137``BPF_RB_NO_WAKEUP``/``BPF_RB_FORCE_WAKEUP`` flags for output/commit/discard138helpers, it allows BPF program a high degree of control and, e.g., more139efficient batched notifications. Default self-balancing strategy, though,140should be adequate for most applications and will work reliable and efficiently141already.142 143Design and Implementation144-------------------------145 146This reserve/commit schema allows a natural way for multiple producers, either147on different CPUs or even on the same CPU/in the same BPF program, to reserve148independent records and work with them without blocking other producers. This149means that if BPF program was interrupted by another BPF program sharing the150same ring buffer, they will both get a record reserved (provided there is151enough space left) and can work with it and submit it independently. This152applies to NMI context as well, except that due to using a spinlock during153reservation, in NMI context, ``bpf_ringbuf_reserve()`` might fail to get154a lock, in which case reservation will fail even if ring buffer is not full.155 156The ring buffer itself internally is implemented as a power-of-2 sized157circular buffer, with two logical and ever-increasing counters (which might158wrap around on 32-bit architectures, that's not a problem):159 160- consumer counter shows up to which logical position consumer consumed the161 data;162- producer counter denotes amount of data reserved by all producers.163 164Each time a record is reserved, producer that "owns" the record will165successfully advance producer counter. At that point, data is still not yet166ready to be consumed, though. Each record has 8 byte header, which contains the167length of reserved record, as well as two extra bits: busy bit to denote that168record is still being worked on, and discard bit, which might be set at commit169time if record is discarded. In the latter case, consumer is supposed to skip170the record and move on to the next one. Record header also encodes record's171relative offset from the beginning of ring buffer data area (in pages). This172allows ``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()`` to accept only the173pointer to the record itself, without requiring also the pointer to ring buffer174itself. Ring buffer memory location will be restored from record metadata175header. This significantly simplifies verifier, as well as improving API176usability.177 178Producer counter increments are serialized under spinlock, so there is179a strict ordering between reservations. Commits, on the other hand, are180completely lockless and independent. All records become available to consumer181in the order of reservations, but only after all previous records where182already committed. It is thus possible for slow producers to temporarily hold183off submitted records, that were reserved later.184 185One interesting implementation bit, that significantly simplifies (and thus186speeds up as well) implementation of both producers and consumers is how data187area is mapped twice contiguously back-to-back in the virtual memory. This188allows to not take any special measures for samples that have to wrap around189at the end of the circular buffer data area, because the next page after the190last data page would be first data page again, and thus the sample will still191appear completely contiguous in virtual memory. See comment and a simple ASCII192diagram showing this visually in ``bpf_ringbuf_area_alloc()``.193 194Another feature that distinguishes BPF ringbuf from perf ring buffer is195a self-pacing notifications of new data being availability.196``bpf_ringbuf_commit()`` implementation will send a notification of new record197being available after commit only if consumer has already caught up right up to198the record being committed. If not, consumer still has to catch up and thus199will see new data anyways without needing an extra poll notification.200Benchmarks (see tools/testing/selftests/bpf/benchs/bench_ringbufs.c) show that201this allows to achieve a very high throughput without having to resort to202tricks like "notify only every Nth sample", which are necessary with perf203buffer. For extreme cases, when BPF program wants more manual control of204notifications, commit/discard/output helpers accept ``BPF_RB_NO_WAKEUP`` and205``BPF_RB_FORCE_WAKEUP`` flags, which give full control over notifications of206data availability, but require extra caution and diligence in using this API.207