850 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3======4AF_XDP5======6 7Overview8========9 10AF_XDP is an address family that is optimized for high performance11packet processing.12 13This document assumes that the reader is familiar with BPF and XDP. If14not, the Cilium project has an excellent reference guide at15http://cilium.readthedocs.io/en/latest/bpf/.16 17Using the XDP_REDIRECT action from an XDP program, the program can18redirect ingress frames to other XDP enabled netdevs, using the19bpf_redirect_map() function. AF_XDP sockets enable the possibility for20XDP programs to redirect frames to a memory buffer in a user-space21application.22 23An AF_XDP socket (XSK) is created with the normal socket()24syscall. Associated with each XSK are two rings: the RX ring and the25TX ring. A socket can receive packets on the RX ring and it can send26packets on the TX ring. These rings are registered and sized with the27setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory28to have at least one of these rings for each socket. An RX or TX29descriptor ring points to a data buffer in a memory area called a30UMEM. RX and TX can share the same UMEM so that a packet does not have31to be copied between RX and TX. Moreover, if a packet needs to be kept32for a while due to a possible retransmit, the descriptor that points33to that packet can be changed to point to another and reused right34away. This again avoids copying data.35 36The UMEM consists of a number of equally sized chunks. A descriptor in37one of the rings references a frame by referencing its addr. The addr38is simply an offset within the entire UMEM region. The user space39allocates memory for this UMEM using whatever means it feels is most40appropriate (malloc, mmap, huge pages, etc). This memory area is then41registered with the kernel using the new setsockopt XDP_UMEM_REG. The42UMEM also has two rings: the FILL ring and the COMPLETION ring. The43FILL ring is used by the application to send down addr for the kernel44to fill in with RX packet data. References to these frames will then45appear in the RX ring once each packet has been received. The46COMPLETION ring, on the other hand, contains frame addr that the47kernel has transmitted completely and can now be used again by user48space, for either TX or RX. Thus, the frame addrs appearing in the49COMPLETION ring are addrs that were previously transmitted using the50TX ring. In summary, the RX and FILL rings are used for the RX path51and the TX and COMPLETION rings are used for the TX path.52 53The socket is then finally bound with a bind() call to a device and a54specific queue id on that device, and it is not until bind is55completed that traffic starts to flow.56 57The UMEM can be shared between processes, if desired. If a process58wants to do this, it simply skips the registration of the UMEM and its59corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind60call and submits the XSK of the process it would like to share UMEM61with as well as its own newly created XSK socket. The new process will62then receive frame addr references in its own RX ring that point to63this shared UMEM. Note that since the ring structures are64single-consumer / single-producer (for performance reasons), the new65process has to create its own socket with associated RX and TX rings,66since it cannot share this with the other process. This is also the67reason that there is only one set of FILL and COMPLETION rings per68UMEM. It is the responsibility of a single process to handle the UMEM.69 70How is then packets distributed from an XDP program to the XSKs? There71is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The72user-space application can place an XSK at an arbitrary place in this73map. The XDP program can then redirect a packet to a specific index in74this map and at this point XDP validates that the XSK in that map was75indeed bound to that device and ring number. If not, the packet is76dropped. If the map is empty at that index, the packet is also77dropped. This also means that it is currently mandatory to have an XDP78program loaded (and one XSK in the XSKMAP) to be able to get any79traffic to user space through the XSK.80 81AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the82driver does not have support for XDP, or XDP_SKB is explicitly chosen83when loading the XDP program, XDP_SKB mode is employed that uses SKBs84together with the generic XDP support and copies out the data to user85space. A fallback mode that works for any network device. On the other86hand, if the driver has support for XDP, it will be used by the AF_XDP87code to provide better performance, but there is still a copy of the88data into user space.89 90Concepts91========92 93In order to use an AF_XDP socket, a number of associated objects need94to be setup. These objects and their options are explained in the95following sections.96 97For an overview on how AF_XDP works, you can also take a look at the98Linux Plumbers paper from 2018 on the subject:99http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do100NOT consult the paper from 2017 on "AF_PACKET v4", the first attempt101at AF_XDP. Nearly everything changed since then. Jonathan Corbet has102also written an excellent article on LWN, "Accelerating networking103with AF_XDP". It can be found at https://lwn.net/Articles/750845/.104 105UMEM106----107 108UMEM is a region of virtual contiguous memory, divided into109equal-sized frames. An UMEM is associated to a netdev and a specific110queue id of that netdev. It is created and configured (chunk size,111headroom, start address and size) by using the XDP_UMEM_REG setsockopt112system call. A UMEM is bound to a netdev and queue id, via the bind()113system call.114 115An AF_XDP is socket linked to a single UMEM, but one UMEM can have116multiple AF_XDP sockets. To share an UMEM created via one socket A,117the next socket B can do this by setting the XDP_SHARED_UMEM flag in118struct sockaddr_xdp member sxdp_flags, and passing the file descriptor119of A to struct sockaddr_xdp member sxdp_shared_umem_fd.120 121The UMEM has two single-producer/single-consumer rings that are used122to transfer ownership of UMEM frames between the kernel and the123user-space application.124 125Rings126-----127 128There are a four different kind of rings: FILL, COMPLETION, RX and129TX. All rings are single-producer/single-consumer, so the user-space130application need explicit synchronization of multiple131processes/threads are reading/writing to them.132 133The UMEM uses two rings: FILL and COMPLETION. Each socket associated134with the UMEM must have an RX queue, TX queue or both. Say, that there135is a setup with four sockets (all doing TX and RX). Then there will be136one FILL ring, one COMPLETION ring, four TX rings and four RX rings.137 138The rings are head(producer)/tail(consumer) based rings. A producer139writes the data ring at the index pointed out by struct xdp_ring140producer member, and increasing the producer index. A consumer reads141the data ring at the index pointed out by struct xdp_ring consumer142member, and increasing the consumer index.143 144The rings are configured and created via the _RING setsockopt system145calls and mmapped to user-space using the appropriate offset to mmap()146(XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and147XDP_UMEM_PGOFF_COMPLETION_RING).148 149The size of the rings need to be of size power of two.150 151UMEM Fill Ring152~~~~~~~~~~~~~~153 154The FILL ring is used to transfer ownership of UMEM frames from155user-space to kernel-space. The UMEM addrs are passed in the ring. As156an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has15716 chunks and can pass addrs between 0 and 64k.158 159Frames passed to the kernel are used for the ingress path (RX rings).160 161The user application produces UMEM addrs to this ring. Note that, if162running the application with aligned chunk mode, the kernel will mask163the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of164the addr will be masked off, meaning that 2048, 2050 and 3000 refers165to the same chunk. If the user application is run in the unaligned166chunks mode, then the incoming addr will be left untouched.167 168 169UMEM Completion Ring170~~~~~~~~~~~~~~~~~~~~171 172The COMPLETION Ring is used transfer ownership of UMEM frames from173kernel-space to user-space. Just like the FILL ring, UMEM indices are174used.175 176Frames passed from the kernel to user-space are frames that has been177sent (TX ring) and can be used by user-space again.178 179The user application consumes UMEM addrs from this ring.180 181 182RX Ring183~~~~~~~184 185The RX ring is the receiving side of a socket. Each entry in the ring186is a struct xdp_desc descriptor. The descriptor contains UMEM offset187(addr) and the length of the data (len).188 189If no frames have been passed to kernel via the FILL ring, no190descriptors will (or can) appear on the RX ring.191 192The user application consumes struct xdp_desc descriptors from this193ring.194 195TX Ring196~~~~~~~197 198The TX ring is used to send frames. The struct xdp_desc descriptor is199filled (index, length and offset) and passed into the ring.200 201To start the transfer a sendmsg() system call is required. This might202be relaxed in the future.203 204The user application produces struct xdp_desc descriptors to this205ring.206 207Libbpf208======209 210Libbpf is a helper library for eBPF and XDP that makes using these211technologies a lot simpler. It also contains specific helper functions212in tools/lib/bpf/xsk.h for facilitating the use of AF_XDP. It213contains two types of functions: those that can be used to make the214setup of AF_XDP socket easier and ones that can be used in the data215plane to access the rings safely and quickly. To see an example on how216to use this API, please take a look at the sample application in217samples/bpf/xdpsock_usr.c which uses libbpf for both setup and data218plane operations.219 220We recommend that you use this library unless you have become a power221user. It will make your program a lot simpler.222 223XSKMAP / BPF_MAP_TYPE_XSKMAP224============================225 226On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that227is used in conjunction with bpf_redirect_map() to pass the ingress228frame to a socket.229 230The user application inserts the socket into the map, via the bpf()231system call.232 233Note that if an XDP program tries to redirect to a socket that does234not match the queue configuration and netdev, the frame will be235dropped. E.g. an AF_XDP socket is bound to netdev eth0 and236queue 17. Only the XDP program executing for eth0 and queue 17 will237successfully pass data to the socket. Please refer to the sample238application (samples/bpf/) in for an example.239 240Configuration Flags and Socket Options241======================================242 243These are the various configuration flags that can be used to control244and monitor the behavior of AF_XDP sockets.245 246XDP_COPY and XDP_ZEROCOPY bind flags247------------------------------------248 249When you bind to a socket, the kernel will first try to use zero-copy250copy. If zero-copy is not supported, it will fall back on using copy251mode, i.e. copying all packets out to user space. But if you would252like to force a certain mode, you can use the following flags. If you253pass the XDP_COPY flag to the bind call, the kernel will force the254socket into copy mode. If it cannot use copy mode, the bind call will255fail with an error. Conversely, the XDP_ZEROCOPY flag will force the256socket into zero-copy mode or fail.257 258XDP_SHARED_UMEM bind flag259-------------------------260 261This flag enables you to bind multiple sockets to the same UMEM. It262works on the same queue id, between queue ids and between263netdevs/devices. In this mode, each socket has their own RX and TX264rings as usual, but you are going to have one or more FILL and265COMPLETION ring pairs. You have to create one of these pairs per266unique netdev and queue id tuple that you bind to.267 268Starting with the case were we would like to share a UMEM between269sockets bound to the same netdev and queue id. The UMEM (tied to the270fist socket created) will only have a single FILL ring and a single271COMPLETION ring as there is only on unique netdev,queue_id tuple that272we have bound to. To use this mode, create the first socket and bind273it in the normal way. Create a second socket and create an RX and a TX274ring, or at least one of them, but no FILL or COMPLETION rings as the275ones from the first socket will be used. In the bind call, set he276XDP_SHARED_UMEM option and provide the initial socket's fd in the277sxdp_shared_umem_fd field. You can attach an arbitrary number of extra278sockets this way.279 280What socket will then a packet arrive on? This is decided by the XDP281program. Put all the sockets in the XSK_MAP and just indicate which282index in the array you would like to send each packet to. A simple283round-robin example of distributing packets is shown below:284 285.. code-block:: c286 287 #include <linux/bpf.h>288 #include "bpf_helpers.h"289 290 #define MAX_SOCKS 16291 292 struct {293 __uint(type, BPF_MAP_TYPE_XSKMAP);294 __uint(max_entries, MAX_SOCKS);295 __uint(key_size, sizeof(int));296 __uint(value_size, sizeof(int));297 } xsks_map SEC(".maps");298 299 static unsigned int rr;300 301 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)302 {303 rr = (rr + 1) & (MAX_SOCKS - 1);304 305 return bpf_redirect_map(&xsks_map, rr, XDP_DROP);306 }307 308Note, that since there is only a single set of FILL and COMPLETION309rings, and they are single producer, single consumer rings, you need310to make sure that multiple processes or threads do not use these rings311concurrently. There are no synchronization primitives in the312libbpf code that protects multiple users at this point in time.313 314Libbpf uses this mode if you create more than one socket tied to the315same UMEM. However, note that you need to supply the316XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the317xsk_socket__create calls and load your own XDP program as there is no318built in one in libbpf that will route the traffic for you.319 320The second case is when you share a UMEM between sockets that are321bound to different queue ids and/or netdevs. In this case you have to322create one FILL ring and one COMPLETION ring for each unique323netdev,queue_id pair. Let us say you want to create two sockets bound324to two different queue ids on the same netdev. Create the first socket325and bind it in the normal way. Create a second socket and create an RX326and a TX ring, or at least one of them, and then one FILL and327COMPLETION ring for this socket. Then in the bind call, set he328XDP_SHARED_UMEM option and provide the initial socket's fd in the329sxdp_shared_umem_fd field as you registered the UMEM on that330socket. These two sockets will now share one and the same UMEM.331 332There is no need to supply an XDP program like the one in the previous333case where sockets were bound to the same queue id and334device. Instead, use the NIC's packet steering capabilities to steer335the packets to the right queue. In the previous example, there is only336one queue shared among sockets, so the NIC cannot do this steering. It337can only steer between queues.338 339In libbpf, you need to use the xsk_socket__create_shared() API as it340takes a reference to a FILL ring and a COMPLETION ring that will be341created for you and bound to the shared UMEM. You can use this342function for all the sockets you create, or you can use it for the343second and following ones and use xsk_socket__create() for the first344one. Both methods yield the same result.345 346Note that a UMEM can be shared between sockets on the same queue id347and device, as well as between queues on the same device and between348devices at the same time.349 350XDP_USE_NEED_WAKEUP bind flag351-----------------------------352 353This option adds support for a new flag called need_wakeup that is354present in the FILL ring and the TX ring, the rings for which user355space is a producer. When this option is set in the bind call, the356need_wakeup flag will be set if the kernel needs to be explicitly357woken up by a syscall to continue processing packets. If the flag is358zero, no syscall is needed.359 360If the flag is set on the FILL ring, the application needs to call361poll() to be able to continue to receive packets on the RX ring. This362can happen, for example, when the kernel has detected that there are no363more buffers on the FILL ring and no buffers left on the RX HW ring of364the NIC. In this case, interrupts are turned off as the NIC cannot365receive any packets (as there are no buffers to put them in), and the366need_wakeup flag is set so that user space can put buffers on the367FILL ring and then call poll() so that the kernel driver can put these368buffers on the HW ring and start to receive packets.369 370If the flag is set for the TX ring, it means that the application371needs to explicitly notify the kernel to send any packets put on the372TX ring. This can be accomplished either by a poll() call, as in the373RX path, or by calling sendto().374 375An example of how to use this flag can be found in376samples/bpf/xdpsock_user.c. An example with the use of libbpf helpers377would look like this for the TX path:378 379.. code-block:: c380 381 if (xsk_ring_prod__needs_wakeup(&my_tx_ring))382 sendto(xsk_socket__fd(xsk_handle), NULL, 0, MSG_DONTWAIT, NULL, 0);383 384I.e., only use the syscall if the flag is set.385 386We recommend that you always enable this mode as it usually leads to387better performance especially if you run the application and the388driver on the same core, but also if you use different cores for the389application and the kernel driver, as it reduces the number of390syscalls needed for the TX path.391 392XDP_{RX|TX|UMEM_FILL|UMEM_COMPLETION}_RING setsockopts393------------------------------------------------------394 395These setsockopts sets the number of descriptors that the RX, TX,396FILL, and COMPLETION rings respectively should have. It is mandatory397to set the size of at least one of the RX and TX rings. If you set398both, you will be able to both receive and send traffic from your399application, but if you only want to do one of them, you can save400resources by only setting up one of them. Both the FILL ring and the401COMPLETION ring are mandatory as you need to have a UMEM tied to your402socket. But if the XDP_SHARED_UMEM flag is used, any socket after the403first one does not have a UMEM and should in that case not have any404FILL or COMPLETION rings created as the ones from the shared UMEM will405be used. Note, that the rings are single-producer single-consumer, so406do not try to access them from multiple processes at the same407time. See the XDP_SHARED_UMEM section.408 409In libbpf, you can create Rx-only and Tx-only sockets by supplying410NULL to the rx and tx arguments, respectively, to the411xsk_socket__create function.412 413If you create a Tx-only socket, we recommend that you do not put any414packets on the fill ring. If you do this, drivers might think you are415going to receive something when you in fact will not, and this can416negatively impact performance.417 418XDP_UMEM_REG setsockopt419-----------------------420 421This setsockopt registers a UMEM to a socket. This is the area that422contain all the buffers that packet can reside in. The call takes a423pointer to the beginning of this area and the size of it. Moreover, it424also has parameter called chunk_size that is the size that the UMEM is425divided into. It can only be 2K or 4K at the moment. If you have an426UMEM area that is 128K and a chunk size of 2K, this means that you427will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM428area and that your largest packet size can be 2K.429 430There is also an option to set the headroom of each single buffer in431the UMEM. If you set this to N bytes, it means that the packet will432start N bytes into the buffer leaving the first N bytes for the433application to use. The final option is the flags field, but it will434be dealt with in separate sections for each UMEM flag.435 436SO_BINDTODEVICE setsockopt437--------------------------438 439This is a generic SOL_SOCKET option that can be used to tie AF_XDP440socket to a particular network interface. It is useful when a socket441is created by a privileged process and passed to a non-privileged one.442Once the option is set, kernel will refuse attempts to bind that socket443to a different interface. Updating the value requires CAP_NET_RAW.444 445XDP_STATISTICS getsockopt446-------------------------447 448Gets drop statistics of a socket that can be useful for debug449purposes. The supported statistics are shown below:450 451.. code-block:: c452 453 struct xdp_statistics {454 __u64 rx_dropped; /* Dropped for reasons other than invalid desc */455 __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */456 __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */457 };458 459XDP_OPTIONS getsockopt460----------------------461 462Gets options from an XDP socket. The only one supported so far is463XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not.464 465Multi-Buffer Support466====================467 468With multi-buffer support, programs using AF_XDP sockets can receive469and transmit packets consisting of multiple buffers both in copy and470zero-copy mode. For example, a packet can consist of two471frames/buffers, one with the header and the other one with the data,472or a 9K Ethernet jumbo frame can be constructed by chaining together473three 4K frames.474 475Some definitions:476 477* A packet consists of one or more frames478 479* A descriptor in one of the AF_XDP rings always refers to a single480 frame. In the case the packet consists of a single frame, the481 descriptor refers to the whole packet.482 483To enable multi-buffer support for an AF_XDP socket, use the new bind484flag XDP_USE_SG. If this is not provided, all multi-buffer packets485will be dropped just as before. Note that the XDP program loaded also486needs to be in multi-buffer mode. This can be accomplished by using487"xdp.frags" as the section name of the XDP program used.488 489To represent a packet consisting of multiple frames, a new flag called490XDP_PKT_CONTD is introduced in the options field of the Rx and Tx491descriptors. If it is true (1) the packet continues with the next492descriptor and if it is false (0) it means this is the last descriptor493of the packet. Why the reverse logic of end-of-packet (eop) flag found494in many NICs? Just to preserve compatibility with non-multi-buffer495applications that have this bit set to false for all packets on Rx,496and the apps set the options field to zero for Tx, as anything else497will be treated as an invalid descriptor.498 499These are the semantics for producing packets onto AF_XDP Tx ring500consisting of multiple frames:501 502* When an invalid descriptor is found, all the other503 descriptors/frames of this packet are marked as invalid and not504 completed. The next descriptor is treated as the start of a new505 packet, even if this was not the intent (because we cannot guess506 the intent). As before, if your program is producing invalid507 descriptors you have a bug that must be fixed.508 509* Zero length descriptors are treated as invalid descriptors.510 511* For copy mode, the maximum supported number of frames in a packet is512 equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all513 descriptors accumulated so far are dropped and treated as514 invalid. To produce an application that will work on any system515 regardless of this config setting, limit the number of frags to 18,516 as the minimum value of the config is 17.517 518* For zero-copy mode, the limit is up to what the NIC HW519 supports. Usually at least five on the NICs we have checked. We520 consciously chose to not enforce a rigid limit (such as521 CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have522 resulted in copy actions under the hood to fit into what limit the523 NIC supports. Kind of defeats the purpose of zero-copy mode. How to524 probe for this limit is explained in the "probe for multi-buffer525 support" section.526 527On the Rx path in copy-mode, the xsk core copies the XDP data into528multiple descriptors, if needed, and sets the XDP_PKT_CONTD flag as529detailed before. Zero-copy mode works the same, though the data is not530copied. When the application gets a descriptor with the XDP_PKT_CONTD531flag set to one, it means that the packet consists of multiple buffers532and it continues with the next buffer in the following533descriptor. When a descriptor with XDP_PKT_CONTD == 0 is received, it534means that this is the last buffer of the packet. AF_XDP guarantees535that only a complete packet (all frames in the packet) is sent to the536application. If there is not enough space in the AF_XDP Rx ring, all537frames of the packet will be dropped.538 539If application reads a batch of descriptors, using for example the libxdp540interfaces, it is not guaranteed that the batch will end with a full541packet. It might end in the middle of a packet and the rest of the542buffers of that packet will arrive at the beginning of the next batch,543since the libxdp interface does not read the whole ring (unless you544have an enormous batch size or a very small ring size).545 546An example program each for Rx and Tx multi-buffer support can be found547later in this document.548 549Usage550-----551 552In order to use AF_XDP sockets two parts are needed. The553user-space application and the XDP program. For a complete setup and554usage example, please refer to the sample application. The user-space555side is xdpsock_user.c and the XDP side is part of libbpf.556 557The XDP code sample included in tools/lib/bpf/xsk.c is the following:558 559.. code-block:: c560 561 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)562 {563 int index = ctx->rx_queue_index;564 565 // A set entry here means that the corresponding queue_id566 // has an active AF_XDP socket bound to it.567 if (bpf_map_lookup_elem(&xsks_map, &index))568 return bpf_redirect_map(&xsks_map, index, 0);569 570 return XDP_PASS;571 }572 573A simple but not so performance ring dequeue and enqueue could look574like this:575 576.. code-block:: c577 578 // struct xdp_rxtx_ring {579 // __u32 *producer;580 // __u32 *consumer;581 // struct xdp_desc *desc;582 // };583 584 // struct xdp_umem_ring {585 // __u32 *producer;586 // __u32 *consumer;587 // __u64 *desc;588 // };589 590 // typedef struct xdp_rxtx_ring RING;591 // typedef struct xdp_umem_ring RING;592 593 // typedef struct xdp_desc RING_TYPE;594 // typedef __u64 RING_TYPE;595 596 int dequeue_one(RING *ring, RING_TYPE *item)597 {598 __u32 entries = *ring->producer - *ring->consumer;599 600 if (entries == 0)601 return -1;602 603 // read-barrier!604 605 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)];606 (*ring->consumer)++;607 return 0;608 }609 610 int enqueue_one(RING *ring, const RING_TYPE *item)611 {612 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer);613 614 if (free_entries == 0)615 return -1;616 617 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item;618 619 // write-barrier!620 621 (*ring->producer)++;622 return 0;623 }624 625But please use the libbpf functions as they are optimized and ready to626use. Will make your life easier.627 628Usage Multi-Buffer Rx629---------------------630 631Here is a simple Rx path pseudo-code example (using libxdp interfaces632for simplicity). Error paths have been excluded to keep it short:633 634.. code-block:: c635 636 void rx_packets(struct xsk_socket_info *xsk)637 {638 static bool new_packet = true;639 u32 idx_rx = 0, idx_fq = 0;640 static char *pkt;641 642 int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx);643 644 xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);645 646 for (int i = 0; i < rcvd; i++) {647 struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);648 char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr);649 bool eop = !(desc->options & XDP_PKT_CONTD);650 651 if (new_packet)652 pkt = frag;653 else654 add_frag_to_pkt(pkt, frag);655 656 if (eop)657 process_pkt(pkt);658 659 new_packet = eop;660 661 *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr;662 }663 664 xsk_ring_prod__submit(&xsk->umem->fq, rcvd);665 xsk_ring_cons__release(&xsk->rx, rcvd);666 }667 668Usage Multi-Buffer Tx669---------------------670 671Here is an example Tx path pseudo-code (using libxdp interfaces for672simplicity) ignoring that the umem is finite in size, and that we673eventually will run out of packets to send. Also assumes pkts.addr674points to a valid location in the umem.675 676.. code-block:: c677 678 void tx_packets(struct xsk_socket_info *xsk, struct pkt *pkts,679 int batch_size)680 {681 u32 idx, i, pkt_nb = 0;682 683 xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx);684 685 for (i = 0; i < batch_size;) {686 u64 addr = pkts[pkt_nb].addr;687 u32 len = pkts[pkt_nb].size;688 689 do {690 struct xdp_desc *tx_desc;691 692 tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++);693 tx_desc->addr = addr;694 695 if (len > xsk_frame_size) {696 tx_desc->len = xsk_frame_size;697 tx_desc->options = XDP_PKT_CONTD;698 } else {699 tx_desc->len = len;700 tx_desc->options = 0;701 pkt_nb++;702 }703 len -= tx_desc->len;704 addr += xsk_frame_size;705 706 if (i == batch_size) {707 /* Remember len, addr, pkt_nb for next iteration.708 * Skipped for simplicity.709 */710 break;711 }712 } while (len);713 }714 715 xsk_ring_prod__submit(&xsk->tx, i);716 }717 718Probing for Multi-Buffer Support719--------------------------------720 721To discover if a driver supports multi-buffer AF_XDP in SKB or DRV722mode, use the XDP_FEATURES feature of netlink in linux/netdev.h to723query for NETDEV_XDP_ACT_RX_SG support. This is the same flag as for724querying for XDP multi-buffer support. If XDP supports multi-buffer in725a driver, then AF_XDP will also support that in SKB and DRV mode.726 727To discover if a driver supports multi-buffer AF_XDP in zero-copy728mode, use XDP_FEATURES and first check the NETDEV_XDP_ACT_XSK_ZEROCOPY729flag. If it is set, it means that at least zero-copy is supported and730you should go and check the netlink attribute731NETDEV_A_DEV_XDP_ZC_MAX_SEGS in linux/netdev.h. An unsigned integer732value will be returned stating the max number of frags that are733supported by this device in zero-copy mode. These are the possible734return values:735 7361: Multi-buffer for zero-copy is not supported by this device, as max737 one fragment supported means that multi-buffer is not possible.738 739>=2: Multi-buffer is supported in zero-copy mode for this device. The740 returned number signifies the max number of frags supported.741 742For an example on how these are used through libbpf, please take a743look at tools/testing/selftests/bpf/xskxceiver.c.744 745Multi-Buffer Support for Zero-Copy Drivers746------------------------------------------747 748Zero-copy drivers usually use the batched APIs for Rx and Tx749processing. Note that the Tx batch API guarantees that it will provide750a batch of Tx descriptors that ends with full packet at the end. This751to facilitate extending a zero-copy driver with multi-buffer support.752 753Sample application754==================755 756There is a xdpsock benchmarking/test application included that757demonstrates how to use AF_XDP sockets with private UMEMs. Say that758you would like your UDP traffic from port 4242 to end up in queue 16,759that we will enable AF_XDP on. Here, we use ethtool for this::760 761 ethtool -N p3p2 rx-flow-hash udp4 fn762 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \763 action 16764 765Running the rxdrop benchmark in XDP_DRV mode can then be done766using::767 768 samples/bpf/xdpsock -i p3p2 -q 16 -r -N769 770For XDP_SKB mode, use the switch "-S" instead of "-N" and all options771can be displayed with "-h", as usual.772 773This sample application uses libbpf to make the setup and usage of774AF_XDP simpler. If you want to know how the raw uapi of AF_XDP is775really used to make something more advanced, take a look at the libbpf776code in tools/lib/bpf/xsk.[ch].777 778FAQ779=======780 781Q: I am not seeing any traffic on the socket. What am I doing wrong?782 783A: When a netdev of a physical NIC is initialized, Linux usually784 allocates one RX and TX queue pair per core. So on a 8 core system,785 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP786 bind call or the xsk_socket__create libbpf function call, you787 specify a specific queue id to bind to and it is only the traffic788 towards that queue you are going to get on you socket. So in the789 example above, if you bind to queue 0, you are NOT going to get any790 traffic that is distributed to queues 1 through 7. If you are791 lucky, you will see the traffic, but usually it will end up on one792 of the queues you have not bound to.793 794 There are a number of ways to solve the problem of getting the795 traffic you want to the queue id you bound to. If you want to see796 all the traffic, you can force the netdev to only have 1 queue, queue797 id 0, and then bind to queue 0. You can use ethtool to do this::798 799 sudo ethtool -L <interface> combined 1800 801 If you want to only see part of the traffic, you can program the802 NIC through ethtool to filter out your traffic to a single queue id803 that you can bind your XDP socket to. Here is one example in which804 UDP traffic to and from port 4242 are sent to queue 2::805 806 sudo ethtool -N <interface> rx-flow-hash udp4 fn807 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \808 4242 action 2809 810 A number of other ways are possible all up to the capabilities of811 the NIC you have.812 813Q: Can I use the XSKMAP to implement a switch between different umems814 in copy mode?815 816A: The short answer is no, that is not supported at the moment. The817 XSKMAP can only be used to switch traffic coming in on queue id X818 to sockets bound to the same queue id X. The XSKMAP can contain819 sockets bound to different queue ids, for example X and Y, but only820 traffic goming in from queue id Y can be directed to sockets bound821 to the same queue id Y. In zero-copy mode, you should use the822 switch, or other distribution mechanism, in your NIC to direct823 traffic to the correct queue id and socket.824 825Q: My packets are sometimes corrupted. What is wrong?826 827A: Care has to be taken not to feed the same buffer in the UMEM into828 more than one ring at the same time. If you for example feed the829 same buffer into the FILL ring and the TX ring at the same time, the830 NIC might receive data into the buffer at the same time it is831 sending it. This will cause some packets to become corrupted. Same832 thing goes for feeding the same buffer into the FILL rings833 belonging to different queue ids or netdevs bound with the834 XDP_SHARED_UMEM flag.835 836Credits837=======838 839- Björn Töpel (AF_XDP core)840- Magnus Karlsson (AF_XDP core)841- Alexander Duyck842- Alexei Starovoitov843- Daniel Borkmann844- Jesper Dangaard Brouer845- John Fastabend846- Jonathan Corbet (LWN coverage)847- Michael S. Tsirkin848- Qi Z Zhang849- Willem de Bruijn850