263 lines · plain
1.. SPDX-License-Identifier: GPL-2.02.. _representors:3 4=============================5Network Function Representors6=============================7 8This document describes the semantics and usage of representor netdevices, as9used to control internal switching on SmartNICs. For the closely-related port10representors on physical (multi-port) switches, see11:ref:`Documentation/networking/switchdev.rst <switchdev>`.12 13Motivation14----------15 16Since the mid-2010s, network cards have started offering more complex17virtualisation capabilities than the legacy SR-IOV approach (with its simple18MAC/VLAN-based switching model) can support. This led to a desire to offload19software-defined networks (such as OpenVSwitch) to these NICs to specify the20network connectivity of each function. The resulting designs are variously21called SmartNICs or DPUs.22 23Network function representors bring the standard Linux networking stack to24virtual switches and IOV devices. Just as each physical port of a Linux-25controlled switch has a separate netdev, so does each virtual port of a virtual26switch.27When the system boots, and before any offload is configured, all packets from28the virtual functions appear in the networking stack of the PF via the29representors. The PF can thus always communicate freely with the virtual30functions.31The PF can configure standard Linux forwarding between representors, the uplink32or any other netdev (routing, bridging, TC classifiers).33 34Thus, a representor is both a control plane object (representing the function in35administrative commands) and a data plane object (one end of a virtual pipe).36As a virtual link endpoint, the representor can be configured like any other37netdevice; in some cases (e.g. link state) the representee will follow the38representor's configuration, while in others there are separate APIs to39configure the representee.40 41Definitions42-----------43 44This document uses the term "switchdev function" to refer to the PCIe function45which has administrative control over the virtual switch on the device.46Typically, this will be a PF, but conceivably a NIC could be configured to grant47these administrative privileges instead to a VF or SF (subfunction).48Depending on NIC design, a multi-port NIC might have a single switchdev function49for the whole device or might have a separate virtual switch, and hence50switchdev function, for each physical network port.51If the NIC supports nested switching, there might be separate switchdev52functions for each nested switch, in which case each switchdev function should53only create representors for the ports on the (sub-)switch it directly54administers.55 56A "representee" is the object that a representor represents. So for example in57the case of a VF representor, the representee is the corresponding VF.58 59What does a representor do?60---------------------------61 62A representor has three main roles.63 641. It is used to configure the network connection the representee sees, e.g.65 link up/down, MTU, etc. For instance, bringing the representor66 administratively UP should cause the representee to see a link up / carrier67 on event.682. It provides the slow path for traffic which does not hit any offloaded69 fast-path rules in the virtual switch. Packets transmitted on the70 representor netdevice should be delivered to the representee; packets71 transmitted by the representee which fail to match any switching rule should72 be received on the representor netdevice. (That is, there is a virtual pipe73 connecting the representor to the representee, similar in concept to a veth74 pair.)75 This allows software switch implementations (such as OpenVSwitch or a Linux76 bridge) to forward packets between representees and the rest of the network.773. It acts as a handle by which switching rules (such as TC filters) can refer78 to the representee, allowing these rules to be offloaded.79 80The combination of 2) and 3) means that the behaviour (apart from performance)81should be the same whether a TC filter is offloaded or not. E.g. a TC rule82on a VF representor applies in software to packets received on that representor83netdevice, while in hardware offload it would apply to packets transmitted by84the representee VF. Conversely, a mirred egress redirect to a VF representor85corresponds in hardware to delivery directly to the representee VF.86 87What functions should have a representor?88-----------------------------------------89 90Essentially, for each virtual port on the device's internal switch, there91should be a representor.92Some vendors have chosen to omit representors for the uplink and the physical93network port, which can simplify usage (the uplink netdev becomes in effect the94physical port's representor) but does not generalise to devices with multiple95ports or uplinks.96 97Thus, the following should all have representors:98 99 - VFs belonging to the switchdev function.100 - Other PFs on the local PCIe controller, and any VFs belonging to them.101 - PFs and VFs on external PCIe controllers on the device (e.g. for any embedded102 System-on-Chip within the SmartNIC).103 - PFs and VFs with other personalities, including network block devices (such104 as a vDPA virtio-blk PF backed by remote/distributed storage), if (and only105 if) their network access is implemented through a virtual switch port. [#]_106 Note that such functions can require a representor despite the representee107 not having a netdev.108 - Subfunctions (SFs) belonging to any of the above PFs or VFs, if they have109 their own port on the switch (as opposed to using their parent PF's port).110 - Any accelerators or plugins on the device whose interface to the network is111 through a virtual switch port, even if they do not have a corresponding PCIe112 PF or VF.113 114This allows the entire switching behaviour of the NIC to be controlled through115representor TC rules.116 117It is a common misunderstanding to conflate virtual ports with PCIe virtual118functions or their netdevs. While in simple cases there will be a 1:1119correspondence between VF netdevices and VF representors, more advanced device120configurations may not follow this.121A PCIe function which does not have network access through the internal switch122(not even indirectly through the hardware implementation of whatever services123the function provides) should *not* have a representor (even if it has a124netdev).125Such a function has no switch virtual port for the representor to configure or126to be the other end of the virtual pipe.127The representor represents the virtual port, not the PCIe function nor the 'end128user' netdevice.129 130.. [#] The concept here is that a hardware IP stack in the device performs the131 translation between block DMA requests and network packets, so that only132 network packets pass through the virtual port onto the switch. The network133 access that the IP stack "sees" would then be configurable through tc rules;134 e.g. its traffic might all be wrapped in a specific VLAN or VxLAN. However,135 any needed configuration of the block device *qua* block device, not being a136 networking entity, would not be appropriate for the representor and would137 thus use some other channel such as devlink.138 Contrast this with the case of a virtio-blk implementation which forwards the139 DMA requests unchanged to another PF whose driver then initiates and140 terminates IP traffic in software; in that case the DMA traffic would *not*141 run over the virtual switch and the virtio-blk PF should thus *not* have a142 representor.143 144How are representors created?145-----------------------------146 147The driver instance attached to the switchdev function should, for each virtual148port on the switch, create a pure-software netdevice which has some form of149in-kernel reference to the switchdev function's own netdevice or driver private150data (``netdev_priv()``).151This may be by enumerating ports at probe time, reacting dynamically to the152creation and destruction of ports at run time, or a combination of the two.153 154The operations of the representor netdevice will generally involve acting155through the switchdev function. For example, ``ndo_start_xmit()`` might send156the packet through a hardware TX queue attached to the switchdev function, with157either packet metadata or queue configuration marking it for delivery to the158representee.159 160How are representors identified?161--------------------------------162 163The representor netdevice should *not* directly refer to a PCIe device (e.g.164through ``net_dev->dev.parent`` / ``SET_NETDEV_DEV()``), either of the165representee or of the switchdev function.166Instead, the driver should use the ``SET_NETDEV_DEVLINK_PORT`` macro to167assign a devlink port instance to the netdevice before registering the168netdevice; the kernel uses the devlink port to provide the ``phys_switch_id``169and ``phys_port_name`` sysfs nodes.170(Some legacy drivers implement ``ndo_get_port_parent_id()`` and171``ndo_get_phys_port_name()`` directly, but this is deprecated.) See172:ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>` for the173details of this API.174 175It is expected that userland will use this information (e.g. through udev rules)176to construct an appropriately informative name or alias for the netdevice. For177instance if the switchdev function is ``eth4`` then a representor with a178``phys_port_name`` of ``p0pf1vf2`` might be renamed ``eth4pf1vf2rep``.179 180There are as yet no established conventions for naming representors which do not181correspond to PCIe functions (e.g. accelerators and plugins).182 183How do representors interact with TC rules?184-------------------------------------------185 186Any TC rule on a representor applies (in software TC) to packets received by187that representor netdevice. Thus, if the delivery part of the rule corresponds188to another port on the virtual switch, the driver may choose to offload it to189hardware, applying it to packets transmitted by the representee.190 191Similarly, since a TC mirred egress action targeting the representor would (in192software) send the packet through the representor (and thus indirectly deliver193it to the representee), hardware offload should interpret this as delivery to194the representee.195 196As a simple example, if ``PORT_DEV`` is the physical port representor and197``REP_DEV`` is a VF representor, the following rules::198 199 tc filter add dev $REP_DEV parent ffff: protocol ipv4 flower \200 action mirred egress redirect dev $PORT_DEV201 tc filter add dev $PORT_DEV parent ffff: protocol ipv4 flower skip_sw \202 action mirred egress mirror dev $REP_DEV203 204would mean that all IPv4 packets from the VF are sent out the physical port, and205all IPv4 packets received on the physical port are delivered to the VF in206addition to ``PORT_DEV``. (Note that without ``skip_sw`` on the second rule,207the VF would get two copies, as the packet reception on ``PORT_DEV`` would208trigger the TC rule again and mirror the packet to ``REP_DEV``.)209 210On devices without separate port and uplink representors, ``PORT_DEV`` would211instead be the switchdev function's own uplink netdevice.212 213Of course the rules can (if supported by the NIC) include packet-modifying214actions (e.g. VLAN push/pop), which should be performed by the virtual switch.215 216Tunnel encapsulation and decapsulation are rather more complicated, as they217involve a third netdevice (a tunnel netdev operating in metadata mode, such as218a VxLAN device created with ``ip link add vxlan0 type vxlan external``) and219require an IP address to be bound to the underlay device (e.g. switchdev220function uplink netdev or port representor). TC rules such as::221 222 tc filter add dev $REP_DEV parent ffff: flower \223 action tunnel_key set id $VNI src_ip $LOCAL_IP dst_ip $REMOTE_IP \224 dst_port 4789 \225 action mirred egress redirect dev vxlan0226 tc filter add dev vxlan0 parent ffff: flower enc_src_ip $REMOTE_IP \227 enc_dst_ip $LOCAL_IP enc_key_id $VNI enc_dst_port 4789 \228 action tunnel_key unset action mirred egress redirect dev $REP_DEV229 230where ``LOCAL_IP`` is an IP address bound to ``PORT_DEV``, and ``REMOTE_IP`` is231another IP address on the same subnet, mean that packets sent by the VF should232be VxLAN encapsulated and sent out the physical port (the driver has to deduce233this by a route lookup of ``LOCAL_IP`` leading to ``PORT_DEV``, and also234perform an ARP/neighbour table lookup to find the MAC addresses to use in the235outer Ethernet frame), while UDP packets received on the physical port with UDP236port 4789 should be parsed as VxLAN and, if their VSID matches ``$VNI``,237decapsulated and forwarded to the VF.238 239If this all seems complicated, just remember the 'golden rule' of TC offload:240the hardware should ensure the same final results as if the packets were241processed through the slow path, traversed software TC (except ignoring any242``skip_hw`` rules and applying any ``skip_sw`` rules) and were transmitted or243received through the representor netdevices.244 245Configuring the representee's MAC246---------------------------------247 248The representee's link state is controlled through the representor. Setting the249representor administratively UP or DOWN should cause carrier ON or OFF at the250representee.251 252Setting an MTU on the representor should cause that same MTU to be reported to253the representee.254(On hardware that allows configuring separate and distinct MTU and MRU values,255the representor MTU should correspond to the representee's MRU and vice-versa.)256 257Currently there is no way to use the representor to set the station permanent258MAC address of the representee; other methods available to do this include:259 260 - legacy SR-IOV (``ip link set DEVICE vf NUM mac LLADDR``)261 - devlink port function (see **devlink-port(8)** and262 :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`)263