565 lines · plain
1.. SPDX-License-Identifier: GPL-2.02.. include:: <isonum.txt>3.. _switchdev:4 5===============================================6Ethernet switch device driver model (switchdev)7===============================================8 9Copyright |copy| 2014 Jiri Pirko <jiri@resnulli.us>10 11Copyright |copy| 2014-2015 Scott Feldman <sfeldma@gmail.com>12 13 14The Ethernet switch device driver model (switchdev) is an in-kernel driver15model for switch devices which offload the forwarding (data) plane from the16kernel.17 18Figure 1 is a block diagram showing the components of the switchdev model for19an example setup using a data-center-class switch ASIC chip. Other setups20with SR-IOV or soft switches, such as OVS, are possible.21 22::23 24 25 User-space tools26 27 user space |28 +-------------------------------------------------------------------+29 kernel | Netlink30 |31 +--------------+-------------------------------+32 | Network stack |33 | (Linux) |34 | |35 +----------------------------------------------+36 37 sw1p2 sw1p4 sw1p638 sw1p1 + sw1p3 + sw1p5 + eth139 + | + | + | +40 | | | | | | |41 +--+----+----+----+----+----+---+ +-----+-----+42 | Switch driver | | mgmt |43 | (this document) | | driver |44 | | | |45 +--------------+----------------+ +-----------+46 |47 kernel | HW bus (eg PCI)48 +-------------------------------------------------------------------+49 hardware |50 +--------------+----------------+51 | Switch device (sw1) |52 | +----+ +--------+53 | | v offloaded data path | mgmt port54 | | | |55 +--|----|----+----+----+----+---+56 | | | | | |57 + + + + + +58 p1 p2 p3 p4 p5 p659 60 front-panel ports61 62 63 Fig 1.64 65 66Include Files67-------------68 69::70 71 #include <linux/netdevice.h>72 #include <net/switchdev.h>73 74 75Configuration76-------------77 78Use "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model79support is built for driver.80 81 82Switch Ports83------------84 85On switchdev driver initialization, the driver will allocate and register a86struct net_device (using register_netdev()) for each enumerated physical switch87port, called the port netdev. A port netdev is the software representation of88the physical port and provides a conduit for control traffic to/from the89controller (the kernel) and the network, as well as an anchor point for higher90level constructs such as bridges, bonds, VLANs, tunnels, and L3 routers. Using91standard netdev tools (iproute2, ethtool, etc), the port netdev can also92provide to the user access to the physical properties of the switch port such93as PHY link state and I/O statistics.94 95There is (currently) no higher-level kernel object for the switch beyond the96port netdevs. All of the switchdev driver ops are netdev ops or switchdev ops.97 98A switch management port is outside the scope of the switchdev driver model.99Typically, the management port is not participating in offloaded data plane and100is loaded with a different driver, such as a NIC driver, on the management port101device.102 103Switch ID104^^^^^^^^^105 106The switchdev driver must implement the net_device operation107ndo_get_port_parent_id for each port netdev, returning the same physical ID for108each port of a switch. The ID must be unique between switches on the same109system. The ID does not need to be unique between switches on different110systems.111 112The switch ID is used to locate ports on a switch and to know if aggregated113ports belong to the same switch.114 115Port Netdev Naming116^^^^^^^^^^^^^^^^^^117 118Udev rules should be used for port netdev naming, using some unique attribute119of the port as a key, for example the port MAC address or the port PHYS name.120Hard-coding of kernel netdev names within the driver is discouraged; let the121kernel pick the default netdev name, and let udev set the final name based on a122port attribute.123 124Using port PHYS name (ndo_get_phys_port_name) for the key is particularly125useful for dynamically-named ports where the device names its ports based on126external configuration. For example, if a physical 40G port is split logically127into 4 10G ports, resulting in 4 port netdevs, the device can give a unique128name for each port using port PHYS name. The udev rule would be::129 130 SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<phys_switch_id>", \131 ATTR{phys_port_name}!="", NAME="swX$attr{phys_port_name}"132 133Suggested naming convention is "swXpYsZ", where X is the switch name or ID, Y134is the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0135would be sub-port 0 on port 1 on switch 1.136 137Port Features138^^^^^^^^^^^^^139 140dev->netns_local141 142If the switchdev driver (and device) only supports offloading of the default143network namespace (netns), the driver should set this private flag to prevent144the port netdev from being moved out of the default netns. A netns-aware145driver/device would not set this flag and be responsible for partitioning146hardware to preserve netns containment. This means hardware cannot forward147traffic from a port in one namespace to another port in another namespace.148 149Port Topology150^^^^^^^^^^^^^151 152The port netdevs representing the physical switch ports can be organized into153higher-level switching constructs. The default construct is a standalone154router port, used to offload L3 forwarding. Two or more ports can be bonded155together to form a LAG. Two or more ports (or LAGs) can be bridged to bridge156L2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3157tunnels can be built on ports. These constructs are built using standard Linux158tools such as the bridge driver, the bonding/team drivers, and netlink-based159tools such as iproute2.160 161The switchdev driver can know a particular port's position in the topology by162monitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into a163bond will see its upper master change. If that bond is moved into a bridge,164the bond's upper master will change. And so on. The driver will track such165movements to know what position a port is in in the overall topology by166registering for netdevice events and acting on NETDEV_CHANGEUPPER.167 168L2 Forwarding Offload169---------------------170 171The idea is to offload the L2 data forwarding (switching) path from the kernel172to the switchdev device by mirroring bridge FDB entries down to the device. An173FDB entry is the {port, MAC, VLAN} tuple forwarding destination.174 175To offloading L2 bridging, the switchdev driver/device should support:176 177 - Static FDB entries installed on a bridge port178 - Notification of learned/forgotten src mac/vlans from device179 - STP state changes on the port180 - VLAN flooding of multicast/broadcast and unknown unicast packets181 182Static FDB Entries183^^^^^^^^^^^^^^^^^^184 185A driver which implements the ``ndo_fdb_add``, ``ndo_fdb_del`` and186``ndo_fdb_dump`` operations is able to support the command below, which adds a187static bridge FDB entry::188 189 bridge fdb add dev DEV ADDRESS [vlan VID] [self] static190 191(the "static" keyword is non-optional: if not specified, the entry defaults to192being "local", which means that it should not be forwarded)193 194The "self" keyword (optional because it is implicit) has the role of195instructing the kernel to fulfill the operation through the ``ndo_fdb_add``196implementation of the ``DEV`` device itself. If ``DEV`` is a bridge port, this197will bypass the bridge and therefore leave the software database out of sync198with the hardware one.199 200To avoid this, the "master" keyword can be used::201 202 bridge fdb add dev DEV ADDRESS [vlan VID] master static203 204The above command instructs the kernel to search for a master interface of205``DEV`` and fulfill the operation through the ``ndo_fdb_add`` method of that.206This time, the bridge generates a ``SWITCHDEV_FDB_ADD_TO_DEVICE`` notification207which the port driver can handle and use it to program its hardware table. This208way, the software and the hardware database will both contain this static FDB209entry.210 211Note: for new switchdev drivers that offload the Linux bridge, implementing the212``ndo_fdb_add`` and ``ndo_fdb_del`` bridge bypass methods is strongly213discouraged: all static FDB entries should be added on a bridge port using the214"master" flag. The ``ndo_fdb_dump`` is an exception and can be implemented to215visualize the hardware tables, if the device does not have an interrupt for216notifying the operating system of newly learned/forgotten dynamic FDB217addresses. In that case, the hardware FDB might end up having entries that the218software FDB does not, and implementing ``ndo_fdb_dump`` is the only way to see219them.220 221Note: by default, the bridge does not filter on VLAN and only bridges untagged222traffic. To enable VLAN support, turn on VLAN filtering::223 224 echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering225 226Notification of Learned/Forgotten Source MAC/VLANs227^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^228 229The switch device will learn/forget source MAC address/VLAN on ingress packets230and notify the switch driver of the mac/vlan/port tuples. The switch driver,231in turn, will notify the bridge driver using the switchdev notifier call::232 233 err = call_switchdev_notifiers(val, dev, info, extack);234 235Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when236forgetting, and info points to a struct switchdev_notifier_fdb_info. On237SWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the238bridge's FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridge239command will label these entries "offload"::240 241 $ bridge fdb242 52:54:00:12:35:01 dev sw1p1 master br0 permanent243 00:02:00:00:02:00 dev sw1p1 master br0 offload244 00:02:00:00:02:00 dev sw1p1 self245 52:54:00:12:35:02 dev sw1p2 master br0 permanent246 00:02:00:00:03:00 dev sw1p2 master br0 offload247 00:02:00:00:03:00 dev sw1p2 self248 33:33:00:00:00:01 dev eth0 self permanent249 01:00:5e:00:00:01 dev eth0 self permanent250 33:33:ff:00:00:00 dev eth0 self permanent251 01:80:c2:00:00:0e dev eth0 self permanent252 33:33:00:00:00:01 dev br0 self permanent253 01:00:5e:00:00:01 dev br0 self permanent254 33:33:ff:12:35:01 dev br0 self permanent255 256Learning on the port should be disabled on the bridge using the bridge command::257 258 bridge link set dev DEV learning off259 260Learning on the device port should be enabled, as well as learning_sync::261 262 bridge link set dev DEV learning on self263 bridge link set dev DEV learning_sync on self264 265Learning_sync attribute enables syncing of the learned/forgotten FDB entry to266the bridge's FDB. It's possible, but not optimal, to enable learning on the267device port and on the bridge port, and disable learning_sync.268 269To support learning, the driver implements switchdev op270switchdev_port_attr_set for SWITCHDEV_ATTR_PORT_ID_{PRE}_BRIDGE_FLAGS.271 272FDB Ageing273^^^^^^^^^^274 275The bridge will skip ageing FDB entries marked with NTF_EXT_LEARNED and it is276the responsibility of the port driver/device to age out these entries. If the277port device supports ageing, when the FDB entry expires, it will notify the278driver which in turn will notify the bridge with SWITCHDEV_FDB_DEL. If the279device does not support ageing, the driver can simulate ageing using a280garbage collection timer to monitor FDB entries. Expired entries will be281notified to the bridge using SWITCHDEV_FDB_DEL. See rocker driver for282example of driver running ageing timer.283 284To keep an NTF_EXT_LEARNED entry "alive", the driver should refresh the FDB285entry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). The286notification will reset the FDB entry's last-used time to now. The driver287should rate limit refresh notifications, for example, no more than once a288second. (The last-used time is visible using the bridge -s fdb option).289 290STP State Change on Port291^^^^^^^^^^^^^^^^^^^^^^^^292 293Internally or with a third-party STP protocol implementation (e.g. mstpd), the294bridge driver maintains the STP state for ports, and will notify the switch295driver of STP state change on a port using the switchdev op296switchdev_attr_port_set for SWITCHDEV_ATTR_PORT_ID_STP_UPDATE.297 298State is one of BR_STATE_*. The switch driver can use STP state updates to299update ingress packet filter list for the port. For example, if port is300DISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs301and other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass.302 303Note that STP BDPUs are untagged and STP state applies to all VLANs on the port304so packet filters should be applied consistently across untagged and tagged305VLANs on the port.306 307Flooding L2 domain308^^^^^^^^^^^^^^^^^^309 310For a given L2 VLAN domain, the switch device should flood multicast/broadcast311and unknown unicast packets to all ports in domain, if allowed by port's312current STP state. The switch driver, knowing which ports are within which313vlan L2 domain, can program the switch device for flooding. The packet may314be sent to the port netdev for processing by the bridge driver. The315bridge should not reflood the packet to the same ports the device flooded,316otherwise there will be duplicate packets on the wire.317 318To avoid duplicate packets, the switch driver should mark a packet as already319forwarded by setting the skb->offload_fwd_mark bit. The bridge driver will mark320the skb using the ingress bridge port's mark and prevent it from being forwarded321through any bridge port with the same mark.322 323It is possible for the switch device to not handle flooding and push the324packets up to the bridge driver for flooding. This is not ideal as the number325of ports scale in the L2 domain as the device is much more efficient at326flooding packets that software.327 328If supported by the device, flood control can be offloaded to it, preventing329certain netdevs from flooding unicast traffic for which there is no FDB entry.330 331IGMP Snooping332^^^^^^^^^^^^^333 334In order to support IGMP snooping, the port netdevs should trap to the bridge335driver all IGMP join and leave messages.336The bridge multicast module will notify port netdevs on every multicast group337changed whether it is static configured or dynamically joined/leave.338The hardware implementation should be forwarding all registered multicast339traffic groups only to the configured ports.340 341L3 Routing Offload342------------------343 344Offloading L3 routing requires that device be programmed with FIB entries from345the kernel, with the device doing the FIB lookup and forwarding. The device346does a longest prefix match (LPM) on FIB entries matching route prefix and347forwards the packet to the matching FIB entry's nexthop(s) egress ports.348 349To program the device, the driver has to register a FIB notifier handler350using register_fib_notifier. The following events are available:351 352=================== ===================================================353FIB_EVENT_ENTRY_ADD used for both adding a new FIB entry to the device,354 or modifying an existing entry on the device.355FIB_EVENT_ENTRY_DEL used for removing a FIB entry356FIB_EVENT_RULE_ADD,357FIB_EVENT_RULE_DEL used to propagate FIB rule changes358=================== ===================================================359 360FIB_EVENT_ENTRY_ADD and FIB_EVENT_ENTRY_DEL events pass::361 362 struct fib_entry_notifier_info {363 struct fib_notifier_info info; /* must be first */364 u32 dst;365 int dst_len;366 struct fib_info *fi;367 u8 tos;368 u8 type;369 u32 tb_id;370 u32 nlflags;371 };372 373to add/modify/delete IPv4 dst/dest_len prefix on table tb_id. The ``*fi``374structure holds details on the route and route's nexthops. ``*dev`` is one375of the port netdevs mentioned in the route's next hop list.376 377Routes offloaded to the device are labeled with "offload" in the ip route378listing::379 380 $ ip route show381 default via 192.168.0.2 dev eth0382 11.0.0.0/30 dev sw1p1 proto kernel scope link src 11.0.0.2 offload383 11.0.0.4/30 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload384 11.0.0.8/30 dev sw1p2 proto kernel scope link src 11.0.0.10 offload385 11.0.0.12/30 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload386 12.0.0.2 proto zebra metric 30 offload387 nexthop via 11.0.0.1 dev sw1p1 weight 1388 nexthop via 11.0.0.9 dev sw1p2 weight 1389 12.0.0.3 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload390 12.0.0.4 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload391 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.15392 393The "offload" flag is set in case at least one device offloads the FIB entry.394 395XXX: add/mod/del IPv6 FIB API396 397Nexthop Resolution398^^^^^^^^^^^^^^^^^^399 400The FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for401the switch device to forward the packet with the correct dst mac address, the402nexthop gateways must be resolved to the neighbor's mac address. Neighbor mac403address discovery comes via the ARP (or ND) process and is available via the404arp_tbl neighbor table. To resolve the routes nexthop gateways, the driver405should trigger the kernel's neighbor resolution process. See the rocker406driver's rocker_port_ipv4_resolve() for an example.407 408The driver can monitor for updates to arp_tbl using the netevent notifier409NETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops410for the routes as arp_tbl updates. The driver implements ndo_neigh_destroy411to know when arp_tbl neighbor entries are purged from the port.412 413Device driver expected behavior414-------------------------------415 416Below is a set of defined behavior that switchdev enabled network devices must417adhere to.418 419Configuration-less state420^^^^^^^^^^^^^^^^^^^^^^^^421 422Upon driver bring up, the network devices must be fully operational, and the423backing driver must configure the network device such that it is possible to424send and receive traffic to this network device and it is properly separated425from other network devices/ports (e.g.: as is frequent with a switch ASIC). How426this is achieved is heavily hardware dependent, but a simple solution can be to427use per-port VLAN identifiers unless a better mechanism is available428(proprietary metadata for each network port for instance).429 430The network device must be capable of running a full IP protocol stack431including multicast, DHCP, IPv4/6, etc. If necessary, it should program the432appropriate filters for VLAN, multicast, unicast etc. The underlying device433driver must effectively be configured in a similar fashion to what it would do434when IGMP snooping is enabled for IP multicast over these switchdev network435devices and unsolicited multicast must be filtered as early as possible in436the hardware.437 438When configuring VLANs on top of the network device, all VLANs must be working,439irrespective of the state of other network devices (e.g.: other ports being part440of a VLAN-aware bridge doing ingress VID checking). See below for details.441 442If the device implements e.g.: VLAN filtering, putting the interface in443promiscuous mode should allow the reception of all VLAN tags (including those444not present in the filter(s)).445 446Bridged switch ports447^^^^^^^^^^^^^^^^^^^^448 449When a switchdev enabled network device is added as a bridge member, it should450not disrupt any functionality of non-bridged network devices and they451should continue to behave as normal network devices. Depending on the bridge452configuration knobs below, the expected behavior is documented.453 454Bridge VLAN filtering455^^^^^^^^^^^^^^^^^^^^^456 457The Linux bridge allows the configuration of a VLAN filtering mode (statically,458at device creation time, and dynamically, during run time) which must be459observed by the underlying switchdev network device/hardware:460 461- with VLAN filtering turned off: the bridge is strictly VLAN unaware and its462 data path will process all Ethernet frames as if they are VLAN-untagged.463 The bridge VLAN database can still be modified, but the modifications should464 have no effect while VLAN filtering is turned off. Frames ingressing the465 device with a VID that is not programmed into the bridge/switch's VLAN table466 must be forwarded and may be processed using a VLAN device (see below).467 468- with VLAN filtering turned on: the bridge is VLAN-aware and frames ingressing469 the device with a VID that is not programmed into the bridges/switch's VLAN470 table must be dropped (strict VID checking).471 472When there is a VLAN device (e.g: sw0p1.100) configured on top of a switchdev473network device which is a bridge port member, the behavior of the software474network stack must be preserved, or the configuration must be refused if that475is not possible.476 477- with VLAN filtering turned off, the bridge will process all ingress traffic478 for the port, except for the traffic tagged with a VLAN ID destined for a479 VLAN upper. The VLAN upper interface (which consumes the VLAN tag) can even480 be added to a second bridge, which includes other switch ports or software481 interfaces. Some approaches to ensure that the forwarding domain for traffic482 belonging to the VLAN upper interfaces are managed properly:483 484 * If forwarding destinations can be managed per VLAN, the hardware could be485 configured to map all traffic, except the packets tagged with a VID486 belonging to a VLAN upper interface, to an internal VID corresponding to487 untagged packets. This internal VID spans all ports of the VLAN-unaware488 bridge. The VID corresponding to the VLAN upper interface spans the489 physical port of that VLAN interface, as well as the other ports that490 might be bridged with it.491 * Treat bridge ports with VLAN upper interfaces as standalone, and let492 forwarding be handled in the software data path.493 494- with VLAN filtering turned on, these VLAN devices can be created as long as495 the bridge does not have an existing VLAN entry with the same VID on any496 bridge port. These VLAN devices cannot be enslaved into the bridge since they497 duplicate functionality/use case with the bridge's VLAN data path processing.498 499Non-bridged network ports of the same switch fabric must not be disturbed in any500way by the enabling of VLAN filtering on the bridge device(s). If the VLAN501filtering setting is global to the entire chip, then the standalone ports502should indicate to the network stack that VLAN filtering is required by setting503'rx-vlan-filter: on [fixed]' in the ethtool features.504 505Because VLAN filtering can be turned on/off at runtime, the switchdev driver506must be able to reconfigure the underlying hardware on the fly to honor the507toggling of that option and behave appropriately. If that is not possible, the508switchdev driver can also refuse to support dynamic toggling of the VLAN509filtering knob at runtime and require a destruction of the bridge device(s) and510creation of new bridge device(s) with a different VLAN filtering value to511ensure VLAN awareness is pushed down to the hardware.512 513Even when VLAN filtering in the bridge is turned off, the underlying switch514hardware and driver may still configure itself in a VLAN-aware mode provided515that the behavior described above is observed.516 517The VLAN protocol of the bridge plays a role in deciding whether a packet is518treated as tagged or not: a bridge using the 802.1ad protocol must treat both519VLAN-untagged packets, as well as packets tagged with 802.1Q headers, as520untagged.521 522The 802.1p (VID 0) tagged packets must be treated in the same way by the device523as untagged packets, since the bridge device does not allow the manipulation of524VID 0 in its database.525 526When the bridge has VLAN filtering enabled and a PVID is not configured on the527ingress port, untagged and 802.1p tagged packets must be dropped. When the bridge528has VLAN filtering enabled and a PVID exists on the ingress port, untagged and529priority-tagged packets must be accepted and forwarded according to the530bridge's port membership of the PVID VLAN. When the bridge has VLAN filtering531disabled, the presence/lack of a PVID should not influence the packet532forwarding decision.533 534Bridge IGMP snooping535^^^^^^^^^^^^^^^^^^^^536 537The Linux bridge allows the configuration of IGMP snooping (statically, at538interface creation time, or dynamically, during runtime) which must be observed539by the underlying switchdev network device/hardware in the following way:540 541- when IGMP snooping is turned off, multicast traffic must be flooded to all542 ports within the same bridge that have mcast_flood=true. The CPU/management543 port should ideally not be flooded (unless the ingress interface has544 IFF_ALLMULTI or IFF_PROMISC) and continue to learn multicast traffic through545 the network stack notifications. If the hardware is not capable of doing that546 then the CPU/management port must also be flooded and multicast filtering547 happens in software.548 549- when IGMP snooping is turned on, multicast traffic must selectively flow550 to the appropriate network ports (including CPU/management port). Flooding of551 unknown multicast should be only towards the ports connected to a multicast552 router (the local device may also act as a multicast router).553 554The switch must adhere to RFC 4541 and flood multicast traffic accordingly555since that is what the Linux bridge implementation does.556 557Because IGMP snooping can be turned on/off at runtime, the switchdev driver558must be able to reconfigure the underlying hardware on the fly to honor the559toggling of that option and behave appropriately.560 561A switchdev driver can also refuse to support dynamic toggling of the multicast562snooping knob at runtime and require the destruction of the bridge device(s)563and creation of a new bridge device(s) with a different multicast snooping564value.565