459 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3=======================================4DSA switch configuration from userspace5=======================================6 7The DSA switch configuration is not integrated into the main userspace8network configuration suites by now and has to be performed manually.9 10.. _dsa-config-showcases:11 12Configuration showcases13-----------------------14 15To configure a DSA switch a couple of commands need to be executed. In this16documentation some common configuration scenarios are handled as showcases:17 18*single port*19 Every switch port acts as a different configurable Ethernet port20 21*bridge*22 Every switch port is part of one configurable Ethernet bridge23 24*gateway*25 Every switch port except one upstream port is part of a configurable26 Ethernet bridge.27 The upstream port acts as different configurable Ethernet port.28 29All configurations are performed with tools from iproute2, which is available30at https://www.kernel.org/pub/linux/utils/net/iproute2/31 32Through DSA every port of a switch is handled like a normal linux Ethernet33interface. The CPU port is the switch port connected to an Ethernet MAC chip.34The corresponding linux Ethernet interface is called the conduit interface.35All other corresponding linux interfaces are called user interfaces.36 37The user interfaces depend on the conduit interface being up in order for them38to send or receive traffic. Prior to kernel v5.12, the state of the conduit39interface had to be managed explicitly by the user. Starting with kernel v5.12,40the behavior is as follows:41 42- when a DSA user interface is brought up, the conduit interface is43 automatically brought up.44- when the conduit interface is brought down, all DSA user interfaces are45 automatically brought down.46 47In this documentation the following Ethernet interfaces are used:48 49*eth0*50 the conduit interface51 52*eth1*53 another conduit interface54 55*lan1*56 a user interface57 58*lan2*59 another user interface60 61*lan3*62 a third user interface63 64*wan*65 A user interface dedicated for upstream traffic66 67Further Ethernet interfaces can be configured similar.68The configured IPs and networks are:69 70*single port*71 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)72 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7)73 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11)74 75*bridge*76 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)77 78*gateway*79 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)80 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)81 82.. _dsa-tagged-configuration:83 84Configuration with tagging support85----------------------------------86 87The tagging based configuration is desired and supported by the majority of88DSA switches. These switches are capable to tag incoming and outgoing traffic89without using a VLAN based configuration.90 91*single port*92 .. code-block:: sh93 94 # configure each interface95 ip addr add 192.0.2.1/30 dev lan196 ip addr add 192.0.2.5/30 dev lan297 ip addr add 192.0.2.9/30 dev lan398 99 # For kernels earlier than v5.12, the conduit interface needs to be100 # brought up manually before the user ports.101 ip link set eth0 up102 103 # bring up the user interfaces104 ip link set lan1 up105 ip link set lan2 up106 ip link set lan3 up107 108*bridge*109 .. code-block:: sh110 111 # For kernels earlier than v5.12, the conduit interface needs to be112 # brought up manually before the user ports.113 ip link set eth0 up114 115 # bring up the user interfaces116 ip link set lan1 up117 ip link set lan2 up118 ip link set lan3 up119 120 # create bridge121 ip link add name br0 type bridge122 123 # add ports to bridge124 ip link set dev lan1 master br0125 ip link set dev lan2 master br0126 ip link set dev lan3 master br0127 128 # configure the bridge129 ip addr add 192.0.2.129/25 dev br0130 131 # bring up the bridge132 ip link set dev br0 up133 134*gateway*135 .. code-block:: sh136 137 # For kernels earlier than v5.12, the conduit interface needs to be138 # brought up manually before the user ports.139 ip link set eth0 up140 141 # bring up the user interfaces142 ip link set wan up143 ip link set lan1 up144 ip link set lan2 up145 146 # configure the upstream port147 ip addr add 192.0.2.1/30 dev wan148 149 # create bridge150 ip link add name br0 type bridge151 152 # add ports to bridge153 ip link set dev lan1 master br0154 ip link set dev lan2 master br0155 156 # configure the bridge157 ip addr add 192.0.2.129/25 dev br0158 159 # bring up the bridge160 ip link set dev br0 up161 162.. _dsa-vlan-configuration:163 164Configuration without tagging support165-------------------------------------166 167A minority of switches are not capable to use a taging protocol168(DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based169configuration.170 171*single port*172 The configuration can only be set up via VLAN tagging and bridge setup.173 174 .. code-block:: sh175 176 # tag traffic on CPU port177 ip link add link eth0 name eth0.1 type vlan id 1178 ip link add link eth0 name eth0.2 type vlan id 2179 ip link add link eth0 name eth0.3 type vlan id 3180 181 # For kernels earlier than v5.12, the conduit interface needs to be182 # brought up manually before the user ports.183 ip link set eth0 up184 ip link set eth0.1 up185 ip link set eth0.2 up186 ip link set eth0.3 up187 188 # bring up the user interfaces189 ip link set lan1 up190 ip link set lan2 up191 ip link set lan3 up192 193 # create bridge194 ip link add name br0 type bridge195 196 # activate VLAN filtering197 ip link set dev br0 type bridge vlan_filtering 1198 199 # add ports to bridges200 ip link set dev lan1 master br0201 ip link set dev lan2 master br0202 ip link set dev lan3 master br0203 204 # tag traffic on ports205 bridge vlan add dev lan1 vid 1 pvid untagged206 bridge vlan add dev lan2 vid 2 pvid untagged207 bridge vlan add dev lan3 vid 3 pvid untagged208 209 # configure the VLANs210 ip addr add 192.0.2.1/30 dev eth0.1211 ip addr add 192.0.2.5/30 dev eth0.2212 ip addr add 192.0.2.9/30 dev eth0.3213 214 # bring up the bridge devices215 ip link set br0 up216 217 218*bridge*219 .. code-block:: sh220 221 # tag traffic on CPU port222 ip link add link eth0 name eth0.1 type vlan id 1223 224 # For kernels earlier than v5.12, the conduit interface needs to be225 # brought up manually before the user ports.226 ip link set eth0 up227 ip link set eth0.1 up228 229 # bring up the user interfaces230 ip link set lan1 up231 ip link set lan2 up232 ip link set lan3 up233 234 # create bridge235 ip link add name br0 type bridge236 237 # activate VLAN filtering238 ip link set dev br0 type bridge vlan_filtering 1239 240 # add ports to bridge241 ip link set dev lan1 master br0242 ip link set dev lan2 master br0243 ip link set dev lan3 master br0244 ip link set eth0.1 master br0245 246 # tag traffic on ports247 bridge vlan add dev lan1 vid 1 pvid untagged248 bridge vlan add dev lan2 vid 1 pvid untagged249 bridge vlan add dev lan3 vid 1 pvid untagged250 251 # configure the bridge252 ip addr add 192.0.2.129/25 dev br0253 254 # bring up the bridge255 ip link set dev br0 up256 257*gateway*258 .. code-block:: sh259 260 # tag traffic on CPU port261 ip link add link eth0 name eth0.1 type vlan id 1262 ip link add link eth0 name eth0.2 type vlan id 2263 264 # For kernels earlier than v5.12, the conduit interface needs to be265 # brought up manually before the user ports.266 ip link set eth0 up267 ip link set eth0.1 up268 ip link set eth0.2 up269 270 # bring up the user interfaces271 ip link set wan up272 ip link set lan1 up273 ip link set lan2 up274 275 # create bridge276 ip link add name br0 type bridge277 278 # activate VLAN filtering279 ip link set dev br0 type bridge vlan_filtering 1280 281 # add ports to bridges282 ip link set dev wan master br0283 ip link set eth0.1 master br0284 ip link set dev lan1 master br0285 ip link set dev lan2 master br0286 287 # tag traffic on ports288 bridge vlan add dev lan1 vid 1 pvid untagged289 bridge vlan add dev lan2 vid 1 pvid untagged290 bridge vlan add dev wan vid 2 pvid untagged291 292 # configure the VLANs293 ip addr add 192.0.2.1/30 dev eth0.2294 ip addr add 192.0.2.129/25 dev br0295 296 # bring up the bridge devices297 ip link set br0 up298 299Forwarding database (FDB) management300------------------------------------301 302The existing DSA switches do not have the necessary hardware support to keep303the software FDB of the bridge in sync with the hardware tables, so the two304tables are managed separately (``bridge fdb show`` queries both, and depending305on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb306add`` or ``bridge fdb del`` command acts upon entries from one or both tables).307 308Up until kernel v4.14, DSA only supported user space management of bridge FDB309entries using the bridge bypass operations (which do not update the software310FDB, just the hardware one) using the ``self`` flag (which is optional and can311be omitted).312 313 .. code-block:: sh314 315 bridge fdb add dev swp0 00:01:02:03:04:05 self static316 # or shorthand317 bridge fdb add dev swp0 00:01:02:03:04:05 static318 319Due to a bug, the bridge bypass FDB implementation provided by DSA did not320distinguish between ``static`` and ``local`` FDB entries (``static`` are meant321to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent322to the host port). Instead, all FDB entries with the ``self`` flag (implicit or323explicit) are treated by DSA as ``static`` even if they are ``local``.324 325 .. code-block:: sh326 327 # This command:328 bridge fdb add dev swp0 00:01:02:03:04:05 static329 # behaves the same for DSA as this command:330 bridge fdb add dev swp0 00:01:02:03:04:05 local331 # or shorthand, because the 'local' flag is implicit if 'static' is not332 # specified, it also behaves the same as:333 bridge fdb add dev swp0 00:01:02:03:04:05334 335The last command is an incorrect way of adding a static bridge FDB entry to a336DSA switch using the bridge bypass operations, and works by mistake. Other337drivers will treat an FDB entry added by the same command as ``local`` and as338such, will not forward it, as opposed to DSA.339 340Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of341adding a bridge FDB entry to the switch: the bridge bypass discussed above, as342well as a new mode using the ``master`` flag which installs FDB entries in the343software bridge too.344 345 .. code-block:: sh346 347 bridge fdb add dev swp0 00:01:02:03:04:05 master static348 349Since kernel v5.14, DSA has gained stronger integration with the bridge's350software FDB, and the support for its bridge bypass FDB implementation (using351the ``self`` flag) has been removed. This results in the following changes:352 353 .. code-block:: sh354 355 # This is the only valid way of adding an FDB entry that is supported,356 # compatible with v4.14 kernels and later:357 bridge fdb add dev swp0 00:01:02:03:04:05 master static358 # This command is no longer buggy and the entry is properly treated as359 # 'local' instead of being forwarded:360 bridge fdb add dev swp0 00:01:02:03:04:05361 # This command no longer installs a static FDB entry to hardware:362 bridge fdb add dev swp0 00:01:02:03:04:05 static363 364Script writers are therefore encouraged to use the ``master static`` set of365flags when working with bridge FDB entries on DSA switch interfaces.366 367Affinity of user ports to CPU ports368-----------------------------------369 370Typically, DSA switches are attached to the host via a single Ethernet371interface, but in cases where the switch chip is discrete, the hardware design372may permit the use of 2 or more ports connected to the host, for an increase in373termination throughput.374 375DSA can make use of multiple CPU ports in two ways. First, it is possible to376statically assign the termination traffic associated with a certain user port377to be processed by a certain CPU port. This way, user space can implement378custom policies of static load balancing between user ports, by spreading the379affinities according to the available CPU ports.380 381Secondly, it is possible to perform load balancing between CPU ports on a per382packet basis, rather than statically assigning user ports to CPU ports.383This can be achieved by placing the DSA conduits under a LAG interface (bonding384or team). DSA monitors this operation and creates a mirror of this software LAG385on the CPU ports facing the physical DSA conduits that constitute the LAG slave386devices.387 388To make use of multiple CPU ports, the firmware (device tree) description of389the switch must mark all the links between CPU ports and their DSA conduits390using the ``ethernet`` reference/phandle. At startup, only a single CPU port391and DSA conduit will be used - the numerically first port from the firmware392description which has an ``ethernet`` property. It is up to the user to393configure the system for the switch to use other conduits.394 395DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow396changing the DSA conduit of a user port. The ``IFLA_DSA_CONDUIT`` u32 netlink397attribute contains the ifindex of the conduit device that handles each user398device. The DSA conduit must be a valid candidate based on firmware node399information, or a LAG interface which contains only slaves which are valid400candidates.401 402Using iproute2, the following manipulations are possible:403 404 .. code-block:: sh405 406 # See the DSA conduit in current use407 ip -d link show dev swp0408 (...)409 dsa master eth0410 411 # Static CPU port distribution412 ip link set swp0 type dsa master eth1413 ip link set swp1 type dsa master eth0414 ip link set swp2 type dsa master eth1415 ip link set swp3 type dsa master eth0416 417 # CPU ports in LAG, using explicit assignment of the DSA conduit418 ip link add bond0 type bond mode balance-xor && ip link set bond0 up419 ip link set eth1 down && ip link set eth1 master bond0420 ip link set swp0 type dsa master bond0421 ip link set swp1 type dsa master bond0422 ip link set swp2 type dsa master bond0423 ip link set swp3 type dsa master bond0424 ip link set eth0 down && ip link set eth0 master bond0425 ip -d link show dev swp0426 (...)427 dsa master bond0428 429 # CPU ports in LAG, relying on implicit migration of the DSA conduit430 ip link add bond0 type bond mode balance-xor && ip link set bond0 up431 ip link set eth0 down && ip link set eth0 master bond0432 ip link set eth1 down && ip link set eth1 master bond0433 ip -d link show dev swp0434 (...)435 dsa master bond0436 437Notice that in the case of CPU ports under a LAG, the use of the438``IFLA_DSA_CONDUIT`` netlink attribute is not strictly needed, but rather, DSA439reacts to the ``IFLA_MASTER`` attribute change of its present conduit (``eth0``)440and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly,441when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports442that were assigned to this interface to the first physical DSA conduit which is443eligible, based on the firmware description (it effectively reverts to the444startup configuration).445 446In a setup with more than 2 physical CPU ports, it is therefore possible to mix447static user to CPU port assignment with LAG between DSA conduits. It is not448possible to statically assign a user port towards a DSA conduit that has any449upper interfaces (this includes LAG devices - the conduit must always be the LAG450in this case).451 452Live changing of the DSA conduit (and thus CPU port) affinity of a user port is453permitted, in order to allow dynamic redistribution in response to traffic.454 455Physical DSA conduits are allowed to join and leave at any time a LAG interface456used as a DSA conduit; however, DSA will reject a LAG interface as a valid457candidate for being a DSA conduit unless it has at least one physical DSA conduit458as a slave device.459