588 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3======================================4Texas Instruments CPSW ethernet driver5======================================6 7Multiqueue & CBS & MQPRIO8=========================9 10 11The cpsw has 3 CBS shapers for each external ports. This document12describes MQPRIO and CBS Qdisc offload configuration for cpsw driver13based on examples. It potentially can be used in audio video bridging14(AVB) and time sensitive networking (TSN).15 16The following examples were tested on AM572x EVM and BBB boards.17 18Test setup19==========20 21Under consideration two examples with AM572x EVM running cpsw driver22in dual_emac mode.23 24Several prerequisites:25 26- TX queues must be rated starting from txq0 that has highest priority27- Traffic classes are used starting from 0, that has highest priority28- CBS shapers should be used with rated queues29- The bandwidth for CBS shapers has to be set a little bit more then30 potential incoming rate, thus, rate of all incoming tx queues has31 to be a little less32- Real rates can differ, due to discreetness33- Map skb-priority to txq is not enough, also skb-priority to l2 prio34 map has to be created with ip or vconfig tool35- Any l2/socket prio (0 - 7) for classes can be used, but for36 simplicity default values are used: 3 and 237- only 2 classes tested: A and B, but checked and can work with more,38 maximum allowed 4, but only for 3 rate can be set.39 40Test setup for examples41=======================42 43::44 45 +-------------------------------+46 |--+ |47 | | Workstation0 |48 |E | MAC 18:03:73:66:87:42 |49 +-----------------------------+ +--|t | |50 | | 1 | E | | |h |./tsn_listener -d \ |51 | Target board: | 0 | t |--+ |0 | 18:03:73:66:87:42 -i eth0 \|52 | AM572x EVM | 0 | h | | | -s 1500 |53 | | 0 | 0 | |--+ |54 | Only 2 classes: |Mb +---| +-------------------------------+55 | class A, class B | |56 | | +---| +-------------------------------+57 | | 1 | E | |--+ |58 | | 0 | t | | | Workstation1 |59 | | 0 | h |--+ |E | MAC 20:cf:30:85:7d:fd |60 | |Mb | 1 | +--|t | |61 +-----------------------------+ |h |./tsn_listener -d \ |62 |0 | 20:cf:30:85:7d:fd -i eth0 \|63 | | -s 1500 |64 |--+ |65 +-------------------------------+66 67 68Example 1: One port tx AVB configuration scheme for target board69----------------------------------------------------------------70 71(prints and scheme for AM572x evm, applicable for single port boards)72 73- tc - traffic class74- txq - transmit queue75- p - priority76- f - fifo (cpsw fifo)77- S - shaper configured78 79::80 81 +------------------------------------------------------------------+ u82 | +---------------+ +---------------+ +------+ +------+ | s83 | | | | | | | | | | e84 | | App 1 | | App 2 | | Apps | | Apps | | r85 | | Class A | | Class B | | Rest | | Rest | |86 | | Eth0 | | Eth0 | | Eth0 | | Eth1 | | s87 | | VLAN100 | | VLAN100 | | | | | | | | p88 | | 40 Mb/s | | 20 Mb/s | | | | | | | | a89 | | SO_PRIORITY=3 | | SO_PRIORITY=2 | | | | | | | | c90 | | | | | | | | | | | | | | e91 | +---|-----------+ +---|-----------+ +---|--+ +---|--+ |92 +-----|------------------|------------------|--------|-------------+93 +-+ +------------+ | |94 | | +-----------------+ +--+95 | | | |96 +---|-------|-------------|-----------------------|----------------+97 | +----+ +----+ +----+ +----+ +----+ |98 | | p3 | | p2 | | p1 | | p0 | | p0 | | k99 | \ / \ / \ / \ / \ / | e100 | \ / \ / \ / \ / \ / | r101 | \/ \/ \/ \/ \/ | n102 | | | | | | e103 | | | +-----+ | | l104 | | | | | |105 | +----+ +----+ +----+ +----+ | s106 | |tc0 | |tc1 | |tc2 | |tc0 | | p107 | \ / \ / \ / \ / | a108 | \ / \ / \ / \ / | c109 | \/ \/ \/ \/ | e110 | | | +-----+ | |111 | | | | | | |112 | | | | | | |113 | | | | | | |114 | +----+ +----+ +----+ +----+ +----+ |115 | |txq0| |txq1| |txq2| |txq3| |txq4| |116 | \ / \ / \ / \ / \ / |117 | \ / \ / \ / \ / \ / |118 | \/ \/ \/ \/ \/ |119 | +-|------|------|------|--+ +--|--------------+ |120 | | | | | | | Eth0.100 | | Eth1 | |121 +---|------|------|------|------------------------|----------------+122 | | | | |123 p p p p |124 3 2 0-1, 4-7 <- L2 priority |125 | | | | |126 | | | | |127 +---|------|------|------|------------------------|----------------+128 | | | | | |----------+ |129 | +----+ +----+ +----+ +----+ +----+ |130 | |dma7| |dma6| |dma5| |dma4| |dma3| |131 | \ / \ / \ / \ / \ / | c132 | \S / \S / \ / \ / \ / | p133 | \/ \/ \/ \/ \/ | s134 | | | | +----- | | w135 | | | | | | |136 | | | | | | | d137 | +----+ +----+ +----+p p+----+ | r138 | | | | | | |o o| | | i139 | | f3 | | f2 | | f0 |r r| f0 | | v140 | |tc0 | |tc1 | |tc2 |t t|tc0 | | e141 | \CBS / \CBS / \CBS /1 2\CBS / | r142 | \S / \S / \ / \ / |143 | \/ \/ \/ \/ |144 +------------------------------------------------------------------+145 146 1471) ::148 149 150 // Add 4 tx queues, for interface Eth0, and 1 tx queue for Eth1151 $ ethtool -L eth0 rx 1 tx 5152 rx unmodified, ignoring153 1542) ::155 156 // Check if num of queues is set correctly:157 $ ethtool -l eth0158 Channel parameters for eth0:159 Pre-set maximums:160 RX: 8161 TX: 8162 Other: 0163 Combined: 0164 Current hardware settings:165 RX: 1166 TX: 5167 Other: 0168 Combined: 0169 1703) ::171 172 // TX queues must be rated starting from 0, so set bws for tx0 and tx1173 // Set rates 40 and 20 Mb/s appropriately.174 // Pay attention, real speed can differ a bit due to discreetness.175 // Leave last 2 tx queues not rated.176 $ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate177 $ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate178 1794) ::180 181 // Check maximum rate of tx (cpdma) queues:182 $ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate183 40184 20185 0186 0187 0188 1895) ::190 191 // Map skb->priority to traffic class:192 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2193 // Map traffic class to transmit queue:194 // tc0 -> txq0, tc1 -> txq1, tc2 -> (txq2, txq3)195 $ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \196 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 1197 1985a) ::199 200 // As two interface sharing same set of tx queues, assign all traffic201 // coming to interface Eth1 to separate queue in order to not mix it202 // with traffic from interface Eth0, so use separate txq to send203 // packets to Eth1, so all prio -> tc0 and tc0 -> txq4204 // Here hw 0, so here still default configuration for eth1 in hw205 $ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 1 \206 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@4 hw 0207 2086) ::209 210 // Check classes settings211 $ tc -g class show dev eth0212 +---(100:ffe2) mqprio213 | +---(100:3) mqprio214 | +---(100:4) mqprio215 |216 +---(100:ffe1) mqprio217 | +---(100:2) mqprio218 |219 +---(100:ffe0) mqprio220 +---(100:1) mqprio221 222 $ tc -g class show dev eth1223 +---(100:ffe0) mqprio224 +---(100:5) mqprio225 2267) ::227 228 // Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc229 // Set it +1 Mb for reserve (important!)230 // here only idle slope is important, others arg are ignored231 // Pay attention, real speed can differ a bit due to discreetness232 $ tc qdisc add dev eth0 parent 100:1 cbs locredit -1438 \233 hicredit 62 sendslope -959000 idleslope 41000 offload 1234 net eth0: set FIFO3 bw = 50235 2368) ::237 238 // Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc:239 // Set it +1 Mb for reserve (important!)240 $ tc qdisc add dev eth0 parent 100:2 cbs locredit -1468 \241 hicredit 65 sendslope -979000 idleslope 21000 offload 1242 net eth0: set FIFO2 bw = 30243 2449) ::245 246 // Create vlan 100 to map sk->priority to vlan qos247 $ ip link add link eth0 name eth0.100 type vlan id 100248 8021q: 802.1Q VLAN Support v1.8249 8021q: adding VLAN 0 to HW filter on device eth0250 8021q: adding VLAN 0 to HW filter on device eth1251 net eth0: Adding vlanid 100 to vlan filter252 25310) ::254 255 // Map skb->priority to L2 prio, 1 to 1256 $ ip link set eth0.100 type vlan \257 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7258 25911) ::260 261 // Check egress map for vlan 100262 $ cat /proc/net/vlan/eth0.100263 [...]264 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0265 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7266 26712) ::268 269 // Run your appropriate tools with socket option "SO_PRIORITY"270 // to 3 for class A and/or to 2 for class B271 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)272 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&273 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&274 27513) ::276 277 // run your listener on workstation (should be in same vlan)278 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)279 ./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500280 Receiving data rate: 39012 kbps281 Receiving data rate: 39012 kbps282 Receiving data rate: 39012 kbps283 Receiving data rate: 39012 kbps284 Receiving data rate: 39012 kbps285 Receiving data rate: 39012 kbps286 Receiving data rate: 39012 kbps287 Receiving data rate: 39012 kbps288 Receiving data rate: 39012 kbps289 Receiving data rate: 39012 kbps290 Receiving data rate: 39012 kbps291 Receiving data rate: 39012 kbps292 Receiving data rate: 39000 kbps293 29414) ::295 296 // Restore default configuration if needed297 $ ip link del eth0.100298 $ tc qdisc del dev eth1 root299 $ tc qdisc del dev eth0 root300 net eth0: Prev FIFO2 is shaped301 net eth0: set FIFO3 bw = 0302 net eth0: set FIFO2 bw = 0303 $ ethtool -L eth0 rx 1 tx 1304 305Example 2: Two port tx AVB configuration scheme for target board306----------------------------------------------------------------307 308(prints and scheme for AM572x evm, for dual emac boards only)309 310::311 312 +------------------------------------------------------------------+ u313 | +----------+ +----------+ +------+ +----------+ +----------+ | s314 | | | | | | | | | | | | e315 | | App 1 | | App 2 | | Apps | | App 3 | | App 4 | | r316 | | Class A | | Class B | | Rest | | Class B | | Class A | |317 | | Eth0 | | Eth0 | | | | | Eth1 | | Eth1 | | s318 | | VLAN100 | | VLAN100 | | | | | VLAN100 | | VLAN100 | | p319 | | 40 Mb/s | | 20 Mb/s | | | | | 10 Mb/s | | 30 Mb/s | | a320 | | SO_PRI=3 | | SO_PRI=2 | | | | | SO_PRI=3 | | SO_PRI=2 | | c321 | | | | | | | | | | | | | | | | | e322 | +---|------+ +---|------+ +---|--+ +---|------+ +---|------+ |323 +-----|-------------|-------------|---------|-------------|--------+324 +-+ +-------+ | +----------+ +----+325 | | +-------+------+ | |326 | | | | | |327 +---|-------|-------------|--------------|-------------|-------|---+328 | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |329 | | p3 | | p2 | | p1 | | p0 | | p0 | | p1 | | p2 | | p3 | | k330 | \ / \ / \ / \ / \ / \ / \ / \ / | e331 | \ / \ / \ / \ / \ / \ / \ / \ / | r332 | \/ \/ \/ \/ \/ \/ \/ \/ | n333 | | | | | | | | e334 | | | +----+ +----+ | | | l335 | | | | | | | |336 | +----+ +----+ +----+ +----+ +----+ +----+ | s337 | |tc0 | |tc1 | |tc2 | |tc2 | |tc1 | |tc0 | | p338 | \ / \ / \ / \ / \ / \ / | a339 | \ / \ / \ / \ / \ / \ / | c340 | \/ \/ \/ \/ \/ \/ | e341 | | | +-----+ +-----+ | | |342 | | | | | | | | | |343 | | | | | | | | | |344 | | | | | E E | | | | |345 | +----+ +----+ +----+ +----+ t t +----+ +----+ +----+ +----+ |346 | |txq0| |txq1| |txq4| |txq5| h h |txq6| |txq7| |txq3| |txq2| |347 | \ / \ / \ / \ / 0 1 \ / \ / \ / \ / |348 | \ / \ / \ / \ / . . \ / \ / \ / \ / |349 | \/ \/ \/ \/ 1 1 \/ \/ \/ \/ |350 | +-|------|------|------|--+ 0 0 +-|------|------|------|--+ |351 | | | | | | | 0 0 | | | | | | |352 +---|------|------|------|---------------|------|------|------|----+353 | | | | | | | |354 p p p p p p p p355 3 2 0-1, 4-7 <-L2 pri-> 0-1, 4-7 2 3356 | | | | | | | |357 | | | | | | | |358 +---|------|------|------|---------------|------|------|------|----+359 | | | | | | | | | |360 | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |361 | |dma7| |dma6| |dma3| |dma2| |dma1| |dma0| |dma4| |dma5| |362 | \ / \ / \ / \ / \ / \ / \ / \ / | c363 | \S / \S / \ / \ / \ / \ / \S / \S / | p364 | \/ \/ \/ \/ \/ \/ \/ \/ | s365 | | | | +----- | | | | | w366 | | | | | +----+ | | | |367 | | | | | | | | | | d368 | +----+ +----+ +----+p p+----+ +----+ +----+ | r369 | | | | | | |o o| | | | | | | i370 | | f3 | | f2 | | f0 |r CPSW r| f3 | | f2 | | f0 | | v371 | |tc0 | |tc1 | |tc2 |t t|tc0 | |tc1 | |tc2 | | e372 | \CBS / \CBS / \CBS /1 2\CBS / \CBS / \CBS / | r373 | \S / \S / \ / \S / \S / \ / |374 | \/ \/ \/ \/ \/ \/ |375 +------------------------------------------------------------------+376 ========================================Eth==========================>377 3781) ::379 380 // Add 8 tx queues, for interface Eth0, but they are common, so are accessed381 // by two interfaces Eth0 and Eth1.382 $ ethtool -L eth1 rx 1 tx 8383 rx unmodified, ignoring384 3852) ::386 387 // Check if num of queues is set correctly:388 $ ethtool -l eth0389 Channel parameters for eth0:390 Pre-set maximums:391 RX: 8392 TX: 8393 Other: 0394 Combined: 0395 Current hardware settings:396 RX: 1397 TX: 8398 Other: 0399 Combined: 0400 4013) ::402 403 // TX queues must be rated starting from 0, so set bws for tx0 and tx1 for Eth0404 // and for tx2 and tx3 for Eth1. That is, rates 40 and 20 Mb/s appropriately405 // for Eth0 and 30 and 10 Mb/s for Eth1.406 // Real speed can differ a bit due to discreetness407 // Leave last 4 tx queues as not rated408 $ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate409 $ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate410 $ echo 30 > /sys/class/net/eth1/queues/tx-2/tx_maxrate411 $ echo 10 > /sys/class/net/eth1/queues/tx-3/tx_maxrate412 4134) ::414 415 // Check maximum rate of tx (cpdma) queues:416 $ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate417 40418 20419 30420 10421 0422 0423 0424 0425 4265) ::427 428 // Map skb->priority to traffic class for Eth0:429 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2430 // Map traffic class to transmit queue:431 // tc0 -> txq0, tc1 -> txq1, tc2 -> (txq4, txq5)432 $ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \433 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@4 hw 1434 4356) ::436 437 // Check classes settings438 $ tc -g class show dev eth0439 +---(100:ffe2) mqprio440 | +---(100:5) mqprio441 | +---(100:6) mqprio442 |443 +---(100:ffe1) mqprio444 | +---(100:2) mqprio445 |446 +---(100:ffe0) mqprio447 +---(100:1) mqprio448 4497) ::450 451 // Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc for Eth0452 // here only idle slope is important, others ignored453 // Real speed can differ a bit due to discreetness454 $ tc qdisc add dev eth0 parent 100:1 cbs locredit -1470 \455 hicredit 62 sendslope -959000 idleslope 41000 offload 1456 net eth0: set FIFO3 bw = 50457 4588) ::459 460 // Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc for Eth0461 $ tc qdisc add dev eth0 parent 100:2 cbs locredit -1470 \462 hicredit 65 sendslope -979000 idleslope 21000 offload 1463 net eth0: set FIFO2 bw = 30464 4659) ::466 467 // Create vlan 100 to map sk->priority to vlan qos for Eth0468 $ ip link add link eth0 name eth0.100 type vlan id 100469 net eth0: Adding vlanid 100 to vlan filter470 47110) ::472 473 // Map skb->priority to L2 prio for Eth0.100, one to one474 $ ip link set eth0.100 type vlan \475 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7476 47711) ::478 479 // Check egress map for vlan 100480 $ cat /proc/net/vlan/eth0.100481 [...]482 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0483 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7484 48512) ::486 487 // Map skb->priority to traffic class for Eth1:488 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2489 // Map traffic class to transmit queue:490 // tc0 -> txq2, tc1 -> txq3, tc2 -> (txq6, txq7)491 $ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 3 \492 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@2 1@3 2@6 hw 1493 49413) ::495 496 // Check classes settings497 $ tc -g class show dev eth1498 +---(100:ffe2) mqprio499 | +---(100:7) mqprio500 | +---(100:8) mqprio501 |502 +---(100:ffe1) mqprio503 | +---(100:4) mqprio504 |505 +---(100:ffe0) mqprio506 +---(100:3) mqprio507 50814) ::509 510 // Set rate for class A - 31 Mbit (tc0, txq2) using CBS Qdisc for Eth1511 // here only idle slope is important, others ignored, but calculated512 // for interface speed - 100Mb for eth1 port.513 // Set it +1 Mb for reserve (important!)514 $ tc qdisc add dev eth1 parent 100:3 cbs locredit -1035 \515 hicredit 465 sendslope -69000 idleslope 31000 offload 1516 net eth1: set FIFO3 bw = 31517 51815) ::519 520 // Set rate for class B - 11 Mbit (tc1, txq3) using CBS Qdisc for Eth1521 // Set it +1 Mb for reserve (important!)522 $ tc qdisc add dev eth1 parent 100:4 cbs locredit -1335 \523 hicredit 405 sendslope -89000 idleslope 11000 offload 1524 net eth1: set FIFO2 bw = 11525 52616) ::527 528 // Create vlan 100 to map sk->priority to vlan qos for Eth1529 $ ip link add link eth1 name eth1.100 type vlan id 100530 net eth1: Adding vlanid 100 to vlan filter531 53217) ::533 534 // Map skb->priority to L2 prio for Eth1.100, one to one535 $ ip link set eth1.100 type vlan \536 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7537 53818) ::539 540 // Check egress map for vlan 100541 $ cat /proc/net/vlan/eth1.100542 [...]543 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0544 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7545 54619) ::547 548 // Run appropriate tools with socket option "SO_PRIORITY" to 3549 // for class A and to 2 for class B. For both interfaces550 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&551 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&552 ./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p2 -s 1500&553 ./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p3 -s 1500&554 55520) ::556 557 // run your listener on workstation (should be in same vlan)558 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)559 ./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500560 Receiving data rate: 39012 kbps561 Receiving data rate: 39012 kbps562 Receiving data rate: 39012 kbps563 Receiving data rate: 39012 kbps564 Receiving data rate: 39012 kbps565 Receiving data rate: 39012 kbps566 Receiving data rate: 39012 kbps567 Receiving data rate: 39012 kbps568 Receiving data rate: 39012 kbps569 Receiving data rate: 39012 kbps570 Receiving data rate: 39012 kbps571 Receiving data rate: 39012 kbps572 Receiving data rate: 39000 kbps573 57421) ::575 576 // Restore default configuration if needed577 $ ip link del eth1.100578 $ ip link del eth0.100579 $ tc qdisc del dev eth1 root580 net eth1: Prev FIFO2 is shaped581 net eth1: set FIFO3 bw = 0582 net eth1: set FIFO2 bw = 0583 $ tc qdisc del dev eth0 root584 net eth0: Prev FIFO2 is shaped585 net eth0: set FIFO3 bw = 0586 net eth0: set FIFO2 bw = 0587 $ ethtool -L eth0 rx 1 tx 1588