270 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3==============================4The QorIQ DPAA Ethernet Driver5==============================6 7Authors:8- Madalin Bucur <madalin.bucur@nxp.com>9- Camelia Groza <camelia.groza@nxp.com>10 11.. Contents12 13 - DPAA Ethernet Overview14 - DPAA Ethernet Supported SoCs15 - Configuring DPAA Ethernet in your kernel16 - DPAA Ethernet Frame Processing17 - DPAA Ethernet Features18 - DPAA IRQ Affinity and Receive Side Scaling19 - Debugging20 21DPAA Ethernet Overview22======================23 24DPAA stands for Data Path Acceleration Architecture and it is a25set of networking acceleration IPs that are available on several26generations of SoCs, both on PowerPC and ARM64.27 28The Freescale DPAA architecture consists of a series of hardware blocks29that support Ethernet connectivity. The Ethernet driver depends upon the30following drivers in the Linux kernel:31 32 - Peripheral Access Memory Unit (PAMU) (* needed only for PPC platforms)33 drivers/iommu/fsl_*34 - Frame Manager (FMan)35 drivers/net/ethernet/freescale/fman36 - Queue Manager (QMan), Buffer Manager (BMan)37 drivers/soc/fsl/qbman38 39A simplified view of the dpaa_eth interfaces mapped to FMan MACs::40 41 dpaa_eth /eth0\ ... /ethN\42 driver | | | |43 ------------- ---- ----------- ---- -------------44 -Ports / Tx Rx \ ... / Tx Rx \45 FMan | | | |46 -MACs | MAC0 | | MACN |47 / dtsec0 \ ... / dtsecN \ (or tgec)48 / \ / \(or memac)49 --------- -------------- --- -------------- ---------50 FMan, FMan Port, FMan SP, FMan MURAM drivers51 ---------------------------------------------------------52 FMan HW blocks: MURAM, MACs, Ports, SP53 ---------------------------------------------------------54 55The dpaa_eth relation to the QMan, BMan and FMan::56 57 ________________________________58 dpaa_eth / eth0 \59 driver / \60 --------- -^- -^- -^- --- ---------61 QMan driver / \ / \ / \ \ / | BMan |62 |Rx | |Rx | |Tx | |Tx | | driver |63 --------- |Dfl| |Err| |Cnf| |FQs| | |64 QMan HW |FQ | |FQ | |FQs| | | | |65 / \ / \ / \ \ / | |66 --------- --- --- --- -v- ---------67 | FMan QMI | |68 | FMan HW FMan BMI | BMan HW |69 ----------------------- --------70 71where the acronyms used above (and in the code) are:72 73=============== ===========================================================74DPAA Data Path Acceleration Architecture75FMan DPAA Frame Manager76QMan DPAA Queue Manager77BMan DPAA Buffers Manager78QMI QMan interface in FMan79BMI BMan interface in FMan80FMan SP FMan Storage Profiles81MURAM Multi-user RAM in FMan82FQ QMan Frame Queue83Rx Dfl FQ default reception FQ84Rx Err FQ Rx error frames FQ85Tx Cnf FQ Tx confirmation FQs86Tx FQs transmission frame queues87dtsec datapath three speed Ethernet controller (10/100/1000 Mbps)88tgec ten gigabit Ethernet controller (10 Gbps)89memac multirate Ethernet MAC (10/100/1000/10000)90=============== ===========================================================91 92DPAA Ethernet Supported SoCs93============================94 95The DPAA drivers enable the Ethernet controllers present on the following SoCs:96 97PPC98- P102399- P2041100- P3041101- P4080102- P5020103- P5040104- T1023105- T1024106- T1040107- T1042108- T2080109- T4240110- B4860111 112ARM113- LS1043A114- LS1046A115 116Configuring DPAA Ethernet in your kernel117========================================118 119To enable the DPAA Ethernet driver, the following Kconfig options are required::120 121 # common for arch/arm64 and arch/powerpc platforms122 CONFIG_FSL_DPAA=y123 CONFIG_FSL_FMAN=y124 CONFIG_FSL_DPAA_ETH=y125 CONFIG_FSL_XGMAC_MDIO=y126 127 # for arch/powerpc only128 CONFIG_FSL_PAMU=y129 130 # common options needed for the PHYs used on the RDBs131 CONFIG_VITESSE_PHY=y132 CONFIG_REALTEK_PHY=y133 CONFIG_AQUANTIA_PHY=y134 135DPAA Ethernet Frame Processing136==============================137 138On Rx, buffers for the incoming frames are retrieved from the buffers found139in the dedicated interface buffer pool. The driver initializes and seeds these140with one page buffers.141 142On Tx, all transmitted frames are returned to the driver through Tx143confirmation frame queues. The driver is then responsible for freeing the144buffers. In order to do this properly, a backpointer is added to the buffer145before transmission that points to the skb. When the buffer returns to the146driver on a confirmation FQ, the skb can be correctly consumed.147 148DPAA Ethernet Features149======================150 151Currently the DPAA Ethernet driver enables the basic features required for152a Linux Ethernet driver. The support for advanced features will be added153gradually.154 155The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx156checksum offload feature is enabled by default and cannot be controlled through157ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS158provides a big performance boost for the forwarding scenarios, allowing159different traffic flows received by one interface to be processed by different160CPUs in parallel.161 162The driver has support for multiple prioritized Tx traffic classes. Priorities163range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with164strict priority levels. Each traffic class contains NR_CPU TX queues. By165default, only one traffic class is enabled and the lowest priority Tx queues166are used. Higher priority traffic classes can be enabled with the mqprio167qdisc. For example, all four traffic classes are enabled on an interface with168the following command. Furthermore, skb priority levels are mapped to traffic169classes as follows:170 171 * priorities 0 to 3 - traffic class 0 (low priority)172 * priorities 4 to 7 - traffic class 1 (medium-low priority)173 * priorities 8 to 11 - traffic class 2 (medium-high priority)174 * priorities 12 to 15 - traffic class 3 (high priority)175 176::177 178 tc qdisc add dev <int> root handle 1: \179 mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1180 181DPAA IRQ Affinity and Receive Side Scaling182==========================================183 184Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation185queues is seen by the CPU as ingress traffic on a certain portal.186The DPAA QMan portal interrupts are affined each to a certain CPU.187The same portal interrupt services all the QMan portal consumers.188 189By default the DPAA Ethernet driver enables RSS, making use of the190DPAA FMan Parser and Keygen blocks to distribute traffic on 128191hardware frame queues using a hash on IP v4/v6 source and destination192and L4 source and destination ports, in present in the received frame.193When RSS is disabled, all traffic received by a certain interface is194received on the default Rx frame queue. The default DPAA Rx frame195queues are configured to put the received traffic into a pool channel196that allows any available CPU portal to dequeue the ingress traffic.197The default frame queues have the HOLDACTIVE option set, ensuring that198traffic bursts from a certain queue are serviced by the same CPU.199This ensures a very low rate of frame reordering. A drawback of this200is that only one CPU at a time can service the traffic received by a201certain interface when RSS is not enabled.202 203To implement RSS, the DPAA Ethernet driver allocates an extra set of204128 Rx frame queues that are configured to dedicated channels, in a205round-robin manner. The mapping of the frame queues to CPUs is now206hardcoded, there is no indirection table to move traffic for a certain207FQ (hash result) to another CPU. The ingress traffic arriving on one208of these frame queues will arrive at the same portal and will always209be processed by the same CPU. This ensures intra-flow order preservation210and workload distribution for multiple traffic flows.211 212RSS can be turned off for a certain interface using ethtool, i.e.::213 214 # ethtool -N fm1-mac9 rx-flow-hash tcp4 ""215 216To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6::217 218 # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn219 220There is no independent control for individual protocols, any command221run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is222going to control the rx-flow-hashing for all protocols on that interface.223 224Besides using the FMan Keygen computed hash for spreading traffic on the225128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when226the NETIF_F_RXHASH feature is on (active by default). This can be turned227on or off through ethtool, i.e.::228 229 # ethtool -K fm1-mac9 rx-hashing off230 # ethtool -k fm1-mac9 | grep hash231 receive-hashing: off232 # ethtool -K fm1-mac9 rx-hashing on233 Actual changes:234 receive-hashing: on235 # ethtool -k fm1-mac9 | grep hash236 receive-hashing: on237 238Please note that Rx hashing depends upon the rx-flow-hashing being on239for that interface - turning off rx-flow-hashing will also disable the240rx-hashing (without ethtool reporting it as off as that depends on the241NETIF_F_RXHASH feature flag).242 243Debugging244=========245 246The following statistics are exported for each interface through ethtool:247 248 - interrupt count per CPU249 - Rx packets count per CPU250 - Tx packets count per CPU251 - Tx confirmed packets count per CPU252 - Tx S/G frames count per CPU253 - Tx error count per CPU254 - Rx error count per CPU255 - Rx error count per type256 - congestion related statistics:257 258 - congestion status259 - time spent in congestion260 - number of time the device entered congestion261 - dropped packets count per cause262 263The driver also exports the following information in sysfs:264 265 - the FQ IDs for each FQ type266 /sys/devices/platform/soc/<addr>.fman/<addr>.ethernet/dpaa-ethernet.<id>/net/fm<nr>-mac<nr>/fqids267 268 - the ID of the buffer pool in use269 /sys/devices/platform/soc/<addr>.fman/<addr>.ethernet/dpaa-ethernet.<id>/net/fm<nr>-mac<nr>/bpids270