407 lines · plain
1.. include:: <isonum.txt>2 3=========================================================4DPAA2 (Data Path Acceleration Architecture Gen2) Overview5=========================================================6 7:Copyright: |copy| 2015 Freescale Semiconductor Inc.8:Copyright: |copy| 2018 NXP9 10This document provides an overview of the Freescale DPAA2 architecture11and how it is integrated into the Linux kernel.12 13Introduction14============15 16DPAA2 is a hardware architecture designed for high-speeed network17packet processing. DPAA2 consists of sophisticated mechanisms for18processing Ethernet packets, queue management, buffer management,19autonomous L2 switching, virtual Ethernet bridging, and accelerator20(e.g. crypto) sharing.21 22A DPAA2 hardware component called the Management Complex (or MC) manages the23DPAA2 hardware resources. The MC provides an object-based abstraction for24software drivers to use the DPAA2 hardware.25The MC uses DPAA2 hardware resources such as queues, buffer pools, and26network ports to create functional objects/devices such as network27interfaces, an L2 switch, or accelerator instances.28The MC provides memory-mapped I/O command interfaces (MC portals)29which DPAA2 software drivers use to operate on DPAA2 objects.30 31The diagram below shows an overview of the DPAA2 resource management32architecture::33 34 +--------------------------------------+35 | OS |36 | DPAA2 drivers |37 | | |38 +-----------------------------|--------+39 |40 | (create,discover,connect41 | config,use,destroy)42 |43 DPAA2 |44 +------------------------| mc portal |-+45 | | |46 | +- - - - - - - - - - - - -V- - -+ |47 | | | |48 | | Management Complex (MC) | |49 | | | |50 | +- - - - - - - - - - - - - - - -+ |51 | |52 | Hardware Hardware |53 | Resources Objects |54 | --------- ------- |55 | -queues -DPRC |56 | -buffer pools -DPMCP |57 | -Eth MACs/ports -DPIO |58 | -network interface -DPNI |59 | profiles -DPMAC |60 | -queue portals -DPBP |61 | -MC portals ... |62 | ... |63 | |64 +--------------------------------------+65 66 67The MC mediates operations such as create, discover,68connect, configuration, and destroy. Fast-path operations69on data, such as packet transmit/receive, are not mediated by70the MC and are done directly using memory mapped regions in71DPIO objects.72 73Overview of DPAA2 Objects74=========================75 76The section provides a brief overview of some key DPAA2 objects.77A simple scenario is described illustrating the objects involved78in creating a network interfaces.79 80DPRC (Datapath Resource Container)81----------------------------------82 83A DPRC is a container object that holds all the other84types of DPAA2 objects. In the example diagram below there85are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)86in the container.87 88::89 90 +---------------------------------------------------------+91 | DPRC |92 | |93 | +-------+ +-------+ +-------+ +-------+ +-------+ |94 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | |95 | +-------+ +-------+ +-------+ +---+---+ +---+---+ |96 | | DPMCP | | DPIO | |97 | +-------+ +-------+ |98 | | DPMCP | |99 | +-------+ |100 | |101 +---------------------------------------------------------+102 103From the point of view of an OS, a DPRC behaves similar to a plug and104play bus, like PCI. DPRC commands can be used to enumerate the contents105of the DPRC, discover the hardware objects present (including mappable106regions and interrupts).107 108::109 110 DPRC.1 (bus)111 |112 +--+--------+-------+-------+-------+113 | | | | |114 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1115 DPMCP.2 DPIO.2116 DPMCP.3117 118Hardware objects can be created and destroyed dynamically, providing119the ability to hot plug/unplug objects in and out of the DPRC.120 121A DPRC has a mappable MMIO region (an MC portal) that can be used122to send MC commands. It has an interrupt for status events (like123hotplug).124All objects in a container share the same hardware "isolation context".125This means that with respect to an IOMMU the isolation granularity126is at the DPRC (container) level, not at the individual object127level.128 129DPRCs can be defined statically and populated with objects130via a config file passed to the MC when firmware starts it.131 132DPAA2 Objects for an Ethernet Network Interface133-----------------------------------------------134 135A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX136queuing mechanisms, configuration mechanisms, buffer management,137physical ports, and interrupts. DPAA2 uses a more granular approach138utilizing multiple hardware objects. Each object provides specialized139functions. Groups of these objects are used by software to provide140Ethernet network interface functionality. This approach provides141efficient use of finite hardware resources, flexibility, and142performance advantages.143 144The diagram below shows the objects needed for a simple145network interface configuration on a system with 2 CPUs.146 147::148 149 +---+---+ +---+---+150 CPU0 CPU1151 +---+---+ +---+---+152 | |153 +---+---+ +---+---+154 DPIO DPIO155 +---+---+ +---+---+156 \ /157 \ /158 \ /159 +---+---+160 DPNI --- DPBP,DPMCP161 +---+---+162 |163 |164 +---+---+165 DPMAC166 +---+---+167 |168 port/PHY169 170Below the objects are described. For each object a brief description171is provided along with a summary of the kinds of operations the object172supports and a summary of key resources of the object (MMIO regions173and IRQs).174 175DPMAC (Datapath Ethernet MAC)176~~~~~~~~~~~~~~~~~~~~~~~~~~~~~177Represents an Ethernet MAC, a hardware device that connects to an Ethernet178PHY and allows physical transmission and reception of Ethernet frames.179 180- MMIO regions: none181- IRQs: DPNI link change182- commands: set link up/down, link config, get stats,183 IRQ config, enable, reset184 185DPNI (Datapath Network Interface)186~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~187Contains TX/RX queues, network interface configuration, and RX buffer pool188configuration mechanisms. The TX/RX queues are in memory and are identified189by queue number.190 191- MMIO regions: none192- IRQs: link state193- commands: port config, offload config, queue config,194 parse/classify config, IRQ config, enable, reset195 196DPIO (Datapath I/O)197~~~~~~~~~~~~~~~~~~~198Provides interfaces to enqueue and dequeue199packets and do hardware buffer pool management operations. The DPAA2200architecture separates the mechanism to access queues (the DPIO object)201from the queues themselves. The DPIO provides an MMIO interface to202enqueue/dequeue packets. To enqueue something a descriptor is written203to the DPIO MMIO region, which includes the target queue number.204There will typically be one DPIO assigned to each CPU. This allows all205CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are206expected to be shared by different DPAA2 drivers.207 208- MMIO regions: queue operations, buffer management209- IRQs: data availability, congestion notification, buffer210 pool depletion211- commands: IRQ config, enable, reset212 213DPBP (Datapath Buffer Pool)214~~~~~~~~~~~~~~~~~~~~~~~~~~~215Represents a hardware buffer pool.216 217- MMIO regions: none218- IRQs: none219- commands: enable, reset220 221DPMCP (Datapath MC Portal)222~~~~~~~~~~~~~~~~~~~~~~~~~~223Provides an MC command portal.224Used by drivers to send commands to the MC to manage225objects.226 227- MMIO regions: MC command portal228- IRQs: command completion229- commands: IRQ config, enable, reset230 231Object Connections232==================233Some objects have explicit relationships that must234be configured:235 236- DPNI <--> DPMAC237- DPNI <--> DPNI238- DPNI <--> L2-switch-port239 240 A DPNI must be connected to something such as a DPMAC,241 another DPNI, or L2 switch port. The DPNI connection242 is made via a DPRC command.243 244::245 246 +-------+ +-------+247 | DPNI | | DPMAC |248 +---+---+ +---+---+249 | |250 +==========+251 252- DPNI <--> DPBP253 254 A network interface requires a 'buffer pool' (DPBP255 object) which provides a list of pointers to memory256 where received Ethernet data is to be copied. The257 Ethernet driver configures the DPBPs associated with258 the network interface.259 260Interrupts261==========262All interrupts generated by DPAA2 objects are message263interrupts. At the hardware level message interrupts264generated by devices will normally have 3 components--2651) a non-spoofable 'device-id' expressed on the hardware266bus, 2) an address, 3) a data value.267 268In the case of DPAA2 devices/objects, all objects in the269same container/DPRC share the same 'device-id'.270For ARM-based SoC this is the same as the stream ID.271 272 273DPAA2 Linux Drivers Overview274============================275 276This section provides an overview of the Linux kernel drivers for277DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"278drivers and 2) functional object drivers (such as Ethernet).279 280As described previously, a DPRC is a container that holds the other281types of DPAA2 objects. It is functionally similar to a plug-and-play282bus controller.283Each object in the DPRC is a Linux "device" and is bound to a driver.284The diagram below shows the Linux drivers involved in a networking285scenario and the objects bound to each driver. A brief description286of each driver follows.287 288::289 290 +------------+291 | OS Network |292 | Stack |293 +------------+ +------------+294 | Allocator |. . . . . . . | Ethernet |295 |(DPMCP,DPBP)| | (DPNI) |296 +-.----------+ +---+---+----+297 . . ^ |298 . . <data avail, | | <enqueue,299 . . tx confirm> | | dequeue>300 +-------------+ . | |301 | DPRC driver | . +---+---V----+ +---------+302 | (DPRC) | . . . . . .| DPIO driver| | MAC |303 +----------+--+ | (DPIO) | | (DPMAC) |304 | +------+-----+ +-----+---+305 |<dev add/remove> | |306 | | |307 +--------+----------+ | +--+---+308 | MC-bus driver | | | PHY |309 | | | |driver|310 | /bus/fsl-mc | | +--+---+311 +-------------------+ | |312 | |313 ========================= HARDWARE =========|=================|======314 DPIO |315 | |316 DPNI---DPBP |317 | |318 DPMAC |319 | |320 PHY ---------------+321 ============================================|========================322 323A brief description of each driver is provided below.324 325MC-bus driver326-------------327The MC-bus driver is a platform driver and is probed from a328node in the device tree (compatible "fsl,qoriq-mc") passed in by boot329firmware. It is responsible for bootstrapping the DPAA2 kernel330infrastructure.331Key functions include:332 333- registering a new bus type named "fsl-mc" with the kernel,334 and implementing bus call-backs (e.g. match/uevent/dev_groups)335- implementing APIs for DPAA2 driver registration and for device336 add/remove337- creates an MSI IRQ domain338- doing a 'device add' to expose the 'root' DPRC, in turn triggering339 a bind of the root DPRC to the DPRC driver340 341The binding for the MC-bus device-tree node can be consulted at342*Documentation/devicetree/bindings/misc/fsl,qoriq-mc.yaml*.343The sysfs bind/unbind interfaces for the MC-bus can be consulted at344*Documentation/ABI/testing/sysfs-bus-fsl-mc*.345 346DPRC driver347-----------348The DPRC driver is bound to DPRC objects and does runtime management349of a bus instance. It performs the initial bus scan of the DPRC350and handles interrupts for container events such as hot plug by351re-scanning the DPRC.352 353Allocator354---------355Certain objects such as DPMCP and DPBP are generic and fungible,356and are intended to be used by other drivers. For example,357the DPAA2 Ethernet driver needs:358 359- DPMCPs to send MC commands, to configure network interfaces360- DPBPs for network buffer pools361 362The allocator driver registers for these allocatable object types363and those objects are bound to the allocator when the bus is probed.364The allocator maintains a pool of objects that are available for365allocation by other DPAA2 drivers.366 367DPIO driver368-----------369The DPIO driver is bound to DPIO objects and provides services that allow370other drivers such as the Ethernet driver to enqueue and dequeue data for371their respective objects.372Key services include:373 374- data availability notifications375- hardware queuing operations (enqueue and dequeue of data)376- hardware buffer pool management377 378To transmit a packet the Ethernet driver puts data on a queue and379invokes a DPIO API. For receive, the Ethernet driver registers380a data availability notification callback. To dequeue a packet381a DPIO API is used.382There is typically one DPIO object per physical CPU for optimum383performance, allowing different CPUs to simultaneously enqueue384and dequeue data.385 386The DPIO driver operates on behalf of all DPAA2 drivers387active in the kernel-- Ethernet, crypto, compression,388etc.389 390Ethernet driver391---------------392The Ethernet driver is bound to a DPNI and implements the kernel393interfaces needed to connect the DPAA2 network interface to394the network stack.395Each DPNI corresponds to a Linux network interface.396 397MAC driver398----------399An Ethernet PHY is an off-chip, board specific component and is managed400by the appropriate PHY driver via an mdio bus. The MAC driver401plays a role of being a proxy between the PHY driver and the402MC. It does this proxy via the MC commands to a DPMAC object.403If the PHY driver signals a link change, the MAC driver notifies404the MC via a DPMAC command. If a network interface is brought405up or down, the MC notifies the DPMAC driver via an interrupt and406the driver can take appropriate action.407