375 lines · plain
1.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)2.. include:: <isonum.txt>3 4===========================================5Network Flow Processor (NFP) Kernel Drivers6===========================================7 8:Copyright: |copy| 2019, Netronome Systems, Inc.9:Copyright: |copy| 2022, Corigine, Inc.10 11Contents12========13 14- `Overview`_15- `Acquiring Firmware`_16- `Devlink Info`_17- `Configure Device`_18- `Statistics`_19 20Overview21========22 23This driver supports Netronome and Corigine's line of Network Flow Processor24devices, including the NFP3800, NFP4000, NFP5000, and NFP6000 models, which25are also incorporated in the companies' family of Agilio SmartNICs. The SR-IOV26physical and virtual functions for these devices are supported by the driver.27 28Acquiring Firmware29==================30 31The NFP3800, NFP4000 and NFP6000 devices require application specific firmware32to function. Application firmware can be located either on the host file system33or in the device flash (if supported by management firmware).34 35Firmware files on the host filesystem contain card type (`AMDA-*` string), media36config etc. They should be placed in `/lib/firmware/netronome` directory to37load firmware from the host file system.38 39Firmware for basic NIC operation is available in the upstream40`linux-firmware.git` repository.41 42A more comprehensive list of firmware can be downloaded from the43`Corigine Support site <https://www.corigine.com/DPUDownload.html>`_.44 45Firmware in NVRAM46-----------------47 48Recent versions of management firmware supports loading application49firmware from flash when the host driver gets probed. The firmware loading50policy configuration may be used to configure this feature appropriately.51 52Devlink or ethtool can be used to update the application firmware on the device53flash by providing the appropriate `nic_AMDA*.nffw` file to the respective54command. Users need to take care to write the correct firmware image for the55card and media configuration to flash.56 57Available storage space in flash depends on the card being used.58 59Dealing with multiple projects60------------------------------61 62NFP hardware is fully programmable therefore there can be different63firmware images targeting different applications.64 65When using application firmware from host, we recommend placing66actual firmware files in application-named subdirectories in67`/lib/firmware/netronome` and linking the desired files, e.g.::68 69 $ tree /lib/firmware/netronome/70 /lib/firmware/netronome/71 ├── bpf72 │ ├── nic_AMDA0081-0001_1x40.nffw73 │ └── nic_AMDA0081-0001_4x10.nffw74 ├── flower75 │ ├── nic_AMDA0081-0001_1x40.nffw76 │ └── nic_AMDA0081-0001_4x10.nffw77 ├── nic78 │ ├── nic_AMDA0081-0001_1x40.nffw79 │ └── nic_AMDA0081-0001_4x10.nffw80 ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw81 └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw82 83 3 directories, 8 files84 85You may need to use hard instead of symbolic links on distributions86which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu).87 88After changing firmware files you may need to regenerate the initramfs89image. Initramfs contains drivers and firmware files your system may90need to boot. Refer to the documentation of your distribution to find91out how to update initramfs. Good indication of stale initramfs92is system loading wrong driver or firmware on boot, but when driver is93later reloaded manually everything works correctly.94 95Selecting firmware per device96-----------------------------97 98Most commonly all cards on the system use the same type of firmware.99If you want to load a specific firmware image for a specific card, you100can use either the PCI bus address or serial number. The driver will101print which files it's looking for when it recognizes a NFP device::102 103 nfp: Looking for firmware file in order of priority:104 nfp: netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found105 nfp: netronome/pci-0000:02:00.0.nffw: not found106 nfp: netronome/nic_AMDA0081-0001_1x40.nffw: found, loading...107 108In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw*109or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this110firmware file will take precedence over `nic_AMDA*` files.111 112Note that `serial-*` and `pci-*` files are **not** automatically included113in initramfs, you will have to refer to documentation of appropriate tools114to find out how to include them.115 116Running firmware version117------------------------118 119The version of the loaded firmware for a particular <netdev> interface,120(e.g. enp4s0), or an interface's port <netdev port> (e.g. enp4s0np0) can121be displayed with the ethtool command::122 123 $ ethtool -i <netdev>124 125Firmware loading policy126-----------------------127 128Firmware loading policy is controlled via three HWinfo parameters129stored as key value pairs in the device flash:130 131app_fw_from_flash132 Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or133 the 'Preferred' (2) firmware. When 'Preferred' is selected, the management134 firmware makes the decision over which firmware will be loaded by comparing135 versions of the flash firmware and the host supplied firmware.136 This variable is configurable using the 'fw_load_policy'137 devlink parameter.138 139abi_drv_reset140 Defines if the driver should reset the firmware when141 the driver is probed, either 'Disk' (0) if firmware was found on disk,142 'Always' (1) reset or 'Never' (2) reset. Note that the device is always143 reset on driver unload if firmware was loaded when the driver was probed.144 This variable is configurable using the 'reset_dev_on_drv_probe'145 devlink parameter.146 147abi_drv_load_ifc148 Defines a list of PF devices allowed to load FW on the device.149 This variable is not currently user configurable.150 151Devlink Info152============153 154The devlink info command displays the running and stored firmware versions155on the device, serial number and board information.156 157Devlink info command example (replace PCI address)::158 159 $ devlink dev info pci/0000:03:00.0160 pci/0000:03:00.0:161 driver nfp162 serial_number CSAAMDA2001-1003000111163 versions:164 fixed:165 board.id AMDA2001-1003166 board.rev 01167 board.manufacture CSA168 board.model mozart169 running:170 fw.mgmt 22.10.0-rc3171 fw.cpld 0x1000003172 fw.app nic-22.09.0173 chip.init AMDA-2001-1003 1003000111174 stored:175 fw.bundle_id bspbundle_1003000111176 fw.mgmt 22.10.0-rc3177 fw.cpld 0x0178 chip.init AMDA-2001-1003 1003000111179 180Configure Device181================182 183This section explains how to use Agilio SmartNICs running basic NIC firmware.184 185Configure interface link-speed186------------------------------187The following steps explains how to change between 10G mode and 25G mode on188Agilio CX 2x25GbE cards. The changing of port speed must be done in order,189port 0 (p0) must be set to 10G before port 1 (p1) may be set to 10G.190 191Down the respective interface(s)::192 193 $ ip link set dev <netdev port 0> down194 $ ip link set dev <netdev port 1> down195 196Set interface link-speed to 10G::197 198 $ ethtool -s <netdev port 0> speed 10000199 $ ethtool -s <netdev port 1> speed 10000200 201Set interface link-speed to 25G::202 203 $ ethtool -s <netdev port 0> speed 25000204 $ ethtool -s <netdev port 1> speed 25000205 206Reload driver for changes to take effect::207 208 $ rmmod nfp; modprobe nfp209 210Configure interface Maximum Transmission Unit (MTU)211---------------------------------------------------212 213The MTU of interfaces can temporarily be set using the iproute2, ip link or214ifconfig tools. Note that this change will not persist. Setting this via215Network Manager, or another appropriate OS configuration tool, is216recommended as changes to the MTU using Network Manager can be made to217persist.218 219Set interface MTU to 9000 bytes::220 221 $ ip link set dev <netdev port> mtu 9000222 223It is the responsibility of the user or the orchestration layer to set224appropriate MTU values when handling jumbo frames or utilizing tunnels. For225example, if packets sent from a VM are to be encapsulated on the card and226egress a physical port, then the MTU of the VF should be set to lower than227that of the physical port to account for the extra bytes added by the228additional header. If a setup is expected to see fallback traffic between229the SmartNIC and the kernel then the user should also ensure that the PF MTU230is appropriately set to avoid unexpected drops on this path.231 232Configure Forward Error Correction (FEC) modes233----------------------------------------------234 235Agilio SmartNICs support FEC mode configuration, e.g. Auto, Firecode Base-R,236ReedSolomon and Off modes. Each physical port's FEC mode can be set237independently using ethtool. The supported FEC modes for an interface can238be viewed using::239 240 $ ethtool <netdev>241 242The currently configured FEC mode can be viewed using::243 244 $ ethtool --show-fec <netdev>245 246To force the FEC mode for a particular port, auto-negotiation must be disabled247(see the `Auto-negotiation`_ section). An example of how to set the FEC mode248to Reed-Solomon is::249 250 $ ethtool --set-fec <netdev> encoding rs251 252Auto-negotiation253----------------254 255To change auto-negotiation settings, the link must first be put down. After the256link is down, auto-negotiation can be enabled or disabled using::257 258 ethtool -s <netdev> autoneg <on|off>259 260Statistics261==========262 263Following device statistics are available through the ``ethtool -S`` interface:264 265.. flat-table:: NFP device statistics266 :header-rows: 1267 :widths: 3 1 11268 269 * - Name270 - ID271 - Meaning272 273 * - dev_rx_discards274 - 1275 - Packet can be discarded on the RX path for one of the following reasons:276 277 * The NIC is not in promisc mode, and the destination MAC address278 doesn't match the interfaces' MAC address.279 * The received packet is larger than the max buffer size on the host.280 I.e. it exceeds the Layer 3 MRU.281 * There is no freelist descriptor available on the host for the packet.282 It is likely that the NIC couldn't cache one in time.283 * A BPF program discarded the packet.284 * The datapath drop action was executed.285 * The MAC discarded the packet due to lack of ingress buffer space286 on the NIC.287 288 * - dev_rx_errors289 - 2290 - A packet can be counted (and dropped) as RX error for the following291 reasons:292 293 * A problem with the VEB lookup (only when SR-IOV is used).294 * A physical layer problem that causes Ethernet errors, like FCS or295 alignment errors. The cause is usually faulty cables or SFPs.296 297 * - dev_rx_bytes298 - 3299 - Total number of bytes received.300 301 * - dev_rx_uc_bytes302 - 4303 - Unicast bytes received.304 305 * - dev_rx_mc_bytes306 - 5307 - Multicast bytes received.308 309 * - dev_rx_bc_bytes310 - 6311 - Broadcast bytes received.312 313 * - dev_rx_pkts314 - 7315 - Total number of packets received.316 317 * - dev_rx_mc_pkts318 - 8319 - Multicast packets received.320 321 * - dev_rx_bc_pkts322 - 9323 - Broadcast packets received.324 325 * - dev_tx_discards326 - 10327 - A packet can be discarded in the TX direction if the MAC is328 being flow controlled and the NIC runs out of TX queue space.329 330 * - dev_tx_errors331 - 11332 - A packet can be counted as TX error (and dropped) for one for the333 following reasons:334 335 * The packet is an LSO segment, but the Layer 3 or Layer 4 offset336 could not be determined. Therefore LSO could not continue.337 * An invalid packet descriptor was received over PCIe.338 * The packet Layer 3 length exceeds the device MTU.339 * An error on the MAC/physical layer. Usually due to faulty cables or340 SFPs.341 * A CTM buffer could not be allocated.342 * The packet offset was incorrect and could not be fixed by the NIC.343 344 * - dev_tx_bytes345 - 12346 - Total number of bytes transmitted.347 348 * - dev_tx_uc_bytes349 - 13350 - Unicast bytes transmitted.351 352 * - dev_tx_mc_bytes353 - 14354 - Multicast bytes transmitted.355 356 * - dev_tx_bc_bytes357 - 15358 - Broadcast bytes transmitted.359 360 * - dev_tx_pkts361 - 16362 - Total number of packets transmitted.363 364 * - dev_tx_mc_pkts365 - 17366 - Multicast packets transmitted.367 368 * - dev_tx_bc_pkts369 - 18370 - Broadcast packets transmitted.371 372Note that statistics unknown to the driver will be displayed as373``dev_unknown_stat$ID``, where ``$ID`` refers to the second column374above.375