477 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3===================4ice devlink support5===================6 7This document describes the devlink features implemented by the ``ice``8device driver.9 10Parameters11==========12 13.. list-table:: Generic parameters implemented14 :widths: 5 5 9015 16 * - Name17 - Mode18 - Notes19 * - ``enable_roce``20 - runtime21 - mutually exclusive with ``enable_iwarp``22 * - ``enable_iwarp``23 - runtime24 - mutually exclusive with ``enable_roce``25 * - ``tx_scheduling_layers``26 - permanent27 - The ice hardware uses hierarchical scheduling for Tx with a fixed28 number of layers in the scheduling tree. Each of them are decision29 points. Root node represents a port, while all the leaves represent30 the queues. This way of configuring the Tx scheduler allows features31 like DCB or devlink-rate (documented below) to configure how much32 bandwidth is given to any given queue or group of queues, enabling33 fine-grained control because scheduling parameters can be configured34 at any given layer of the tree.35 36 The default 9-layer tree topology was deemed best for most workloads,37 as it gives an optimal ratio of performance to configurability. However,38 for some specific cases, this 9-layer topology might not be desired.39 One example would be sending traffic to queues that are not a multiple40 of 8. Because the maximum radix is limited to 8 in 9-layer topology,41 the 9th queue has a different parent than the rest, and it's given42 more bandwidth credits. This causes a problem when the system is43 sending traffic to 9 queues:44 45 | tx_queue_0_packets: 2416339646 | tx_queue_1_packets: 2416462347 | tx_queue_2_packets: 2416318848 | tx_queue_3_packets: 2416370149 | tx_queue_4_packets: 2416368350 | tx_queue_5_packets: 2416466851 | tx_queue_6_packets: 2332720052 | tx_queue_7_packets: 2416385353 | tx_queue_8_packets: 91101417 < Too much traffic is sent from 9th54 55 To address this need, you can switch to a 5-layer topology, which56 changes the maximum topology radix to 512. With this enhancement,57 the performance characteristic is equal as all queues can be assigned58 to the same parent in the tree. The obvious drawback of this solution59 is a lower configuration depth of the tree.60 61 Use the ``tx_scheduling_layer`` parameter with the devlink command62 to change the transmit scheduler topology. To use 5-layer topology,63 use a value of 5. For example:64 $ devlink dev param set pci/0000:16:00.0 name tx_scheduling_layers65 value 5 cmode permanent66 Use a value of 9 to set it back to the default value.67 68 You must do PCI slot powercycle for the selected topology to take effect.69 70 To verify that value has been set:71 $ devlink dev param show pci/0000:16:00.0 name tx_scheduling_layers72.. list-table:: Driver specific parameters implemented73 :widths: 5 5 9074 75 * - Name76 - Mode77 - Description78 * - ``local_forwarding``79 - runtime80 - Controls loopback behavior by tuning scheduler bandwidth.81 It impacts all kinds of functions: physical, virtual and82 subfunctions.83 Supported values are:84 85 ``enabled`` - loopback traffic is allowed on port86 87 ``disabled`` - loopback traffic is not allowed on this port88 89 ``prioritized`` - loopback traffic is prioritized on this port90 91 Default value of ``local_forwarding`` parameter is ``enabled``.92 ``prioritized`` provides ability to adjust loopback traffic rate to increase93 one port capacity at cost of the another. User needs to disable94 local forwarding on one of the ports in order have increased capacity95 on the ``prioritized`` port.96 97Info versions98=============99 100The ``ice`` driver reports the following versions101 102.. list-table:: devlink info versions implemented103 :widths: 5 5 5 90104 105 * - Name106 - Type107 - Example108 - Description109 * - ``board.id``110 - fixed111 - K65390-000112 - The Product Board Assembly (PBA) identifier of the board.113 * - ``cgu.id``114 - fixed115 - 36116 - The Clock Generation Unit (CGU) hardware revision identifier.117 * - ``fw.mgmt``118 - running119 - 2.1.7120 - 3-digit version number of the management firmware running on the121 Embedded Management Processor of the device. It controls the PHY,122 link, access to device resources, etc. Intel documentation refers to123 this as the EMP firmware.124 * - ``fw.mgmt.api``125 - running126 - 1.5.1127 - 3-digit version number (major.minor.patch) of the API exported over128 the AdminQ by the management firmware. Used by the driver to129 identify what commands are supported. Historical versions of the130 kernel only displayed a 2-digit version number (major.minor).131 * - ``fw.mgmt.build``132 - running133 - 0x305d955f134 - Unique identifier of the source for the management firmware.135 * - ``fw.undi``136 - running137 - 1.2581.0138 - Version of the Option ROM containing the UEFI driver. The version is139 reported in ``major.minor.patch`` format. The major version is140 incremented whenever a major breaking change occurs, or when the141 minor version would overflow. The minor version is incremented for142 non-breaking changes and reset to 1 when the major version is143 incremented. The patch version is normally 0 but is incremented when144 a fix is delivered as a patch against an older base Option ROM.145 * - ``fw.psid.api``146 - running147 - 0.80148 - Version defining the format of the flash contents.149 * - ``fw.bundle_id``150 - running151 - 0x80002ec0152 - Unique identifier of the firmware image file that was loaded onto153 the device. Also referred to as the EETRACK identifier of the NVM.154 * - ``fw.app.name``155 - running156 - ICE OS Default Package157 - The name of the DDP package that is active in the device. The DDP158 package is loaded by the driver during initialization. Each159 variation of the DDP package has a unique name.160 * - ``fw.app``161 - running162 - 1.3.1.0163 - The version of the DDP package that is active in the device. Note164 that both the name (as reported by ``fw.app.name``) and version are165 required to uniquely identify the package.166 * - ``fw.app.bundle_id``167 - running168 - 0xc0000001169 - Unique identifier for the DDP package loaded in the device. Also170 referred to as the DDP Track ID. Can be used to uniquely identify171 the specific DDP package.172 * - ``fw.netlist``173 - running174 - 1.1.2000-6.7.0175 - The version of the netlist module. This module defines the device's176 Ethernet capabilities and default settings, and is used by the177 management firmware as part of managing link and device178 connectivity.179 * - ``fw.netlist.build``180 - running181 - 0xee16ced7182 - The first 4 bytes of the hash of the netlist module contents.183 * - ``fw.cgu``184 - running185 - 8032.16973825.6021186 - The version of Clock Generation Unit (CGU). Format:187 <CGU type>.<configuration version>.<firmware version>.188 189Flash Update190============191 192The ``ice`` driver implements support for flash update using the193``devlink-flash`` interface. It supports updating the device flash using a194combined flash image that contains the ``fw.mgmt``, ``fw.undi``, and195``fw.netlist`` components.196 197.. list-table:: List of supported overwrite modes198 :widths: 5 95199 200 * - Bits201 - Behavior202 * - ``DEVLINK_FLASH_OVERWRITE_SETTINGS``203 - Do not preserve settings stored in the flash components being204 updated. This includes overwriting the port configuration that205 determines the number of physical functions the device will206 initialize with.207 * - ``DEVLINK_FLASH_OVERWRITE_SETTINGS`` and ``DEVLINK_FLASH_OVERWRITE_IDENTIFIERS``208 - Do not preserve either settings or identifiers. Overwrite everything209 in the flash with the contents from the provided image, without210 performing any preservation. This includes overwriting device211 identifying fields such as the MAC address, VPD area, and device212 serial number. It is expected that this combination be used with an213 image customized for the specific device.214 215The ice hardware does not support overwriting only identifiers while216preserving settings, and thus ``DEVLINK_FLASH_OVERWRITE_IDENTIFIERS`` on its217own will be rejected. If no overwrite mask is provided, the firmware will be218instructed to preserve all settings and identifying fields when updating.219 220Reload221======222 223The ``ice`` driver supports activating new firmware after a flash update224using ``DEVLINK_CMD_RELOAD`` with the ``DEVLINK_RELOAD_ACTION_FW_ACTIVATE``225action.226 227.. code:: shell228 229 $ devlink dev reload pci/0000:01:00.0 reload action fw_activate230 231The new firmware is activated by issuing a device specific Embedded232Management Processor reset which requests the device to reset and reload the233EMP firmware image.234 235The driver does not currently support reloading the driver via236``DEVLINK_RELOAD_ACTION_DRIVER_REINIT``.237 238Port split239==========240 241The ``ice`` driver supports port splitting only for port 0, as the FW has242a predefined set of available port split options for the whole device.243 244A system reboot is required for port split to be applied.245 246The following command will select the port split option with 4 ports:247 248.. code:: shell249 250 $ devlink port split pci/0000:16:00.0/0 count 4251 252The list of all available port options will be printed to dynamic debug after253each ``split`` and ``unsplit`` command. The first option is the default.254 255.. code:: shell256 257 ice 0000:16:00.0: Available port split options and max port speeds (Gbps):258 ice 0000:16:00.0: Status Split Quad 0 Quad 1259 ice 0000:16:00.0: count L0 L1 L2 L3 L4 L5 L6 L7260 ice 0000:16:00.0: Active 2 100 - - - 100 - - -261 ice 0000:16:00.0: 2 50 - 50 - - - - -262 ice 0000:16:00.0: Pending 4 25 25 25 25 - - - -263 ice 0000:16:00.0: 4 25 25 - - 25 25 - -264 ice 0000:16:00.0: 8 10 10 10 10 10 10 10 10265 ice 0000:16:00.0: 1 100 - - - - - - -266 267There could be multiple FW port options with the same port split count. When268the same port split count request is issued again, the next FW port option with269the same port split count will be selected.270 271``devlink port unsplit`` will select the option with a split count of 1. If272there is no FW option available with split count 1, you will receive an error.273 274Regions275=======276 277The ``ice`` driver implements the following regions for accessing internal278device data.279 280.. list-table:: regions implemented281 :widths: 15 85282 283 * - Name284 - Description285 * - ``nvm-flash``286 - The contents of the entire flash chip, sometimes referred to as287 the device's Non Volatile Memory.288 * - ``shadow-ram``289 - The contents of the Shadow RAM, which is loaded from the beginning290 of the flash. Although the contents are primarily from the flash,291 this area also contains data generated during device boot which is292 not stored in flash.293 * - ``device-caps``294 - The contents of the device firmware's capabilities buffer. Useful to295 determine the current state and configuration of the device.296 297Both the ``nvm-flash`` and ``shadow-ram`` regions can be accessed without a298snapshot. The ``device-caps`` region requires a snapshot as the contents are299sent by firmware and can't be split into separate reads.300 301Users can request an immediate capture of a snapshot for all three regions302via the ``DEVLINK_CMD_REGION_NEW`` command.303 304.. code:: shell305 306 $ devlink region show307 pci/0000:01:00.0/nvm-flash: size 10485760 snapshot [] max 1308 pci/0000:01:00.0/device-caps: size 4096 snapshot [] max 10309 310 $ devlink region new pci/0000:01:00.0/nvm-flash snapshot 1311 $ devlink region dump pci/0000:01:00.0/nvm-flash snapshot 1312 313 $ devlink region dump pci/0000:01:00.0/nvm-flash snapshot 1314 0000000000000000 0014 95dc 0014 9514 0035 1670 0034 db30315 0000000000000010 0000 0000 ffff ff04 0029 8c00 0028 8cc8316 0000000000000020 0016 0bb8 0016 1720 0000 0000 c00f 3ffc317 0000000000000030 bada cce5 bada cce5 bada cce5 bada cce5318 319 $ devlink region read pci/0000:01:00.0/nvm-flash snapshot 1 address 0 length 16320 0000000000000000 0014 95dc 0014 9514 0035 1670 0034 db30321 322 $ devlink region delete pci/0000:01:00.0/nvm-flash snapshot 1323 324 $ devlink region new pci/0000:01:00.0/device-caps snapshot 1325 $ devlink region dump pci/0000:01:00.0/device-caps snapshot 1326 0000000000000000 01 00 01 00 00 00 00 00 01 00 00 00 00 00 00 00327 0000000000000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00328 0000000000000020 02 00 02 01 32 03 00 00 0a 00 00 00 25 00 00 00329 0000000000000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00330 0000000000000040 04 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00331 0000000000000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00332 0000000000000060 05 00 01 00 03 00 00 00 00 00 00 00 00 00 00 00333 0000000000000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00334 0000000000000080 06 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00335 0000000000000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00336 00000000000000a0 08 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00337 00000000000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00338 00000000000000c0 12 00 01 00 01 00 00 00 01 00 01 00 00 00 00 00339 00000000000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00340 00000000000000e0 13 00 01 00 00 01 00 00 00 00 00 00 00 00 00 00341 00000000000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00342 0000000000000100 14 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00343 0000000000000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00344 0000000000000120 15 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00345 0000000000000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00346 0000000000000140 16 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00347 0000000000000150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00348 0000000000000160 17 00 01 00 06 00 00 00 00 00 00 00 00 00 00 00349 0000000000000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00350 0000000000000180 18 00 01 00 01 00 00 00 01 00 00 00 08 00 00 00351 0000000000000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00352 00000000000001a0 22 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00353 00000000000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00354 00000000000001c0 40 00 01 00 00 08 00 00 08 00 00 00 00 00 00 00355 00000000000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00356 00000000000001e0 41 00 01 00 00 08 00 00 00 00 00 00 00 00 00 00357 00000000000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00358 0000000000000200 42 00 01 00 00 08 00 00 00 00 00 00 00 00 00 00359 0000000000000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00360 361 $ devlink region delete pci/0000:01:00.0/device-caps snapshot 1362 363Devlink Rate364============365 366The ``ice`` driver implements devlink-rate API. It allows for offload of367the Hierarchical QoS to the hardware. It enables user to group Virtual368Functions in a tree structure and assign supported parameters: tx_share,369tx_max, tx_priority and tx_weight to each node in a tree. So effectively370user gains an ability to control how much bandwidth is allocated for each371VF group. This is later enforced by the HW.372 373It is assumed that this feature is mutually exclusive with DCB performed374in FW and ADQ, or any driver feature that would trigger changes in QoS,375for example creation of the new traffic class. The driver will prevent DCB376or ADQ configuration if user started making any changes to the nodes using377devlink-rate API. To configure those features a driver reload is necessary.378Correspondingly if ADQ or DCB will get configured the driver won't export379hierarchy at all, or will remove the untouched hierarchy if those380features are enabled after the hierarchy is exported, but before any381changes are made.382 383This feature is also dependent on switchdev being enabled in the system.384It's required because devlink-rate requires devlink-port objects to be385present, and those objects are only created in switchdev mode.386 387If the driver is set to the switchdev mode, it will export internal388hierarchy the moment VF's are created. Root of the tree is always389represented by the node_0. This node can't be deleted by the user. Leaf390nodes and nodes with children also can't be deleted.391 392.. list-table:: Attributes supported393 :widths: 15 85394 395 * - Name396 - Description397 * - ``tx_max``398 - maximum bandwidth to be consumed by the tree Node. Rate Limit is399 an absolute number specifying a maximum amount of bytes a Node may400 consume during the course of one second. Rate limit guarantees401 that a link will not oversaturate the receiver on the remote end402 and also enforces an SLA between the subscriber and network403 provider.404 * - ``tx_share``405 - minimum bandwidth allocated to a tree node when it is not blocked.406 It specifies an absolute BW. While tx_max defines the maximum407 bandwidth the node may consume, the tx_share marks committed BW408 for the Node.409 * - ``tx_priority``410 - allows for usage of strict priority arbiter among siblings. This411 arbitration scheme attempts to schedule nodes based on their412 priority as long as the nodes remain within their bandwidth limit.413 Range 0-7. Nodes with priority 7 have the highest priority and are414 selected first, while nodes with priority 0 have the lowest415 priority. Nodes that have the same priority are treated equally.416 * - ``tx_weight``417 - allows for usage of Weighted Fair Queuing arbitration scheme among418 siblings. This arbitration scheme can be used simultaneously with419 the strict priority. Range 1-200. Only relative values matter for420 arbitration.421 422``tx_priority`` and ``tx_weight`` can be used simultaneously. In that case423nodes with the same priority form a WFQ subgroup in the sibling group424and arbitration among them is based on assigned weights.425 426.. code:: shell427 428 # enable switchdev429 $ devlink dev eswitch set pci/0000:4b:00.0 mode switchdev430 431 # at this point driver should export internal hierarchy432 $ echo 2 > /sys/class/net/ens785np0/device/sriov_numvfs433 434 $ devlink port function rate show435 pci/0000:4b:00.0/node_25: type node parent node_24436 pci/0000:4b:00.0/node_24: type node parent node_0437 pci/0000:4b:00.0/node_32: type node parent node_31438 pci/0000:4b:00.0/node_31: type node parent node_30439 pci/0000:4b:00.0/node_30: type node parent node_16440 pci/0000:4b:00.0/node_19: type node parent node_18441 pci/0000:4b:00.0/node_18: type node parent node_17442 pci/0000:4b:00.0/node_17: type node parent node_16443 pci/0000:4b:00.0/node_14: type node parent node_5444 pci/0000:4b:00.0/node_5: type node parent node_3445 pci/0000:4b:00.0/node_13: type node parent node_4446 pci/0000:4b:00.0/node_12: type node parent node_4447 pci/0000:4b:00.0/node_11: type node parent node_4448 pci/0000:4b:00.0/node_10: type node parent node_4449 pci/0000:4b:00.0/node_9: type node parent node_4450 pci/0000:4b:00.0/node_8: type node parent node_4451 pci/0000:4b:00.0/node_7: type node parent node_4452 pci/0000:4b:00.0/node_6: type node parent node_4453 pci/0000:4b:00.0/node_4: type node parent node_3454 pci/0000:4b:00.0/node_3: type node parent node_16455 pci/0000:4b:00.0/node_16: type node parent node_15456 pci/0000:4b:00.0/node_15: type node parent node_0457 pci/0000:4b:00.0/node_2: type node parent node_1458 pci/0000:4b:00.0/node_1: type node parent node_0459 pci/0000:4b:00.0/node_0: type node460 pci/0000:4b:00.0/1: type leaf parent node_25461 pci/0000:4b:00.0/2: type leaf parent node_25462 463 # let's create some custom node464 $ devlink port function rate add pci/0000:4b:00.0/node_custom parent node_0465 466 # second custom node467 $ devlink port function rate add pci/0000:4b:00.0/node_custom_1 parent node_custom468 469 # reassign second VF to newly created branch470 $ devlink port function rate set pci/0000:4b:00.0/2 parent node_custom_1471 472 # assign tx_weight to the VF473 $ devlink port function rate set pci/0000:4b:00.0/2 tx_weight 5474 475 # assign tx_share to the VF476 $ devlink port function rate set pci/0000:4b:00.0/2 tx_share 500Mbps477