641 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3============4Devlink Trap5============6 7Background8==========9 10Devices capable of offloading the kernel's datapath and perform functions such11as bridging and routing must also be able to send specific packets to the12kernel (i.e., the CPU) for processing.13 14For example, a device acting as a multicast-aware bridge must be able to send15IGMP membership reports to the kernel for processing by the bridge module.16Without processing such packets, the bridge module could never populate its17MDB.18 19As another example, consider a device acting as router which has received an IP20packet with a TTL of 1. Upon routing the packet the device must send it to the21kernel so that it will route it as well and generate an ICMP Time Exceeded22error datagram. Without letting the kernel route such packets itself, utilities23such as ``traceroute`` could never work.24 25The fundamental ability of sending certain packets to the kernel for processing26is called "packet trapping".27 28Overview29========30 31The ``devlink-trap`` mechanism allows capable device drivers to register their32supported packet traps with ``devlink`` and report trapped packets to33``devlink`` for further analysis.34 35Upon receiving trapped packets, ``devlink`` will perform a per-trap packets and36bytes accounting and potentially report the packet to user space via a netlink37event along with all the provided metadata (e.g., trap reason, timestamp, input38port). This is especially useful for drop traps (see :ref:`Trap-Types`)39as it allows users to obtain further visibility into packet drops that would40otherwise be invisible.41 42The following diagram provides a general overview of ``devlink-trap``::43 44 Netlink event: Packet w/ metadata45 Or a summary of recent drops46 ^47 |48 Userspace |49 +---------------------------------------------------+50 Kernel |51 |52 +-------+--------+53 | |54 | drop_monitor |55 | |56 +-------^--------+57 |58 | Non-control traps59 |60 +----+----+61 | | Kernel's Rx path62 | devlink | (non-drop traps)63 | |64 +----^----+ ^65 | |66 +-----------+67 |68 +-------+-------+69 | |70 | Device driver |71 | |72 +-------^-------+73 Kernel |74 +---------------------------------------------------+75 Hardware |76 | Trapped packet77 |78 +--+---+79 | |80 | ASIC |81 | |82 +------+83 84.. _Trap-Types:85 86Trap Types87==========88 89The ``devlink-trap`` mechanism supports the following packet trap types:90 91 * ``drop``: Trapped packets were dropped by the underlying device. Packets92 are only processed by ``devlink`` and not injected to the kernel's Rx path.93 The trap action (see :ref:`Trap-Actions`) can be changed.94 * ``exception``: Trapped packets were not forwarded as intended by the95 underlying device due to an exception (e.g., TTL error, missing neighbour96 entry) and trapped to the control plane for resolution. Packets are97 processed by ``devlink`` and injected to the kernel's Rx path. Changing the98 action of such traps is not allowed, as it can easily break the control99 plane.100 * ``control``: Trapped packets were trapped by the device because these are101 control packets required for the correct functioning of the control plane.102 For example, ARP request and IGMP query packets. Packets are injected to103 the kernel's Rx path, but not reported to the kernel's drop monitor.104 Changing the action of such traps is not allowed, as it can easily break105 the control plane.106 107.. _Trap-Actions:108 109Trap Actions110============111 112The ``devlink-trap`` mechanism supports the following packet trap actions:113 114 * ``trap``: The sole copy of the packet is sent to the CPU.115 * ``drop``: The packet is dropped by the underlying device and a copy is not116 sent to the CPU.117 * ``mirror``: The packet is forwarded by the underlying device and a copy is118 sent to the CPU.119 120Generic Packet Traps121====================122 123Generic packet traps are used to describe traps that trap well-defined packets124or packets that are trapped due to well-defined conditions (e.g., TTL error).125Such traps can be shared by multiple device drivers and their description must126be added to the following table:127 128.. list-table:: List of Generic Packet Traps129 :widths: 5 5 90130 131 * - Name132 - Type133 - Description134 * - ``source_mac_is_multicast``135 - ``drop``136 - Traps incoming packets that the device decided to drop because of a137 multicast source MAC138 * - ``vlan_tag_mismatch``139 - ``drop``140 - Traps incoming packets that the device decided to drop in case of VLAN141 tag mismatch: The ingress bridge port is not configured with a PVID and142 the packet is untagged or prio-tagged143 * - ``ingress_vlan_filter``144 - ``drop``145 - Traps incoming packets that the device decided to drop in case they are146 tagged with a VLAN that is not configured on the ingress bridge port147 * - ``ingress_spanning_tree_filter``148 - ``drop``149 - Traps incoming packets that the device decided to drop in case the STP150 state of the ingress bridge port is not "forwarding"151 * - ``port_list_is_empty``152 - ``drop``153 - Traps packets that the device decided to drop in case they need to be154 flooded (e.g., unknown unicast, unregistered multicast) and there are155 no ports the packets should be flooded to156 * - ``port_loopback_filter``157 - ``drop``158 - Traps packets that the device decided to drop in case after layer 2159 forwarding the only port from which they should be transmitted through160 is the port from which they were received161 * - ``blackhole_route``162 - ``drop``163 - Traps packets that the device decided to drop in case they hit a164 blackhole route165 * - ``ttl_value_is_too_small``166 - ``exception``167 - Traps unicast packets that should be forwarded by the device whose TTL168 was decremented to 0 or less169 * - ``tail_drop``170 - ``drop``171 - Traps packets that the device decided to drop because they could not be172 enqueued to a transmission queue which is full173 * - ``non_ip``174 - ``drop``175 - Traps packets that the device decided to drop because they need to176 undergo a layer 3 lookup, but are not IP or MPLS packets177 * - ``uc_dip_over_mc_dmac``178 - ``drop``179 - Traps packets that the device decided to drop because they need to be180 routed and they have a unicast destination IP and a multicast destination181 MAC182 * - ``dip_is_loopback_address``183 - ``drop``184 - Traps packets that the device decided to drop because they need to be185 routed and their destination IP is the loopback address (i.e., 127.0.0.0/8186 and ::1/128)187 * - ``sip_is_mc``188 - ``drop``189 - Traps packets that the device decided to drop because they need to be190 routed and their source IP is multicast (i.e., 224.0.0.0/8 and ff::/8)191 * - ``sip_is_loopback_address``192 - ``drop``193 - Traps packets that the device decided to drop because they need to be194 routed and their source IP is the loopback address (i.e., 127.0.0.0/8 and ::1/128)195 * - ``ip_header_corrupted``196 - ``drop``197 - Traps packets that the device decided to drop because they need to be198 routed and their IP header is corrupted: wrong checksum, wrong IP version199 or too short Internet Header Length (IHL)200 * - ``ipv4_sip_is_limited_bc``201 - ``drop``202 - Traps packets that the device decided to drop because they need to be203 routed and their source IP is limited broadcast (i.e., 255.255.255.255/32)204 * - ``ipv6_mc_dip_reserved_scope``205 - ``drop``206 - Traps IPv6 packets that the device decided to drop because they need to207 be routed and their IPv6 multicast destination IP has a reserved scope208 (i.e., ffx0::/16)209 * - ``ipv6_mc_dip_interface_local_scope``210 - ``drop``211 - Traps IPv6 packets that the device decided to drop because they need to212 be routed and their IPv6 multicast destination IP has an interface-local scope213 (i.e., ffx1::/16)214 * - ``mtu_value_is_too_small``215 - ``exception``216 - Traps packets that should have been routed by the device, but were bigger217 than the MTU of the egress interface218 * - ``unresolved_neigh``219 - ``exception``220 - Traps packets that did not have a matching IP neighbour after routing221 * - ``mc_reverse_path_forwarding``222 - ``exception``223 - Traps multicast IP packets that failed reverse-path forwarding (RPF)224 check during multicast routing225 * - ``reject_route``226 - ``exception``227 - Traps packets that hit reject routes (i.e., "unreachable", "prohibit")228 * - ``ipv4_lpm_miss``229 - ``exception``230 - Traps unicast IPv4 packets that did not match any route231 * - ``ipv6_lpm_miss``232 - ``exception``233 - Traps unicast IPv6 packets that did not match any route234 * - ``non_routable_packet``235 - ``drop``236 - Traps packets that the device decided to drop because they are not237 supposed to be routed. For example, IGMP queries can be flooded by the238 device in layer 2 and reach the router. Such packets should not be239 routed and instead dropped240 * - ``decap_error``241 - ``exception``242 - Traps NVE and IPinIP packets that the device decided to drop because of243 failure during decapsulation (e.g., packet being too short, reserved244 bits set in VXLAN header)245 * - ``overlay_smac_is_mc``246 - ``drop``247 - Traps NVE packets that the device decided to drop because their overlay248 source MAC is multicast249 * - ``ingress_flow_action_drop``250 - ``drop``251 - Traps packets dropped during processing of ingress flow action drop252 * - ``egress_flow_action_drop``253 - ``drop``254 - Traps packets dropped during processing of egress flow action drop255 * - ``stp``256 - ``control``257 - Traps STP packets258 * - ``lacp``259 - ``control``260 - Traps LACP packets261 * - ``lldp``262 - ``control``263 - Traps LLDP packets264 * - ``igmp_query``265 - ``control``266 - Traps IGMP Membership Query packets267 * - ``igmp_v1_report``268 - ``control``269 - Traps IGMP Version 1 Membership Report packets270 * - ``igmp_v2_report``271 - ``control``272 - Traps IGMP Version 2 Membership Report packets273 * - ``igmp_v3_report``274 - ``control``275 - Traps IGMP Version 3 Membership Report packets276 * - ``igmp_v2_leave``277 - ``control``278 - Traps IGMP Version 2 Leave Group packets279 * - ``mld_query``280 - ``control``281 - Traps MLD Multicast Listener Query packets282 * - ``mld_v1_report``283 - ``control``284 - Traps MLD Version 1 Multicast Listener Report packets285 * - ``mld_v2_report``286 - ``control``287 - Traps MLD Version 2 Multicast Listener Report packets288 * - ``mld_v1_done``289 - ``control``290 - Traps MLD Version 1 Multicast Listener Done packets291 * - ``ipv4_dhcp``292 - ``control``293 - Traps IPv4 DHCP packets294 * - ``ipv6_dhcp``295 - ``control``296 - Traps IPv6 DHCP packets297 * - ``arp_request``298 - ``control``299 - Traps ARP request packets300 * - ``arp_response``301 - ``control``302 - Traps ARP response packets303 * - ``arp_overlay``304 - ``control``305 - Traps NVE-decapsulated ARP packets that reached the overlay network.306 This is required, for example, when the address that needs to be307 resolved is a local address308 * - ``ipv6_neigh_solicit``309 - ``control``310 - Traps IPv6 Neighbour Solicitation packets311 * - ``ipv6_neigh_advert``312 - ``control``313 - Traps IPv6 Neighbour Advertisement packets314 * - ``ipv4_bfd``315 - ``control``316 - Traps IPv4 BFD packets317 * - ``ipv6_bfd``318 - ``control``319 - Traps IPv6 BFD packets320 * - ``ipv4_ospf``321 - ``control``322 - Traps IPv4 OSPF packets323 * - ``ipv6_ospf``324 - ``control``325 - Traps IPv6 OSPF packets326 * - ``ipv4_bgp``327 - ``control``328 - Traps IPv4 BGP packets329 * - ``ipv6_bgp``330 - ``control``331 - Traps IPv6 BGP packets332 * - ``ipv4_vrrp``333 - ``control``334 - Traps IPv4 VRRP packets335 * - ``ipv6_vrrp``336 - ``control``337 - Traps IPv6 VRRP packets338 * - ``ipv4_pim``339 - ``control``340 - Traps IPv4 PIM packets341 * - ``ipv6_pim``342 - ``control``343 - Traps IPv6 PIM packets344 * - ``uc_loopback``345 - ``control``346 - Traps unicast packets that need to be routed through the same layer 3347 interface from which they were received. Such packets are routed by the348 kernel, but also cause it to potentially generate ICMP redirect packets349 * - ``local_route``350 - ``control``351 - Traps unicast packets that hit a local route and need to be locally352 delivered353 * - ``external_route``354 - ``control``355 - Traps packets that should be routed through an external interface (e.g.,356 management interface) that does not belong to the same device (e.g.,357 switch ASIC) as the ingress interface358 * - ``ipv6_uc_dip_link_local_scope``359 - ``control``360 - Traps unicast IPv6 packets that need to be routed and have a destination361 IP address with a link-local scope (i.e., fe80::/10). The trap allows362 device drivers to avoid programming link-local routes, but still receive363 packets for local delivery364 * - ``ipv6_dip_all_nodes``365 - ``control``366 - Traps IPv6 packets that their destination IP address is the "All Nodes367 Address" (i.e., ff02::1)368 * - ``ipv6_dip_all_routers``369 - ``control``370 - Traps IPv6 packets that their destination IP address is the "All Routers371 Address" (i.e., ff02::2)372 * - ``ipv6_router_solicit``373 - ``control``374 - Traps IPv6 Router Solicitation packets375 * - ``ipv6_router_advert``376 - ``control``377 - Traps IPv6 Router Advertisement packets378 * - ``ipv6_redirect``379 - ``control``380 - Traps IPv6 Redirect Message packets381 * - ``ipv4_router_alert``382 - ``control``383 - Traps IPv4 packets that need to be routed and include the Router Alert384 option. Such packets need to be locally delivered to raw sockets that385 have the IP_ROUTER_ALERT socket option set386 * - ``ipv6_router_alert``387 - ``control``388 - Traps IPv6 packets that need to be routed and include the Router Alert389 option in their Hop-by-Hop extension header. Such packets need to be390 locally delivered to raw sockets that have the IPV6_ROUTER_ALERT socket391 option set392 * - ``ptp_event``393 - ``control``394 - Traps PTP time-critical event messages (Sync, Delay_req, Pdelay_Req and395 Pdelay_Resp)396 * - ``ptp_general``397 - ``control``398 - Traps PTP general messages (Announce, Follow_Up, Delay_Resp,399 Pdelay_Resp_Follow_Up, management and signaling)400 * - ``flow_action_sample``401 - ``control``402 - Traps packets sampled during processing of flow action sample (e.g., via403 tc's sample action)404 * - ``flow_action_trap``405 - ``control``406 - Traps packets logged during processing of flow action trap (e.g., via407 tc's trap action)408 * - ``early_drop``409 - ``drop``410 - Traps packets dropped due to the RED (Random Early Detection) algorithm411 (i.e., early drops)412 * - ``vxlan_parsing``413 - ``drop``414 - Traps packets dropped due to an error in the VXLAN header parsing which415 might be because of packet truncation or the I flag is not set.416 * - ``llc_snap_parsing``417 - ``drop``418 - Traps packets dropped due to an error in the LLC+SNAP header parsing419 * - ``vlan_parsing``420 - ``drop``421 - Traps packets dropped due to an error in the VLAN header parsing. Could422 include unexpected packet truncation.423 * - ``pppoe_ppp_parsing``424 - ``drop``425 - Traps packets dropped due to an error in the PPPoE+PPP header parsing.426 This could include finding a session ID of 0xFFFF (which is reserved and427 not for use), a PPPoE length which is larger than the frame received or428 any common error on this type of header429 * - ``mpls_parsing``430 - ``drop``431 - Traps packets dropped due to an error in the MPLS header parsing which432 could include unexpected header truncation433 * - ``arp_parsing``434 - ``drop``435 - Traps packets dropped due to an error in the ARP header parsing436 * - ``ip_1_parsing``437 - ``drop``438 - Traps packets dropped due to an error in the first IP header parsing.439 This packet trap could include packets which do not pass an IP checksum440 check, a header length check (a minimum of 20 bytes), which might suffer441 from packet truncation thus the total length field exceeds the received442 packet length etc443 * - ``ip_n_parsing``444 - ``drop``445 - Traps packets dropped due to an error in the parsing of the last IP446 header (the inner one in case of an IP over IP tunnel). The same common447 error checking is performed here as for the ip_1_parsing trap448 * - ``gre_parsing``449 - ``drop``450 - Traps packets dropped due to an error in the GRE header parsing451 * - ``udp_parsing``452 - ``drop``453 - Traps packets dropped due to an error in the UDP header parsing.454 This packet trap could include checksum errorrs, an improper UDP455 length detected (smaller than 8 bytes) or detection of header456 truncation.457 * - ``tcp_parsing``458 - ``drop``459 - Traps packets dropped due to an error in the TCP header parsing.460 This could include TCP checksum errors, improper combination of SYN, FIN461 and/or RESET etc.462 * - ``ipsec_parsing``463 - ``drop``464 - Traps packets dropped due to an error in the IPSEC header parsing465 * - ``sctp_parsing``466 - ``drop``467 - Traps packets dropped due to an error in the SCTP header parsing.468 This would mean that port number 0 was used or that the header is469 truncated.470 * - ``dccp_parsing``471 - ``drop``472 - Traps packets dropped due to an error in the DCCP header parsing473 * - ``gtp_parsing``474 - ``drop``475 - Traps packets dropped due to an error in the GTP header parsing476 * - ``esp_parsing``477 - ``drop``478 - Traps packets dropped due to an error in the ESP header parsing479 * - ``blackhole_nexthop``480 - ``drop``481 - Traps packets that the device decided to drop in case they hit a482 blackhole nexthop483 * - ``dmac_filter``484 - ``drop``485 - Traps incoming packets that the device decided to drop because486 the destination MAC is not configured in the MAC table and487 the interface is not in promiscuous mode488 * - ``eapol``489 - ``control``490 - Traps "Extensible Authentication Protocol over LAN" (EAPOL) packets491 specified in IEEE 802.1X492 * - ``locked_port``493 - ``drop``494 - Traps packets that the device decided to drop because they failed the495 locked bridge port check. That is, packets that were received via a496 locked port and whose {SMAC, VID} does not correspond to an FDB entry497 pointing to the port498 499Driver-specific Packet Traps500============================501 502Device drivers can register driver-specific packet traps, but these must be503clearly documented. Such traps can correspond to device-specific exceptions and504help debug packet drops caused by these exceptions. The following list includes505links to the description of driver-specific traps registered by various device506drivers:507 508 * Documentation/networking/devlink/netdevsim.rst509 * Documentation/networking/devlink/mlxsw.rst510 * Documentation/networking/devlink/prestera.rst511 512.. _Generic-Packet-Trap-Groups:513 514Generic Packet Trap Groups515==========================516 517Generic packet trap groups are used to aggregate logically related packet518traps. These groups allow the user to batch operations such as setting the trap519action of all member traps. In addition, ``devlink-trap`` can report aggregated520per-group packets and bytes statistics, in case per-trap statistics are too521narrow. The description of these groups must be added to the following table:522 523.. list-table:: List of Generic Packet Trap Groups524 :widths: 10 90525 526 * - Name527 - Description528 * - ``l2_drops``529 - Contains packet traps for packets that were dropped by the device during530 layer 2 forwarding (i.e., bridge)531 * - ``l3_drops``532 - Contains packet traps for packets that were dropped by the device during533 layer 3 forwarding534 * - ``l3_exceptions``535 - Contains packet traps for packets that hit an exception (e.g., TTL536 error) during layer 3 forwarding537 * - ``buffer_drops``538 - Contains packet traps for packets that were dropped by the device due to539 an enqueue decision540 * - ``tunnel_drops``541 - Contains packet traps for packets that were dropped by the device during542 tunnel encapsulation / decapsulation543 * - ``acl_drops``544 - Contains packet traps for packets that were dropped by the device during545 ACL processing546 * - ``stp``547 - Contains packet traps for STP packets548 * - ``lacp``549 - Contains packet traps for LACP packets550 * - ``lldp``551 - Contains packet traps for LLDP packets552 * - ``mc_snooping``553 - Contains packet traps for IGMP and MLD packets required for multicast554 snooping555 * - ``dhcp``556 - Contains packet traps for DHCP packets557 * - ``neigh_discovery``558 - Contains packet traps for neighbour discovery packets (e.g., ARP, IPv6559 ND)560 * - ``bfd``561 - Contains packet traps for BFD packets562 * - ``ospf``563 - Contains packet traps for OSPF packets564 * - ``bgp``565 - Contains packet traps for BGP packets566 * - ``vrrp``567 - Contains packet traps for VRRP packets568 * - ``pim``569 - Contains packet traps for PIM packets570 * - ``uc_loopback``571 - Contains a packet trap for unicast loopback packets (i.e.,572 ``uc_loopback``). This trap is singled-out because in cases such as573 one-armed router it will be constantly triggered. To limit the impact on574 the CPU usage, a packet trap policer with a low rate can be bound to the575 group without affecting other traps576 * - ``local_delivery``577 - Contains packet traps for packets that should be locally delivered after578 routing, but do not match more specific packet traps (e.g.,579 ``ipv4_bgp``)580 * - ``external_delivery``581 - Contains packet traps for packets that should be routed through an582 external interface (e.g., management interface) that does not belong to583 the same device (e.g., switch ASIC) as the ingress interface584 * - ``ipv6``585 - Contains packet traps for various IPv6 control packets (e.g., Router586 Advertisements)587 * - ``ptp_event``588 - Contains packet traps for PTP time-critical event messages (Sync,589 Delay_req, Pdelay_Req and Pdelay_Resp)590 * - ``ptp_general``591 - Contains packet traps for PTP general messages (Announce, Follow_Up,592 Delay_Resp, Pdelay_Resp_Follow_Up, management and signaling)593 * - ``acl_sample``594 - Contains packet traps for packets that were sampled by the device during595 ACL processing596 * - ``acl_trap``597 - Contains packet traps for packets that were trapped (logged) by the598 device during ACL processing599 * - ``parser_error_drops``600 - Contains packet traps for packets that were marked by the device during601 parsing as erroneous602 * - ``eapol``603 - Contains packet traps for "Extensible Authentication Protocol over LAN"604 (EAPOL) packets specified in IEEE 802.1X605 606Packet Trap Policers607====================608 609As previously explained, the underlying device can trap certain packets to the610CPU for processing. In most cases, the underlying device is capable of handling611packet rates that are several orders of magnitude higher compared to those that612can be handled by the CPU.613 614Therefore, in order to prevent the underlying device from overwhelming the CPU,615devices usually include packet trap policers that are able to police the616trapped packets to rates that can be handled by the CPU.617 618The ``devlink-trap`` mechanism allows capable device drivers to register their619supported packet trap policers with ``devlink``. The device driver can choose620to associate these policers with supported packet trap groups (see621:ref:`Generic-Packet-Trap-Groups`) during its initialization, thereby exposing622its default control plane policy to user space.623 624Device drivers should allow user space to change the parameters of the policers625(e.g., rate, burst size) as well as the association between the policers and626trap groups by implementing the relevant callbacks.627 628If possible, device drivers should implement a callback that allows user space629to retrieve the number of packets that were dropped by the policer because its630configured policy was violated.631 632Testing633=======634 635See ``tools/testing/selftests/drivers/net/netdevsim/devlink_trap.sh`` for a636test covering the core infrastructure. Test cases should be added for any new637functionality.638 639Device drivers should focus their tests on device-specific functionality, such640as the triggering of supported packet traps.641