1790 lines · plain
1============2SNMP counter3============4 5This document explains the meaning of SNMP counters.6 7General IPv4 counters8=====================9All layer 4 packets and ICMP packets will change these counters, but10these counters won't be changed by layer 2 packets (such as STP) or11ARP packets.12 13* IpInReceives14 15Defined in `RFC1213 ipInReceives`_16 17.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-2618 19The number of packets received by the IP layer. It gets increasing at the20beginning of ip_rcv function, always be updated together with21IpExtInOctets. It will be increased even if the packet is dropped22later (e.g. due to the IP header is invalid or the checksum is wrong23and so on). It indicates the number of aggregated segments after24GRO/LRO.25 26* IpInDelivers27 28Defined in `RFC1213 ipInDelivers`_29 30.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-2831 32The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,33ICMP and so on. If no one listens on a raw socket, only kernel34supported protocols will be delivered, if someone listens on the raw35socket, all valid IP packets will be delivered.36 37* IpOutRequests38 39Defined in `RFC1213 ipOutRequests`_40 41.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-2842 43The number of packets sent via IP layer, for both single cast and44multicast packets, and would always be updated together with45IpExtOutOctets.46 47* IpExtInOctets and IpExtOutOctets48 49They are Linux kernel extensions, no RFC definitions. Please note,50RFC1213 indeed defines ifInOctets and ifOutOctets, but they51are different things. The ifInOctets and ifOutOctets include the MAC52layer header size but IpExtInOctets and IpExtOutOctets don't, they53only include the IP layer header and the IP layer data.54 55* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts56 57They indicate the number of four kinds of ECN IP packets, please refer58`Explicit Congestion Notification`_ for more details.59 60.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-661 62These 4 counters calculate how many packets received per ECN63status. They count the real frame number regardless the LRO/GRO. So64for the same packet, you might find that IpInReceives count 1, but65IpExtInNoECTPkts counts 2 or more.66 67* IpInHdrErrors68 69Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is70dropped due to the IP header error. It might happen in both IP input71and IP forward paths.72 73.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-2774 75* IpInAddrErrors76 77Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two78scenarios: (1) The IP address is invalid. (2) The destination IP79address is not a local address and IP forwarding is not enabled80 81.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-2782 83* IpExtInNoRoutes84 85This counter means the packet is dropped when the IP stack receives a86packet and can't find a route for it from the route table. It might87happen when IP forwarding is enabled and the destination IP address is88not a local address and there is no route for the destination IP89address.90 91* IpInUnknownProtos92 93Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the94layer 4 protocol is unsupported by kernel. If an application is using95raw socket, kernel will always deliver the packet to the raw socket96and this counter won't be increased.97 98.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-2799 100* IpExtInTruncatedPkts101 102For IPv4 packet, it means the actual data size is smaller than the103"Total Length" field in the IPv4 header.104 105* IpInDiscards106 107Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped108in the IP receiving path and due to kernel internal reasons (e.g. no109enough memory).110 111.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28112 113* IpOutDiscards114 115Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is116dropped in the IP sending path and due to kernel internal reasons.117 118.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28119 120* IpOutNoRoutes121 122Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is123dropped in the IP sending path and no route is found for it.124 125.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29126 127ICMP counters128=============129* IcmpInMsgs and IcmpOutMsgs130 131Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_132 133.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41134.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43135 136As mentioned in the RFC1213, these two counters include errors, they137would be increased even if the ICMP packet has an invalid type. The138ICMP output path will check the header of a raw socket, so the139IcmpOutMsgs would still be updated if the IP header is constructed by140a userspace program.141 142* ICMP named types143 144| These counters include most of common ICMP types, they are:145| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_146| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_147| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_148| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_149| IcmpInRedirects: `RFC1213 icmpInRedirects`_150| IcmpInEchos: `RFC1213 icmpInEchos`_151| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_152| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_153| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_154| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_155| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_156| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_157| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_158| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_159| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_160| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_161| IcmpOutEchos: `RFC1213 icmpOutEchos`_162| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_163| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_164| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_165| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_166| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_167 168.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41169.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41170.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42171.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42172.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42173.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42174.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42175.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42176.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43177.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43178.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43179 180.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44181.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44182.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44183.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44184.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44185.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45186.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45187.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45188.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45189.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45190.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46191 192Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP193Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are194straightforward. The 'In' counter means kernel receives such a packet195and the 'Out' counter means kernel sends such a packet.196 197* ICMP numeric types198 199They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the200ICMP type number. These counters track all kinds of ICMP packets. The201ICMP type number definition could be found in the `ICMP parameters`_202document.203 204.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml205 206For example, if the Linux kernel sends an ICMP Echo packet, the207IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply208packet, IcmpMsgInType0 would increase 1.209 210* IcmpInCsumErrors211 212This counter indicates the checksum of the ICMP packet is213wrong. Kernel verifies the checksum after updating the IcmpInMsgs and214before updating IcmpMsgInType[N]. If a packet has bad checksum, the215IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.216 217* IcmpInErrors and IcmpOutErrors218 219Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_220 221.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41222.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43223 224When an error occurs in the ICMP packet handler path, these two225counters would be updated. The receiving packet path use IcmpInErrors226and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors227is increased, IcmpInErrors would always be increased too.228 229relationship of the ICMP counters230---------------------------------231The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they232are updated at the same time. The sum of IcmpMsgInType[N] plus233IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel234receives an ICMP packet, kernel follows below logic:235 2361. increase IcmpInMsgs2372. if has any error, update IcmpInErrors and finish the process2383. update IcmpMsgOutType[N]2394. handle the packet depending on the type, if has any error, update240 IcmpInErrors and finish the process241 242So if all errors occur in step (2), IcmpInMsgs should be equal to the243sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in244step (4), IcmpInMsgs should be equal to the sum of245IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),246IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus247IcmpInErrors.248 249General TCP counters250====================251* TcpInSegs252 253Defined in `RFC1213 tcpInSegs`_254 255.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48256 257The number of packets received by the TCP layer. As mentioned in258RFC1213, it includes the packets received in error, such as checksum259error, invalid TCP header and so on. Only one error won't be included:260if the layer 2 destination address is not the NIC's layer 2261address. It might happen if the packet is a multicast or broadcast262packet, or the NIC is in promiscuous mode. In these situations, the263packets would be delivered to the TCP layer, but the TCP layer will discard264these packets before increasing TcpInSegs. The TcpInSegs counter265isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs266counter would only increase 1.267 268* TcpOutSegs269 270Defined in `RFC1213 tcpOutSegs`_271 272.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48273 274The number of packets sent by the TCP layer. As mentioned in RFC1213,275it excludes the retransmitted packets. But it includes the SYN, ACK276and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of277GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will278increase 2.279 280* TcpActiveOpens281 282Defined in `RFC1213 tcpActiveOpens`_283 284.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47285 286It means the TCP layer sends a SYN, and come into the SYN-SENT287state. Every time TcpActiveOpens increases 1, TcpOutSegs should always288increase 1.289 290* TcpPassiveOpens291 292Defined in `RFC1213 tcpPassiveOpens`_293 294.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47295 296It means the TCP layer receives a SYN, replies a SYN+ACK, come into297the SYN-RCVD state.298 299* TcpExtTCPRcvCoalesce300 301When packets are received by the TCP layer and are not be read by the302application, the TCP layer will try to merge them. This counter303indicate how many packets are merged in such situation. If GRO is304enabled, lots of packets would be merged by GRO, these packets305wouldn't be counted to TcpExtTCPRcvCoalesce.306 307* TcpExtTCPAutoCorking308 309When sending packets, the TCP layer will try to merge small packets to310a bigger one. This counter increase 1 for every packet merged in such311situation. Please refer to the LWN article for more details:312https://lwn.net/Articles/576263/313 314* TcpExtTCPOrigDataSent315 316This counter is explained by kernel commit f19c29e3e391, I pasted the317explanation below::318 319 TCPOrigDataSent: number of outgoing packets with original data (excluding320 retransmission but including data-in-SYN). This counter is different from321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is322 more useful to track the TCP retransmission rate.323 324* TCPSynRetrans325 326This counter is explained by kernel commit f19c29e3e391, I pasted the327explanation below::328 329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc.331 332* TCPFastOpenActiveFail333 334This counter is explained by kernel commit f19c29e3e391, I pasted the335explanation below::336 337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because338 the remote does not accept it or the attempts timed out.339 340* TcpExtListenOverflows and TcpExtListenDrops341 342When kernel receives a SYN from a client, and if the TCP accept queue343is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.344At the same time kernel will also add 1 to TcpExtListenDrops. When a345TCP socket is in LISTEN state, and kernel need to drop a packet,346kernel would always add 1 to TcpExtListenDrops. So increase347TcpExtListenOverflows would let TcpExtListenDrops increasing at the348same time, but TcpExtListenDrops would also increase without349TcpExtListenOverflows increasing, e.g. a memory allocation fail would350also let TcpExtListenDrops increase.351 352Note: The above explanation is based on kernel 4.10 or above version, on353an old kernel, the TCP stack has different behavior when TCP accept354queue is full. On the old kernel, TCP stack won't drop the SYN, it355would complete the 3-way handshake. As the accept queue is full, TCP356stack will keep the socket in the TCP half-open queue. As it is in the357half open queue, TCP stack will send SYN+ACK on an exponential backoff358timer, after client replies ACK, TCP stack checks whether the accept359queue is still full, if it is not full, moves the socket to the accept360queue, if it is full, keeps the socket in the half-open queue, at next361time client replies ACK, this socket will get another chance to move362to the accept queue.363 364 365TCP Fast Open366=============367* TcpEstabResets368 369Defined in `RFC1213 tcpEstabResets`_.370 371.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48372 373* TcpAttemptFails374 375Defined in `RFC1213 tcpAttemptFails`_.376 377.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48378 379* TcpOutRsts380 381Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates382the 'segments sent containing the RST flag', but in linux kernel, this383counter indicates the segments kernel tried to send. The sending384process might be failed due to some errors (e.g. memory alloc failed).385 386.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52387 388* TcpExtTCPSpuriousRtxHostQueues389 390When the TCP stack wants to retransmit a packet, and finds that packet391is not lost in the network, but the packet is not sent yet, the TCP392stack would give up the retransmission and update this counter. It393might happen if a packet stays too long time in a qdisc or driver394queue.395 396* TcpEstabResets397 398The socket receives a RST packet in Establish or CloseWait state.399 400* TcpExtTCPKeepAlive401 402This counter indicates many keepalive packets were sent. The keepalive403won't be enabled by default. A userspace program could enable it by404setting the SO_KEEPALIVE socket option.405 406* TcpExtTCPSpuriousRTOs407 408The spurious retransmission timeout detected by the `F-RTO`_409algorithm.410 411.. _F-RTO: https://tools.ietf.org/html/rfc5682412 413TCP Fast Path414=============415When kernel receives a TCP packet, it has two paths to handler the416packet, one is fast path, another is slow path. The comment in kernel417code provides a good explanation of them, I pasted them below::418 419 It is split into a fast path and a slow path. The fast path is420 disabled when:421 422 - A zero window was announced from us423 - zero window probing424 is only handled properly on the slow path.425 - Out of order segments arrived.426 - Urgent data is expected.427 - There is no buffer space left428 - Unexpected TCP flags/window values/header lengths are received429 (detected by checking the TCP header against pred_flags)430 - Data is sent in both directions. The fast path only supports pure senders431 or pure receivers (this means either the sequence number or the ack432 value must stay constant)433 - Unexpected TCP option.434 435Kernel will try to use fast path unless any of the above conditions436are satisfied. If the packets are out of order, kernel will handle437them in slow path, which means the performance might be not very438good. Kernel would also come into slow path if the "Delayed ack" is439used, because when using "Delayed ack", the data is sent in both440directions. When the TCP window scale option is not used, kernel will441try to enable fast path immediately when the connection comes into the442established state, but if the TCP window scale option is used, kernel443will disable the fast path at first, and try to enable it after kernel444receives packets.445 446* TcpExtTCPPureAcks and TcpExtTCPHPAcks447 448If a packet set ACK flag and has no data, it is a pure ACK packet, if449kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,450if kernel handles it in the slow path, TcpExtTCPPureAcks will451increase 1.452 453* TcpExtTCPHPHits454 455If a TCP packet has data (which means it is not a pure ACK packet),456and this packet is handled in the fast path, TcpExtTCPHPHits will457increase 1.458 459 460TCP abort461=========462* TcpExtTCPAbortOnData463 464It means TCP layer has data in flight, but need to close the465connection. So TCP layer sends a RST to the other side, indicate the466connection is not closed very graceful. An easy way to increase this467counter is using the SO_LINGER option. Please refer to the SO_LINGER468section of the `socket man page`_:469 470.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html471 472By default, when an application closes a connection, the close function473will return immediately and kernel will try to send the in-flight data474async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger475to a positive number, the close function won't return immediately, but476wait for the in-flight data are acked by the other side, the max wait477time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,478when the application closes a connection, kernel will send a RST479immediately and increase the TcpExtTCPAbortOnData counter.480 481* TcpExtTCPAbortOnClose482 483This counter means the application has unread data in the TCP layer when484the application wants to close the TCP connection. In such a situation,485kernel will send a RST to the other side of the TCP connection.486 487* TcpExtTCPAbortOnMemory488 489When an application closes a TCP connection, kernel still need to track490the connection, let it complete the TCP disconnect process. E.g. an491app calls the close method of a socket, kernel sends fin to the other492side of the connection, then the app has no relationship with the493socket any more, but kernel need to keep the socket, this socket494becomes an orphan socket, kernel waits for the reply of the other side,495and would come to the TIME_WAIT state finally. When kernel has no496enough memory to keep the orphan socket, kernel would send an RST to497the other side, and delete the socket, in such situation, kernel will498increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger499TcpExtTCPAbortOnMemory:500 5011. the memory used by the TCP protocol is higher than the third value of502the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:503 504.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html505 5062. the orphan socket count is higher than net.ipv4.tcp_max_orphans507 508 509* TcpExtTCPAbortOnTimeout510 511This counter will increase when any of the TCP timers expire. In such512situation, kernel won't send RST, just give up the connection.513 514* TcpExtTCPAbortOnLinger515 516When a TCP connection comes into FIN_WAIT_2 state, instead of waiting517for the fin packet from the other side, kernel could send a RST and518delete the socket immediately. This is not the default behavior of519Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,520you could let kernel follow this behavior.521 522* TcpExtTCPAbortFailed523 524The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is525satisfied. If an internal error occurs during this process,526TcpExtTCPAbortFailed will be increased.527 528.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50529 530TCP Hybrid Slow Start531=====================532The Hybrid Slow Start algorithm is an enhancement of the traditional533TCP congestion window Slow Start algorithm. It uses two pieces of534information to detect whether the max bandwidth of the TCP path is535approached. The two pieces of information are ACK train length and536increase in packet delay. For detail information, please refer the537`Hybrid Slow Start paper`_. Either ACK train length or packet delay538hits a specific threshold, the congestion control algorithm will come539into the Congestion Avoidance state. Until v4.20, two congestion540control algorithms are using Hybrid Slow Start, they are cubic (the541default congestion control algorithm) and cdg. Four snmp counters542relate with the Hybrid Slow Start algorithm.543 544.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf545 546* TcpExtTCPHystartTrainDetect547 548How many times the ACK train length threshold is detected549 550* TcpExtTCPHystartTrainCwnd551 552The sum of CWND detected by ACK train length. Dividing this value by553TcpExtTCPHystartTrainDetect is the average CWND which detected by the554ACK train length.555 556* TcpExtTCPHystartDelayDetect557 558How many times the packet delay threshold is detected.559 560* TcpExtTCPHystartDelayCwnd561 562The sum of CWND detected by packet delay. Dividing this value by563TcpExtTCPHystartDelayDetect is the average CWND which detected by the564packet delay.565 566TCP retransmission and congestion control567=========================================568The TCP protocol has two retransmission mechanisms: SACK and fast569recovery. They are exclusive with each other. When SACK is enabled,570the kernel TCP stack would use SACK, or kernel would use fast571recovery. The SACK is a TCP option, which is defined in `RFC2018`_,572the fast recovery is defined in `RFC6582`_, which is also called573'Reno'.574 575The TCP congestion control is a big and complex topic. To understand576the related snmp counter, we need to know the states of the congestion577control state machine. There are 5 states: Open, Disorder, CWR,578Recovery and Loss. For details about these states, please refer page 5579and page 6 of this document:580https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf581 582.. _RFC2018: https://tools.ietf.org/html/rfc2018583.. _RFC6582: https://tools.ietf.org/html/rfc6582584 585* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery586 587When the congestion control comes into Recovery state, if sack is588used, TcpExtTCPSackRecovery increases 1, if sack is not used,589TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP590stack begins to retransmit the lost packets.591 592* TcpExtTCPSACKReneging593 594A packet was acknowledged by SACK, but the receiver has dropped this595packet, so the sender needs to retransmit this packet. In this596situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver597could drop a packet which has been acknowledged by SACK, although it is598unusual, it is allowed by the TCP protocol. The sender doesn't really599know what happened on the receiver side. The sender just waits until600the RTO expires for this packet, then the sender assumes this packet601has been dropped by the receiver.602 603* TcpExtTCPRenoReorder604 605The reorder packet is detected by fast recovery. It would only be used606if SACK is disabled. The fast recovery algorithm detects recorder by607the duplicate ACK number. E.g., if retransmission is triggered, and608the original retransmitted packet is not lost, it is just out of609order, the receiver would acknowledge multiple times, one for the610retransmitted packet, another for the arriving of the original out of611order packet. Thus the sender would find more ACks than its612expectation, and the sender knows out of order occurs.613 614* TcpExtTCPTSReorder615 616The reorder packet is detected when a hole is filled. E.g., assume the617sender sends packet 1,2,3,4,5, and the receiving order is6181,2,4,5,3. When the sender receives the ACK of packet 3 (which will619fill the hole), two conditions will let TcpExtTCPTSReorder increase6201: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet6213 is retransmitted but the timestamp of the packet 3's ACK is earlier622than the retransmission timestamp.623 624* TcpExtTCPSACKReorder625 626The reorder packet detected by SACK. The SACK has two methods to627detect reorder: (1) DSACK is received by the sender. It means the628sender sends the same packet more than one times. And the only reason629is the sender believes an out of order packet is lost so it sends the630packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and631the sender has received SACKs for packet 2 and 5, now the sender632receives SACK for packet 4 and the sender doesn't retransmit the633packet yet, the sender would know packet 4 is out of order. The TCP634stack of kernel will increase TcpExtTCPSACKReorder for both of the635above scenarios.636 637* TcpExtTCPSlowStartRetrans638 639The TCP stack wants to retransmit a packet and the congestion control640state is 'Loss'.641 642* TcpExtTCPFastRetrans643 644The TCP stack wants to retransmit a packet and the congestion control645state is not 'Loss'.646 647* TcpExtTCPLostRetransmit648 649A SACK points out that a retransmission packet is lost again.650 651* TcpExtTCPRetransFail652 653The TCP stack tries to deliver a retransmission packet to lower layers654but the lower layers return an error.655 656* TcpExtTCPSynRetrans657 658The TCP stack retransmits a SYN packet.659 660DSACK661=====662The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report663duplicate packets to the sender. There are two kinds of664duplications: (1) a packet which has been acknowledged is665duplicate. (2) an out of order packet is duplicate. The TCP stack666counts these two kinds of duplications on both receiver side and667sender side.668 669.. _RFC2883 : https://tools.ietf.org/html/rfc2883670 671* TcpExtTCPDSACKOldSent672 673The TCP stack receives a duplicate packet which has been acked, so it674sends a DSACK to the sender.675 676* TcpExtTCPDSACKOfoSent677 678The TCP stack receives an out of order duplicate packet, so it sends a679DSACK to the sender.680 681* TcpExtTCPDSACKRecv682 683The TCP stack receives a DSACK, which indicates an acknowledged684duplicate packet is received.685 686* TcpExtTCPDSACKOfoRecv687 688The TCP stack receives a DSACK, which indicate an out of order689duplicate packet is received.690 691invalid SACK and DSACK692======================693When a SACK (or DSACK) block is invalid, a corresponding counter would694be updated. The validation method is base on the start/end sequence695number of the SACK block. For more details, please refer the comment696of the function tcp_is_sackblock_valid in the kernel source code. A697SACK option could have up to 4 blocks, they are checked698individually. E.g., if 3 blocks of a SACk is invalid, the699corresponding counter would be updated 3 times. The comment of commit70018f02545a9a1 ("[TCP] MIB: Add counters for discarded SACK blocks")701has additional explanation:702 703* TcpExtTCPSACKDiscard704 705This counter indicates how many SACK blocks are invalid. If the invalid706SACK block is caused by ACK recording, the TCP stack will only ignore707it and won't update this counter.708 709* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo710 711When a DSACK block is invalid, one of these two counters would be712updated. Which counter will be updated depends on the undo_marker flag713of the TCP socket. If the undo_marker is not set, the TCP stack isn't714likely to re-transmit any packets, and we still receive an invalid715DSACK block, the reason might be that the packet is duplicated in the716middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo717will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld718will be updated. As implied in its name, it might be an old packet.719 720SACK shift721==========722The linux networking stack stores data in sk_buff struct (skb for723short). If a SACK block acrosses multiple skb, the TCP stack will try724to re-arrange data in these skb. E.g. if a SACK block acknowledges seq72510 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and72615 in skb2 would be moved to skb1. This operation is 'shift'. If a727SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has728seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be729discard, this operation is 'merge'.730 731* TcpExtTCPSackShifted732 733A skb is shifted734 735* TcpExtTCPSackMerged736 737A skb is merged738 739* TcpExtTCPSackShiftFallback740 741A skb should be shifted or merged, but the TCP stack doesn't do it for742some reasons.743 744TCP out of order745================746* TcpExtTCPOFOQueue747 748The TCP layer receives an out of order packet and has enough memory749to queue it.750 751* TcpExtTCPOFODrop752 753The TCP layer receives an out of order packet but doesn't have enough754memory, so drops it. Such packets won't be counted into755TcpExtTCPOFOQueue.756 757* TcpExtTCPOFOMerge758 759The received out of order packet has an overlay with the previous760packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge761packets will also be counted into TcpExtTCPOFOQueue.762 763TCP PAWS764========765PAWS (Protection Against Wrapped Sequence numbers) is an algorithm766which is used to drop old packets. It depends on the TCP767timestamps. For detail information, please refer the `timestamp wiki`_768and the `RFC of PAWS`_.769 770.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17771.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps772 773* TcpExtPAWSActive774 775Packets are dropped by PAWS in Syn-Sent status.776 777* TcpExtPAWSEstab778 779Packets are dropped by PAWS in any status other than Syn-Sent.780 781TCP ACK skip782============783In some scenarios, kernel would avoid sending duplicate ACKs too784frequently. Please find more details in the tcp_invalid_ratelimit785section of the `sysctl document`_. When kernel decides to skip an ACK786due to tcp_invalid_ratelimit, kernel would update one of below787counters to indicate the ACK is skipped in which scenario. The ACK788would only be skipped if the received packet is either a SYN packet or789it has no data.790 791.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst792 793* TcpExtTCPACKSkippedSynRecv794 795The ACK is skipped in Syn-Recv status. The Syn-Recv status means the796TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is797waiting for an ACK. Generally, the TCP stack doesn't need to send ACK798in the Syn-Recv status. But in several scenarios, the TCP stack need799to send an ACK. E.g., the TCP stack receives the same SYN packet800repeately, the received packet does not pass the PAWS check, or the801received packet sequence number is out of window. In these scenarios,802the TCP stack needs to send ACK. If the ACk sending frequency is higher than803tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and804increase TcpExtTCPACKSkippedSynRecv.805 806 807* TcpExtTCPACKSkippedPAWS808 809The ACK is skipped due to PAWS (Protect Against Wrapped Sequence810numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2811or Time-Wait statuses, the skipped ACK would be counted to812TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or813TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK814would be counted to TcpExtTCPACKSkippedPAWS.815 816* TcpExtTCPACKSkippedSeq817 818The sequence number is out of window and the timestamp passes the PAWS819check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.820 821* TcpExtTCPACKSkippedFinWait2822 823The ACK is skipped in Fin-Wait-2 status, the reason would be either824PAWS check fails or the received sequence number is out of window.825 826* TcpExtTCPACKSkippedTimeWait827 828The ACK is skipped in Time-Wait status, the reason would be either829PAWS check failed or the received sequence number is out of window.830 831* TcpExtTCPACKSkippedChallenge832 833The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines8343 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,835`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these836three scenarios, In some TCP status, the linux TCP stack would also837send challenge ACKs if the ACK number is before the first838unacknowledged number (more strict than `RFC 5961 section 5.2`_).839 840.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7841.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9842.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11843 844TCP receive window845==================846* TcpExtTCPWantZeroWindowAdv847 848Depending on current memory usage, the TCP stack tries to set receive849window to zero. But the receive window might still be a no-zero850value. For example, if the previous window size is 10, and the TCP851stack receives 3 bytes, the current window size would be 7 even if the852window size calculated by the memory usage is zero.853 854* TcpExtTCPToZeroWindowAdv855 856The TCP receive window is set to zero from a no-zero value.857 858* TcpExtTCPFromZeroWindowAdv859 860The TCP receive window is set to no-zero value from zero.861 862 863Delayed ACK864===========865The TCP Delayed ACK is a technique which is used for reducing the866packet count in the network. For more details, please refer the867`Delayed ACK wiki`_868 869.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment870 871* TcpExtDelayedACKs872 873A delayed ACK timer expires. The TCP stack will send a pure ACK packet874and exit the delayed ACK mode.875 876* TcpExtDelayedACKLocked877 878A delayed ACK timer expires, but the TCP stack can't send an ACK879immediately due to the socket is locked by a userspace program. The880TCP stack will send a pure ACK later (after the userspace program881unlock the socket). When the TCP stack sends the pure ACK later, the882TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK883mode.884 885* TcpExtDelayedACKLost886 887It will be updated when the TCP stack receives a packet which has been888ACKed. A Delayed ACK loss might cause this issue, but it would also be889triggered by other reasons, such as a packet is duplicated in the890network.891 892Tail Loss Probe (TLP)893=====================894TLP is an algorithm which is used to detect TCP packet loss. For more895details, please refer the `TLP paper`_.896 897.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01898 899* TcpExtTCPLossProbes900 901A TLP probe packet is sent.902 903* TcpExtTCPLossProbeRecovery904 905A packet loss is detected and recovered by TLP.906 907TCP Fast Open description908=========================909TCP Fast Open is a technology which allows data transfer before the9103-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a911general description.912 913.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open914 915* TcpExtTCPFastOpenActive916 917When the TCP stack receives an ACK packet in the SYN-SENT status, and918the ACK packet acknowledges the data in the SYN packet, the TCP stack919understand the TFO cookie is accepted by the other side, then it920updates this counter.921 922* TcpExtTCPFastOpenActiveFail923 924This counter indicates that the TCP stack initiated a TCP Fast Open,925but it failed. This counter would be updated in three scenarios: (1)926the other side doesn't acknowledge the data in the SYN packet. (2) The927SYN packet which has the TFO cookie is timeout at least once. (3)928after the 3-way handshake, the retransmission timeout happens929net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole930fast open after the handshake.931 932* TcpExtTCPFastOpenPassive933 934This counter indicates how many times the TCP stack accepts the fast935open request.936 937* TcpExtTCPFastOpenPassiveFail938 939This counter indicates how many times the TCP stack rejects the fast940open request. It is caused by either the TFO cookie is invalid or the941TCP stack finds an error during the socket creating process.942 943* TcpExtTCPFastOpenListenOverflow944 945When the pending fast open request number is larger than946fastopenq->max_qlen, the TCP stack will reject the fast open request947and update this counter. When this counter is updated, the TCP stack948won't update TcpExtTCPFastOpenPassive or949TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the950TCP_FASTOPEN socket operation and it could not be larger than951net.core.somaxconn. For example:952 953setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));954 955* TcpExtTCPFastOpenCookieReqd956 957This counter indicates how many times a client wants to request a TFO958cookie.959 960SYN cookies961===========962SYN cookies are used to mitigate SYN flood, for details, please refer963the `SYN cookies wiki`_.964 965.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies966 967* TcpExtSyncookiesSent968 969It indicates how many SYN cookies are sent.970 971* TcpExtSyncookiesRecv972 973How many reply packets of the SYN cookies the TCP stack receives.974 975* TcpExtSyncookiesFailed976 977The MSS decoded from the SYN cookie is invalid. When this counter is978updated, the received packet won't be treated as a SYN cookie and the979TcpExtSyncookiesRecv counter won't be updated.980 981Challenge ACK982=============983For details of challenge ACK, please refer the explanation of984TcpExtTCPACKSkippedChallenge.985 986* TcpExtTCPChallengeACK987 988The number of challenge acks sent.989 990* TcpExtTCPSYNChallenge991 992The number of challenge acks sent in response to SYN packets. After993updates this counter, the TCP stack might send a challenge ACK and994update the TcpExtTCPChallengeACK counter, or it might also skip to995send the challenge and update the TcpExtTCPACKSkippedChallenge.996 997prune998=====999When a socket is under memory pressure, the TCP stack will try to1000reclaim memory from the receiving queue and out of order queue. One of1001the reclaiming method is 'collapse', which means allocate a big skb,1002copy the contiguous skbs to the single big skb, and free these1003contiguous skbs.1004 1005* TcpExtPruneCalled1006 1007The TCP stack tries to reclaim memory for a socket. After updates this1008counter, the TCP stack will try to collapse the out of order queue and1009the receiving queue. If the memory is still not enough, the TCP stack1010will try to discard packets from the out of order queue (and update the1011TcpExtOfoPruned counter)1012 1013* TcpExtOfoPruned1014 1015The TCP stack tries to discard packet on the out of order queue.1016 1017* TcpExtRcvPruned1018 1019After 'collapse' and discard packets from the out of order queue, if1020the actually used memory is still larger than the max allowed memory,1021this counter will be updated. It means the 'prune' fails.1022 1023* TcpExtTCPRcvCollapsed1024 1025This counter indicates how many skbs are freed during 'collapse'.1026 1027examples1028========1029 1030ping test1031---------1032Run the ping command against the public dns server 8.8.8.8::1033 1034 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 11035 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.1036 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms1037 1038 --- 8.8.8.8 ping statistics ---1039 1 packets transmitted, 1 received, 0% packet loss, time 0ms1040 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms1041 1042The nstayt result::1043 1044 nstatuser@nstat-a:~$ nstat1045 #kernel1046 IpInReceives 1 0.01047 IpInDelivers 1 0.01048 IpOutRequests 1 0.01049 IcmpInMsgs 1 0.01050 IcmpInEchoReps 1 0.01051 IcmpOutMsgs 1 0.01052 IcmpOutEchos 1 0.01053 IcmpMsgInType0 1 0.01054 IcmpMsgOutType8 1 0.01055 IpExtInOctets 84 0.01056 IpExtOutOctets 84 0.01057 IpExtInNoECTPkts 1 0.01058 1059The Linux server sent an ICMP Echo packet, so IpOutRequests,1060IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The1061server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,1062IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply1063was passed to the ICMP layer via IP layer, so IpInDelivers was1064increased 1. The default ping data size is 48, so an ICMP Echo packet1065and its corresponding Echo Reply packet are constructed by:1066 1067* 14 bytes MAC header1068* 20 bytes IP header1069* 16 bytes ICMP header1070* 48 bytes data (default value of the ping command)1071 1072So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.1073 1074tcp 3-way handshake1075-------------------1076On server side, we run::1077 1078 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 90001079 Listening on [0.0.0.0] (family 0, port 9000)1080 1081On client side, we run::1082 1083 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 90001084 Connection to 192.168.122.251 9000 port [tcp/*] succeeded!1085 1086The server listened on tcp 9000 port, the client connected to it, they1087completed the 3-way handshake.1088 1089On server side, we can find below nstat output::1090 1091 nstatuser@nstat-b:~$ nstat | grep -i tcp1092 TcpPassiveOpens 1 0.01093 TcpInSegs 2 0.01094 TcpOutSegs 1 0.01095 TcpExtTCPPureAcks 1 0.01096 1097On client side, we can find below nstat output::1098 1099 nstatuser@nstat-a:~$ nstat | grep -i tcp1100 TcpActiveOpens 1 0.01101 TcpInSegs 1 0.01102 TcpOutSegs 2 0.01103 1104When the server received the first SYN, it replied a SYN+ACK, and came into1105SYN-RCVD state, so TcpPassiveOpens increased 1. The server received1106SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 21107packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK1108of the 3-way handshake is a pure ACK without data, so1109TcpExtTCPPureAcks increased 1.1110 1111When the client sent SYN, the client came into the SYN-SENT state, so1112TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent1113ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased11141, TcpOutSegs increased 2.1115 1116TCP normal traffic1117------------------1118Run nc on server::1119 1120 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 90001121 Listening on [0.0.0.0] (family 0, port 9000)1122 1123Run nc on client::1124 1125 nstatuser@nstat-a:~$ nc -v nstat-b 90001126 Connection to nstat-b 9000 port [tcp/*] succeeded!1127 1128Input a string in the nc client ('hello' in our example)::1129 1130 nstatuser@nstat-a:~$ nc -v nstat-b 90001131 Connection to nstat-b 9000 port [tcp/*] succeeded!1132 hello1133 1134The client side nstat output::1135 1136 nstatuser@nstat-a:~$ nstat1137 #kernel1138 IpInReceives 1 0.01139 IpInDelivers 1 0.01140 IpOutRequests 1 0.01141 TcpInSegs 1 0.01142 TcpOutSegs 1 0.01143 TcpExtTCPPureAcks 1 0.01144 TcpExtTCPOrigDataSent 1 0.01145 IpExtInOctets 52 0.01146 IpExtOutOctets 58 0.01147 IpExtInNoECTPkts 1 0.01148 1149The server side nstat output::1150 1151 nstatuser@nstat-b:~$ nstat1152 #kernel1153 IpInReceives 1 0.01154 IpInDelivers 1 0.01155 IpOutRequests 1 0.01156 TcpInSegs 1 0.01157 TcpOutSegs 1 0.01158 IpExtInOctets 58 0.01159 IpExtOutOctets 52 0.01160 IpExtInNoECTPkts 1 0.01161 1162Input a string in nc client side again ('world' in our example)::1163 1164 nstatuser@nstat-a:~$ nc -v nstat-b 90001165 Connection to nstat-b 9000 port [tcp/*] succeeded!1166 hello1167 world1168 1169Client side nstat output::1170 1171 nstatuser@nstat-a:~$ nstat1172 #kernel1173 IpInReceives 1 0.01174 IpInDelivers 1 0.01175 IpOutRequests 1 0.01176 TcpInSegs 1 0.01177 TcpOutSegs 1 0.01178 TcpExtTCPHPAcks 1 0.01179 TcpExtTCPOrigDataSent 1 0.01180 IpExtInOctets 52 0.01181 IpExtOutOctets 58 0.01182 IpExtInNoECTPkts 1 0.01183 1184 1185Server side nstat output::1186 1187 nstatuser@nstat-b:~$ nstat1188 #kernel1189 IpInReceives 1 0.01190 IpInDelivers 1 0.01191 IpOutRequests 1 0.01192 TcpInSegs 1 0.01193 TcpOutSegs 1 0.01194 TcpExtTCPHPHits 1 0.01195 IpExtInOctets 58 0.01196 IpExtOutOctets 52 0.01197 IpExtInNoECTPkts 1 0.01198 1199Compare the first client-side nstat and the second client-side nstat,1200we could find one difference: the first one had a 'TcpExtTCPPureAcks',1201but the second one had a 'TcpExtTCPHPAcks'. The first server-side1202nstat and the second server-side nstat had a difference too: the1203second server-side nstat had a TcpExtTCPHPHits, but the first1204server-side nstat didn't have it. The network traffic patterns were1205exactly the same: the client sent a packet to the server, the server1206replied an ACK. But kernel handled them in different ways. When the1207TCP window scale option is not used, kernel will try to enable fast1208path immediately when the connection comes into the established state,1209but if the TCP window scale option is used, kernel will disable the1210fast path at first, and try to enable it after kernel receives1211packets. We could use the 'ss' command to verify whether the window1212scale option is used. e.g. run below command on either server or1213client::1214 1215 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )1216 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port1217 tcp 0 0 192.168.122.250:40654 192.168.122.251:90001218 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.981219 1220The 'wscale:7,7' means both server and client set the window scale1221option to 7. Now we could explain the nstat output in our test:1222 1223In the first nstat output of client side, the client sent a packet, server1224reply an ACK, when kernel handled this ACK, the fast path was not1225enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.1226 1227In the second nstat output of client side, the client sent a packet again,1228and received another ACK from the server, in this time, the fast path is1229enabled, and the ACK was qualified for fast path, so it was handled by1230the fast path, so this ACK was counted into TcpExtTCPHPAcks.1231 1232In the first nstat output of server side, fast path was not enabled,1233so there was no 'TcpExtTCPHPHits'.1234 1235In the second nstat output of server side, the fast path was enabled,1236and the packet received from client qualified for fast path, so it1237was counted into 'TcpExtTCPHPHits'.1238 1239TcpExtTCPAbortOnClose1240---------------------1241On the server side, we run below python script::1242 1243 import socket1244 import time1245 1246 port = 90001247 1248 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1249 s.bind(('0.0.0.0', port))1250 s.listen(1)1251 sock, addr = s.accept()1252 while True:1253 time.sleep(9999999)1254 1255This python script listen on 9000 port, but doesn't read anything from1256the connection.1257 1258On the client side, we send the string "hello" by nc::1259 1260 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 90001261 1262Then, we come back to the server side, the server has received the "hello"1263packet, and the TCP layer has acked this packet, but the application didn't1264read it yet. We type Ctrl-C to terminate the server script. Then we1265could find TcpExtTCPAbortOnClose increased 1 on the server side::1266 1267 nstatuser@nstat-b:~$ nstat | grep -i abort1268 TcpExtTCPAbortOnClose 1 0.01269 1270If we run tcpdump on the server side, we could find the server sent a1271RST after we type Ctrl-C.1272 1273TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout1274---------------------------------------------------1275Below is an example which let the orphan socket count be higher than1276net.ipv4.tcp_max_orphans.1277Change tcp_max_orphans to a smaller value on client::1278 1279 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"1280 1281Client code (create 64 connection to server)::1282 1283 nstatuser@nstat-a:~$ cat client_orphan.py1284 import socket1285 import time1286 1287 server = 'nstat-b' # server address1288 port = 90001289 1290 count = 641291 1292 connection_list = []1293 1294 for i in range(64):1295 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1296 s.connect((server, port))1297 connection_list.append(s)1298 print("connection_count: %d" % len(connection_list))1299 1300 while True:1301 time.sleep(99999)1302 1303Server code (accept 64 connection from client)::1304 1305 nstatuser@nstat-b:~$ cat server_orphan.py1306 import socket1307 import time1308 1309 port = 90001310 count = 641311 1312 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1313 s.bind(('0.0.0.0', port))1314 s.listen(count)1315 connection_list = []1316 while True:1317 sock, addr = s.accept()1318 connection_list.append((sock, addr))1319 print("connection_count: %d" % len(connection_list))1320 1321Run the python scripts on server and client.1322 1323On server::1324 1325 python3 server_orphan.py1326 1327On client::1328 1329 python3 client_orphan.py1330 1331Run iptables on server::1332 1333 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP1334 1335Type Ctrl-C on client, stop client_orphan.py.1336 1337Check TcpExtTCPAbortOnMemory on client::1338 1339 nstatuser@nstat-a:~$ nstat | grep -i abort1340 TcpExtTCPAbortOnMemory 54 0.01341 1342Check orphaned socket count on client::1343 1344 nstatuser@nstat-a:~$ ss -s1345 Total: 131 (kernel 0)1346 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 01347 1348 Transport Total IP IPv61349 * 0 - -1350 RAW 1 0 11351 UDP 1 1 01352 TCP 14 13 11353 INET 16 14 21354 FRAG 0 0 01355 1356The explanation of the test: after run server_orphan.py and1357client_orphan.py, we set up 64 connections between server and1358client. Run the iptables command, the server will drop all packets from1359the client, type Ctrl-C on client_orphan.py, the system of the client1360would try to close these connections, and before they are closed1361gracefully, these connections became orphan sockets. As the iptables1362of the server blocked packets from the client, the server won't receive fin1363from the client, so all connection on clients would be stuck on FIN_WAIT_11364stage, so they will keep as orphan sockets until timeout. We have echo136510 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would1366only keep 10 orphan sockets, for all other orphan sockets, the client1367system sent RST for them and delete them. We have 64 connections, so1368the 'ss -s' command shows the system has 10 orphan sockets, and the1369value of TcpExtTCPAbortOnMemory was 54.1370 1371An additional explanation about orphan socket count: You could find the1372exactly orphan socket count by the 'ss -s' command, but when kernel1373decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel1374doesn't always check the exactly orphan socket count. For increasing1375performance, kernel checks an approximate count firstly, if the1376approximate count is more than tcp_max_orphans, kernel checks the1377exact count again. So if the approximate count is less than1378tcp_max_orphans, but exactly count is more than tcp_max_orphans, you1379would find TcpExtTCPAbortOnMemory is not increased at all. If1380tcp_max_orphans is large enough, it won't occur, but if you decrease1381tcp_max_orphans to a small value like our test, you might find this1382issue. So in our test, the client set up 64 connections although the1383tcp_max_orphans is 10. If the client only set up 11 connections, we1384can't find the change of TcpExtTCPAbortOnMemory.1385 1386Continue the previous test, we wait for several minutes. Because of the1387iptables on the server blocked the traffic, the server wouldn't receive1388fin, and all the client's orphan sockets would timeout on the1389FIN_WAIT_1 state finally. So we wait for a few minutes, we could find139010 timeout on the client::1391 1392 nstatuser@nstat-a:~$ nstat | grep -i abort1393 TcpExtTCPAbortOnTimeout 10 0.01394 1395TcpExtTCPAbortOnLinger1396----------------------1397The server side code::1398 1399 nstatuser@nstat-b:~$ cat server_linger.py1400 import socket1401 import time1402 1403 port = 90001404 1405 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1406 s.bind(('0.0.0.0', port))1407 s.listen(1)1408 sock, addr = s.accept()1409 while True:1410 time.sleep(9999999)1411 1412The client side code::1413 1414 nstatuser@nstat-a:~$ cat client_linger.py1415 import socket1416 import struct1417 1418 server = 'nstat-b' # server address1419 port = 90001420 1421 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1422 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))1423 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))1424 s.connect((server, port))1425 s.close()1426 1427Run server_linger.py on server::1428 1429 nstatuser@nstat-b:~$ python3 server_linger.py1430 1431Run client_linger.py on client::1432 1433 nstatuser@nstat-a:~$ python3 client_linger.py1434 1435After run client_linger.py, check the output of nstat::1436 1437 nstatuser@nstat-a:~$ nstat | grep -i abort1438 TcpExtTCPAbortOnLinger 1 0.01439 1440TcpExtTCPRcvCoalesce1441--------------------1442On the server, we run a program which listen on TCP port 9000, but1443doesn't read any data::1444 1445 import socket1446 import time1447 port = 90001448 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1449 s.bind(('0.0.0.0', port))1450 s.listen(1)1451 sock, addr = s.accept()1452 while True:1453 time.sleep(9999999)1454 1455Save the above code as server_coalesce.py, and run::1456 1457 python3 server_coalesce.py1458 1459On the client, save below code as client_coalesce.py::1460 1461 import socket1462 server = 'nstat-b'1463 port = 90001464 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)1465 s.connect((server, port))1466 1467Run::1468 1469 nstatuser@nstat-a:~$ python3 -i client_coalesce.py1470 1471We use '-i' to come into the interactive mode, then a packet::1472 1473 >>> s.send(b'foo')1474 31475 1476Send a packet again::1477 1478 >>> s.send(b'bar')1479 31480 1481On the server, run nstat::1482 1483 ubuntu@nstat-b:~$ nstat1484 #kernel1485 IpInReceives 2 0.01486 IpInDelivers 2 0.01487 IpOutRequests 2 0.01488 TcpInSegs 2 0.01489 TcpOutSegs 2 0.01490 TcpExtTCPRcvCoalesce 1 0.01491 IpExtInOctets 110 0.01492 IpExtOutOctets 104 0.01493 IpExtInNoECTPkts 2 0.01494 1495The client sent two packets, server didn't read any data. When1496the second packet arrived at server, the first packet was still in1497the receiving queue. So the TCP layer merged the two packets, and we1498could find the TcpExtTCPRcvCoalesce increased 1.1499 1500TcpExtListenOverflows and TcpExtListenDrops1501-------------------------------------------1502On server, run the nc command, listen on port 9000::1503 1504 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 90001505 Listening on [0.0.0.0] (family 0, port 9000)1506 1507On client, run 3 nc commands in different terminals::1508 1509 nstatuser@nstat-a:~$ nc -v nstat-b 90001510 Connection to nstat-b 9000 port [tcp/*] succeeded!1511 1512The nc command only accepts 1 connection, and the accept queue length1513is 1. On current linux implementation, set queue length to n means the1514actual queue length is n+1. Now we create 3 connections, 1 is accepted1515by nc, 2 in accepted queue, so the accept queue is full.1516 1517Before running the 4th nc, we clean the nstat history on the server::1518 1519 nstatuser@nstat-b:~$ nstat -n1520 1521Run the 4th nc on the client::1522 1523 nstatuser@nstat-a:~$ nc -v nstat-b 90001524 1525If the nc server is running on kernel 4.10 or higher version, you1526won't see the "Connection to ... succeeded!" string, because kernel1527will drop the SYN if the accept queue is full. If the nc client is running1528on an old kernel, you would see that the connection is succeeded,1529because kernel would complete the 3 way handshake and keep the socket1530on half open queue. I did the test on kernel 4.15. Below is the nstat1531on the server::1532 1533 nstatuser@nstat-b:~$ nstat1534 #kernel1535 IpInReceives 4 0.01536 IpInDelivers 4 0.01537 TcpInSegs 4 0.01538 TcpExtListenOverflows 4 0.01539 TcpExtListenDrops 4 0.01540 IpExtInOctets 240 0.01541 IpExtInNoECTPkts 4 0.01542 1543Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time1544between the 4th nc and the nstat was longer, the value of1545TcpExtListenOverflows and TcpExtListenDrops would be larger, because1546the SYN of the 4th nc was dropped, the client was retrying.1547 1548IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes1549-------------------------------------------------1550server A IP address: 192.168.122.2501551server B IP address: 192.168.122.2511552Prepare on server A, add a route to server B::1553 1554 $ sudo ip route add 8.8.8.8/32 via 192.168.122.2511555 1556Prepare on server B, disable send_redirects for all interfaces::1557 1558 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=01559 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=01560 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=01561 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=01562 1563We want to let sever A send a packet to 8.8.8.8, and route the packet1564to server B. When server B receives such packet, it might send a ICMP1565Redirect message to server A, set send_redirects to 0 will disable1566this behavior.1567 1568First, generate InAddrErrors. On server B, we disable IP forwarding::1569 1570 $ sudo sysctl -w net.ipv4.conf.all.forwarding=01571 1572On server A, we send packets to 8.8.8.8::1573 1574 $ nc -v 8.8.8.8 531575 1576On server B, we check the output of nstat::1577 1578 $ nstat1579 #kernel1580 IpInReceives 3 0.01581 IpInAddrErrors 3 0.01582 IpExtInOctets 180 0.01583 IpExtInNoECTPkts 3 0.01584 1585As we have let server A route 8.8.8.8 to server B, and we disabled IP1586forwarding on server B, Server A sent packets to server B, then server B1587dropped packets and increased IpInAddrErrors. As the nc command would1588re-send the SYN packet if it didn't receive a SYN+ACK, we could find1589multiple IpInAddrErrors.1590 1591Second, generate IpExtInNoRoutes. On server B, we enable IP1592forwarding::1593 1594 $ sudo sysctl -w net.ipv4.conf.all.forwarding=11595 1596Check the route table of server B and remove the default route::1597 1598 $ ip route show1599 default via 192.168.122.1 dev ens3 proto static1600 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.2511601 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static1602 1603On server A, we contact 8.8.8.8 again::1604 1605 $ nc -v 8.8.8.8 531606 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable1607 1608On server B, run nstat::1609 1610 $ nstat1611 #kernel1612 IpInReceives 1 0.01613 IpOutRequests 1 0.01614 IcmpOutMsgs 1 0.01615 IcmpOutDestUnreachs 1 0.01616 IcmpMsgOutType3 1 0.01617 IpExtInNoRoutes 1 0.01618 IpExtInOctets 60 0.01619 IpExtOutOctets 88 0.01620 IpExtInNoECTPkts 1 0.01621 1622We enabled IP forwarding on server B, when server B received a packet1623which destination IP address is 8.8.8.8, server B will try to forward1624this packet. We have deleted the default route, there was no route for16258.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP1626Destination Unreachable" message to server A.1627 1628Third, generate IpOutNoRoutes. Run ping command on server B::1629 1630 $ ping -c 1 8.8.8.81631 connect: Network is unreachable1632 1633Run nstat on server B::1634 1635 $ nstat1636 #kernel1637 IpOutNoRoutes 1 0.01638 1639We have deleted the default route on server B. Server B couldn't find1640a route for the 8.8.8.8 IP address, so server B increased1641IpOutNoRoutes.1642 1643TcpExtTCPACKSkippedSynRecv1644--------------------------1645In this test, we send 3 same SYN packets from client to server. The1646first SYN will let server create a socket, set it to Syn-Recv status,1647and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK1648again, and record the reply time (the duplicate ACK reply time). The1649third SYN will let server check the previous duplicate ACK reply time,1650and decide to skip the duplicate ACK, then increase the1651TcpExtTCPACKSkippedSynRecv counter.1652 1653Run tcpdump to capture a SYN packet::1654 1655 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 90001656 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes1657 1658Open another terminal, run nc command::1659 1660 nstatuser@nstat-a:~$ nc nstat-b 90001661 1662As the nstat-b didn't listen on port 9000, it should reply a RST, and1663the nc command exited immediately. It was enough for the tcpdump1664command to capture a SYN packet. A linux server might use hardware1665offload for the TCP checksum, so the checksum in the /tmp/syn.pcap1666might be not correct. We call tcprewrite to fix it::1667 1668 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum1669 1670On nstat-b, we run nc to listen on port 9000::1671 1672 nstatuser@nstat-b:~$ nc -lkv 90001673 Listening on [0.0.0.0] (family 0, port 9000)1674 1675On nstat-a, we blocked the packet from port 9000, or nstat-a would send1676RST to nstat-b::1677 1678 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP1679 1680Send 3 SYN repeatedly to nstat-b::1681 1682 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done1683 1684Check snmp counter on nstat-b::1685 1686 nstatuser@nstat-b:~$ nstat | grep -i skip1687 TcpExtTCPACKSkippedSynRecv 1 0.01688 1689As we expected, TcpExtTCPACKSkippedSynRecv is 1.1690 1691TcpExtTCPACKSkippedPAWS1692-----------------------1693To trigger PAWS, we could send an old SYN.1694 1695On nstat-b, let nc listen on port 9000::1696 1697 nstatuser@nstat-b:~$ nc -lkv 90001698 Listening on [0.0.0.0] (family 0, port 9000)1699 1700On nstat-a, run tcpdump to capture a SYN::1701 1702 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 90001703 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes1704 1705On nstat-a, run nc as a client to connect nstat-b::1706 1707 nstatuser@nstat-a:~$ nc -v nstat-b 90001708 Connection to nstat-b 9000 port [tcp/*] succeeded!1709 1710Now the tcpdump has captured the SYN and exit. We should fix the1711checksum::1712 1713 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum1714 1715Send the SYN packet twice::1716 1717 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done1718 1719On nstat-b, check the snmp counter::1720 1721 nstatuser@nstat-b:~$ nstat | grep -i skip1722 TcpExtTCPACKSkippedPAWS 1 0.01723 1724We sent two SYN via tcpreplay, both of them would let PAWS check1725failed, the nstat-b replied an ACK for the first SYN, skipped the ACK1726for the second SYN, and updated TcpExtTCPACKSkippedPAWS.1727 1728TcpExtTCPACKSkippedSeq1729----------------------1730To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid1731timestamp (to pass PAWS check) but the sequence number is out of1732window. The linux TCP stack would avoid to skip if the packet has1733data, so we need a pure ACK packet. To generate such a packet, we1734could create two sockets: one on port 9000, another on port 9001. Then1735we capture an ACK on port 9001, change the source/destination port1736numbers to match the port 9000 socket. Then we could trigger1737TcpExtTCPACKSkippedSeq via this packet.1738 1739On nstat-b, open two terminals, run two nc commands to listen on both1740port 9000 and port 9001::1741 1742 nstatuser@nstat-b:~$ nc -lkv 90001743 Listening on [0.0.0.0] (family 0, port 9000)1744 1745 nstatuser@nstat-b:~$ nc -lkv 90011746 Listening on [0.0.0.0] (family 0, port 9001)1747 1748On nstat-a, run two nc clients::1749 1750 nstatuser@nstat-a:~$ nc -v nstat-b 90001751 Connection to nstat-b 9000 port [tcp/*] succeeded!1752 1753 nstatuser@nstat-a:~$ nc -v nstat-b 90011754 Connection to nstat-b 9001 port [tcp/*] succeeded!1755 1756On nstat-a, run tcpdump to capture an ACK::1757 1758 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 90011759 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes1760 1761On nstat-b, send a packet via the port 9001 socket. E.g. we sent a1762string 'foo' in our example::1763 1764 nstatuser@nstat-b:~$ nc -lkv 90011765 Listening on [0.0.0.0] (family 0, port 9001)1766 Connection from nstat-a 42132 received!1767 foo1768 1769On nstat-a, the tcpdump should have captured the ACK. We should check1770the source port numbers of the two nc clients::1771 1772 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee1773 State Recv-Q Send-Q Local Address:Port Peer Address:Port1774 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:90001775 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:90011776 1777Run tcprewrite, change port 9001 to port 9000, change port 42132 to1778port 50208::1779 1780 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum1781 1782Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::1783 1784 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done1785 1786Check TcpExtTCPACKSkippedSeq on nstat-b::1787 1788 nstatuser@nstat-b:~$ nstat | grep -i skip1789 TcpExtTCPACKSkippedSeq 1 0.01790