822 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3============4Timestamping5============6 7 81. Control Interfaces9=====================10 11The interfaces for receiving network packages timestamps are:12 13SO_TIMESTAMP14 Generates a timestamp for each incoming packet in (not necessarily15 monotonic) system time. Reports the timestamp via recvmsg() in a16 control message in usec resolution.17 SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD18 based on the architecture type and time_t representation of libc.19 Control message format is in struct __kernel_old_timeval for20 SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for21 SO_TIMESTAMP_NEW options respectively.22 23SO_TIMESTAMPNS24 Same timestamping mechanism as SO_TIMESTAMP, but reports the25 timestamp as struct timespec in nsec resolution.26 SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD27 based on the architecture type and time_t representation of libc.28 Control message format is in struct timespec for SO_TIMESTAMPNS_OLD29 and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options30 respectively.31 32IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]33 Only for multicast:approximate transmit timestamp obtained by34 reading the looped packet receive timestamp.35 36SO_TIMESTAMPING37 Generates timestamps on reception, transmission or both. Supports38 multiple timestamp sources, including hardware. Supports generating39 timestamps for stream sockets.40 41 421.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW)43-------------------------------------------------------------44 45This socket option enables timestamping of datagrams on the reception46path. Because the destination socket, if any, is not known early in47the network stack, the feature has to be enabled for all packets. The48same is true for all early receive timestamp options.49 50For interface details, see `man 7 socket`.51 52Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in53struct __kernel_sock_timeval format.54 55SO_TIMESTAMP_OLD returns incorrect timestamps after the year 203856on 32 bit machines.57 581.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW)59-------------------------------------------------------------------60 61This option is identical to SO_TIMESTAMP except for the returned data type.62Its struct timespec allows for higher resolution (ns) timestamps than the63timeval of SO_TIMESTAMP (ms).64 65Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in66struct __kernel_timespec format.67 68SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 203869on 32 bit machines.70 711.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW)72----------------------------------------------------------------------73 74Supports multiple types of timestamp requests. As a result, this75socket option takes a bitmap of flags, not a boolean. In::76 77 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));78 79val is an integer with any of the following bits set. Setting other80bit returns EINVAL and does not change the current state.81 82The socket option configures timestamp generation for individual83sk_buffs (1.3.1), timestamp reporting to the socket's error84queue (1.3.2) and options (1.3.3). Timestamp generation can also85be enabled for individual sendmsg calls using cmsg (1.3.4).86 87 881.3.1 Timestamp Generation89^^^^^^^^^^^^^^^^^^^^^^^^^^90 91Some bits are requests to the stack to try to generate timestamps. Any92combination of them is valid. Changes to these bits apply to newly93created packets, not to packets already in the stack. As a result, it94is possible to selectively request timestamps for a subset of packets95(e.g., for sampling) by embedding an send() call within two setsockopt96calls, one to enable timestamp generation and one to disable it.97Timestamps may also be generated for reasons other than being98requested by a particular socket, such as when receive timestamping is99enabled system wide, as explained earlier.100 101SOF_TIMESTAMPING_RX_HARDWARE:102 Request rx timestamps generated by the network adapter.103 104SOF_TIMESTAMPING_RX_SOFTWARE:105 Request rx timestamps when data enters the kernel. These timestamps106 are generated just after a device driver hands a packet to the107 kernel receive stack.108 109SOF_TIMESTAMPING_TX_HARDWARE:110 Request tx timestamps generated by the network adapter. This flag111 can be enabled via both socket options and control messages.112 113SOF_TIMESTAMPING_TX_SOFTWARE:114 Request tx timestamps when data leaves the kernel. These timestamps115 are generated in the device driver as close as possible, but always116 prior to, passing the packet to the network interface. Hence, they117 require driver support and may not be available for all devices.118 This flag can be enabled via both socket options and control messages.119 120SOF_TIMESTAMPING_TX_SCHED:121 Request tx timestamps prior to entering the packet scheduler. Kernel122 transmit latency is, if long, often dominated by queuing delay. The123 difference between this timestamp and one taken at124 SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent125 of protocol processing. The latency incurred in protocol126 processing, if any, can be computed by subtracting a userspace127 timestamp taken immediately before send() from this timestamp. On128 machines with virtual devices where a transmitted packet travels129 through multiple devices and, hence, multiple packet schedulers,130 a timestamp is generated at each layer. This allows for fine131 grained measurement of queuing delay. This flag can be enabled132 via both socket options and control messages.133 134SOF_TIMESTAMPING_TX_ACK:135 Request tx timestamps when all data in the send buffer has been136 acknowledged. This only makes sense for reliable protocols. It is137 currently only implemented for TCP. For that protocol, it may138 over-report measurement, because the timestamp is generated when all139 data up to and including the buffer at send() was acknowledged: the140 cumulative acknowledgment. The mechanism ignores SACK and FACK.141 This flag can be enabled via both socket options and control messages.142 143 1441.3.2 Timestamp Reporting145^^^^^^^^^^^^^^^^^^^^^^^^^146 147The other three bits control which timestamps will be reported in a148generated control message. Changes to the bits take immediate149effect at the timestamp reporting locations in the stack. Timestamps150are only reported for packets that also have the relevant timestamp151generation request set.152 153SOF_TIMESTAMPING_SOFTWARE:154 Report any software timestamps when available.155 156SOF_TIMESTAMPING_SYS_HARDWARE:157 This option is deprecated and ignored.158 159SOF_TIMESTAMPING_RAW_HARDWARE:160 Report hardware timestamps as generated by161 SOF_TIMESTAMPING_TX_HARDWARE or SOF_TIMESTAMPING_RX_HARDWARE162 when available.163 164 1651.3.3 Timestamp Options166^^^^^^^^^^^^^^^^^^^^^^^167 168The interface supports the options169 170SOF_TIMESTAMPING_OPT_ID:171 Generate a unique identifier along with each packet. A process can172 have multiple concurrent timestamping requests outstanding. Packets173 can be reordered in the transmit path, for instance in the packet174 scheduler. In that case timestamps will be queued onto the error175 queue out of order from the original send() calls. It is not always176 possible to uniquely match timestamps to the original send() calls177 based on timestamp order or payload inspection alone, then.178 179 This option associates each packet at send() with a unique180 identifier and returns that along with the timestamp. The identifier181 is derived from a per-socket u32 counter (that wraps). For datagram182 sockets, the counter increments with each sent packet. For stream183 sockets, it increments with every byte. For stream sockets, also set184 SOF_TIMESTAMPING_OPT_ID_TCP, see the section below.185 186 The counter starts at zero. It is initialized the first time that187 the socket option is enabled. It is reset each time the option is188 enabled after having been disabled. Resetting the counter does not189 change the identifiers of existing packets in the system.190 191 This option is implemented only for transmit timestamps. There, the192 timestamp is always looped along with a struct sock_extended_err.193 The option modifies field ee_data to pass an id that is unique194 among all possibly concurrently outstanding timestamp requests for195 that socket.196 197SOF_TIMESTAMPING_OPT_ID_TCP:198 Pass this modifier along with SOF_TIMESTAMPING_OPT_ID for new TCP199 timestamping applications. SOF_TIMESTAMPING_OPT_ID defines how the200 counter increments for stream sockets, but its starting point is201 not entirely trivial. This option fixes that.202 203 For stream sockets, if SOF_TIMESTAMPING_OPT_ID is set, this should204 always be set too. On datagram sockets the option has no effect.205 206 A reasonable expectation is that the counter is reset to zero with207 the system call, so that a subsequent write() of N bytes generates208 a timestamp with counter N-1. SOF_TIMESTAMPING_OPT_ID_TCP209 implements this behavior under all conditions.210 211 SOF_TIMESTAMPING_OPT_ID without modifier often reports the same,212 especially when the socket option is set when no data is in213 transmission. If data is being transmitted, it may be off by the214 length of the output queue (SIOCOUTQ).215 216 The difference is due to being based on snd_una versus write_seq.217 snd_una is the offset in the stream acknowledged by the peer. This218 depends on factors outside of process control, such as network RTT.219 write_seq is the last byte written by the process. This offset is220 not affected by external inputs.221 222 The difference is subtle and unlikely to be noticed when configured223 at initial socket creation, when no data is queued or sent. But224 SOF_TIMESTAMPING_OPT_ID_TCP behavior is more robust regardless of225 when the socket option is set.226 227SOF_TIMESTAMPING_OPT_CMSG:228 Support recv() cmsg for all timestamped packets. Control messages229 are already supported unconditionally on all packets with receive230 timestamps and on IPv6 packets with transmit timestamp. This option231 extends them to IPv4 packets with transmit timestamp. One use case232 is to correlate packets with their egress device, by enabling socket233 option IP_PKTINFO simultaneously.234 235 236SOF_TIMESTAMPING_OPT_TSONLY:237 Applies to transmit timestamps only. Makes the kernel return the238 timestamp as a cmsg alongside an empty packet, as opposed to239 alongside the original packet. This reduces the amount of memory240 charged to the socket's receive budget (SO_RCVBUF) and delivers241 the timestamp even if sysctl net.core.tstamp_allow_data is 0.242 This option disables SOF_TIMESTAMPING_OPT_CMSG.243 244SOF_TIMESTAMPING_OPT_STATS:245 Optional stats that are obtained along with the transmit timestamps.246 It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the247 transmit timestamp is available, the stats are available in a248 separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a249 list of TLVs (struct nlattr) of types. These stats allow the250 application to associate various transport layer stats with251 the transmit timestamps, such as how long a certain block of252 data was limited by peer's receiver window.253 254SOF_TIMESTAMPING_OPT_PKTINFO:255 Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming256 packets with hardware timestamps. The message contains struct257 scm_ts_pktinfo, which supplies the index of the real interface which258 received the packet and its length at layer 2. A valid (non-zero)259 interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is260 enabled and the driver is using NAPI. The struct contains also two261 other fields, but they are reserved and undefined.262 263SOF_TIMESTAMPING_OPT_TX_SWHW:264 Request both hardware and software timestamps for outgoing packets265 when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE266 are enabled at the same time. If both timestamps are generated,267 two separate messages will be looped to the socket's error queue,268 each containing just one timestamp.269 270SOF_TIMESTAMPING_OPT_RX_FILTER:271 Filter out spurious receive timestamps: report a receive timestamp272 only if the matching timestamp generation flag is enabled.273 274 Receive timestamps are generated early in the ingress path, before a275 packet's destination socket is known. If any socket enables receive276 timestamps, packets for all socket will receive timestamped packets.277 Including those that request timestamp reporting with278 SOF_TIMESTAMPING_SOFTWARE and/or SOF_TIMESTAMPING_RAW_HARDWARE, but279 do not request receive timestamp generation. This can happen when280 requesting transmit timestamps only.281 282 Receiving spurious timestamps is generally benign. A process can283 ignore the unexpected non-zero value. But it makes behavior subtly284 dependent on other sockets. This flag isolates the socket for more285 deterministic behavior.286 287New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to288disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate289regardless of the setting of sysctl net.core.tstamp_allow_data.290 291An exception is when a process needs additional cmsg data, for292instance SOL_IP/IP_PKTINFO to detect the egress network interface.293Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on294having access to the contents of the original packet, so cannot be295combined with SOF_TIMESTAMPING_OPT_TSONLY.296 297 2981.3.4. Enabling timestamps via control messages299^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^300 301In addition to socket options, timestamp generation can be requested302per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).303Using this feature, applications can sample timestamps per sendmsg()304without paying the overhead of enabling and disabling timestamps via305setsockopt::306 307 struct msghdr *msg;308 ...309 cmsg = CMSG_FIRSTHDR(msg);310 cmsg->cmsg_level = SOL_SOCKET;311 cmsg->cmsg_type = SO_TIMESTAMPING;312 cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));313 *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |314 SOF_TIMESTAMPING_TX_SOFTWARE |315 SOF_TIMESTAMPING_TX_ACK;316 err = sendmsg(fd, msg, 0);317 318The SOF_TIMESTAMPING_TX_* flags set via cmsg will override319the SOF_TIMESTAMPING_TX_* flags set via setsockopt.320 321Moreover, applications must still enable timestamp reporting via322setsockopt to receive timestamps::323 324 __u32 val = SOF_TIMESTAMPING_SOFTWARE |325 SOF_TIMESTAMPING_OPT_ID /* or any other flag */;326 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));327 328 3291.4 Bytestream Timestamps330-------------------------331 332The SO_TIMESTAMPING interface supports timestamping of bytes in a333bytestream. Each request is interpreted as a request for when the334entire contents of the buffer has passed a timestamping point. That335is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record336when all bytes have reached the device driver, regardless of how337many packets the data has been converted into.338 339In general, bytestreams have no natural delimiters and therefore340correlating a timestamp with data is non-trivial. A range of bytes341may be split across segments, any segments may be merged (possibly342coalescing sections of previously segmented buffers associated with343independent send() calls). Segments can be reordered and the same344byte range can coexist in multiple segments for protocols that345implement retransmissions.346 347It is essential that all timestamps implement the same semantics,348regardless of these possible transformations, as otherwise they are349incomparable. Handling "rare" corner cases differently from the350simple case (a 1:1 mapping from buffer to skb) is insufficient351because performance debugging often needs to focus on such outliers.352 353In practice, timestamps can be correlated with segments of a354bytestream consistently, if both semantics of the timestamp and the355timing of measurement are chosen correctly. This challenge is no356different from deciding on a strategy for IP fragmentation. There, the357definition is that only the first fragment is timestamped. For358bytestreams, we chose that a timestamp is generated only when all359bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to360implement and reason about. An implementation that has to take into361account SACK would be more complex due to possible transmission holes362and out of order arrival.363 364On the host, TCP can also break the simple 1:1 mapping from buffer to365skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The366implementation ensures correctness in all cases by tracking the367individual last byte passed to send(), even if it is no longer the368last byte after an skbuff extend or merge operation. It stores the369relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff370has only one such field, only one timestamp can be generated.371 372In rare cases, a timestamp request can be missed if two requests are373collapsed onto the same skb. A process can detect this situation by374enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at375send time with the value returned for each timestamp. It can prevent376the situation by always flushing the TCP stack in between requests,377for instance by enabling TCP_NODELAY and disabling TCP_CORK and378autocork. After linux-4.7, a better way to prevent coalescing is379to use MSG_EOR flag at sendmsg() time.380 381These precautions ensure that the timestamp is generated only when all382bytes have passed a timestamp point, assuming that the network stack383itself does not reorder the segments. The stack indeed tries to avoid384reordering. The one exception is under administrator control: it is385possible to construct a packet scheduler configuration that delays386segments from the same stream differently. Such a setup would be387unusual.388 389 3902 Data Interfaces391==================392 393Timestamps are read using the ancillary data feature of recvmsg().394See `man 3 cmsg` for details of this interface. The socket manual395page (`man 7 socket`) describes how timestamps generated with396SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.397 398 3992.1 SCM_TIMESTAMPING records400----------------------------401 402These timestamps are returned in a control message with cmsg_level403SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type404 405For SO_TIMESTAMPING_OLD::406 407 struct scm_timestamping {408 struct timespec ts[3];409 };410 411For SO_TIMESTAMPING_NEW::412 413 struct scm_timestamping64 {414 struct __kernel_timespec ts[3];415 416Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in417struct scm_timestamping64 format.418 419SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038420on 32 bit machines.421 422The structure can return up to three timestamps. This is a legacy423feature. At least one field is non-zero at any time. Most timestamps424are passed in ts[0]. Hardware timestamps are passed in ts[2].425 426ts[1] used to hold hardware timestamps converted to system time.427Instead, expose the hardware clock device on the NIC directly as428a HW PTP clock source, to allow time conversion in userspace and429optionally synchronize system time with a userspace PTP stack such430as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.431 432Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled433together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false434software timestamp will be generated in the recvmsg() call and passed435in ts[0] when a real software timestamp is missing. This happens also436on hardware transmit timestamps.437 4382.1.1 Transmit timestamps with MSG_ERRQUEUE439^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^440 441For transmit timestamps the outgoing packet is looped back to the442socket's error queue with the send timestamp(s) attached. A process443receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE444set and with a msg_control buffer sufficiently large to receive the445relevant metadata structures. The recvmsg call returns the original446outgoing data packet with two ancillary messages attached.447 448A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR449embeds a struct sock_extended_err. This defines the error type. For450timestamps, the ee_errno field is ENOMSG. The other ancillary message451will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This452embeds the struct scm_timestamping.453 454 4552.1.1.2 Timestamp types456~~~~~~~~~~~~~~~~~~~~~~~457 458The semantics of the three struct timespec are defined by field459ee_info in the extended error structure. It contains a value of460type SCM_TSTAMP_* to define the actual timestamp passed in461scm_timestamping.462 463The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*464control fields discussed previously, with one exception. For legacy465reasons, SCM_TSTAMP_SND is equal to zero and can be set for both466SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It467is the first if ts[2] is non-zero, the second otherwise, in which468case the timestamp is stored in ts[0].469 470 4712.1.1.3 Fragmentation472~~~~~~~~~~~~~~~~~~~~~473 474Fragmentation of outgoing datagrams is rare, but is possible, e.g., by475explicitly disabling PMTU discovery. If an outgoing packet is fragmented,476then only the first fragment is timestamped and returned to the sending477socket.478 479 4802.1.1.4 Packet Payload481~~~~~~~~~~~~~~~~~~~~~~482 483The calling application is often not interested in receiving the whole484packet payload that it passed to the stack originally: the socket485error queue mechanism is just a method to piggyback the timestamp on.486In this case, the application can choose to read datagrams with a487smaller buffer, possibly even of length 0. The payload is truncated488accordingly. Until the process calls recvmsg() on the error queue,489however, the full packet is queued, taking up budget from SO_RCVBUF.490 491 4922.1.1.5 Blocking Read493~~~~~~~~~~~~~~~~~~~~~494 495Reading from the error queue is always a non-blocking operation. To496block waiting on a timestamp, use poll or select. poll() will return497POLLERR in pollfd.revents if any data is ready on the error queue.498There is no need to pass this flag in pollfd.events. This flag is499ignored on request. See also `man 2 poll`.500 501 5022.1.2 Receive timestamps503^^^^^^^^^^^^^^^^^^^^^^^^504 505On reception, there is no reason to read from the socket error queue.506The SCM_TIMESTAMPING ancillary data is sent along with the packet data507on a normal recvmsg(). Since this is not a socket error, it is not508accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,509the meaning of the three fields in struct scm_timestamping is510implicitly defined. ts[0] holds a software timestamp if set, ts[1]511is again deprecated and ts[2] holds a hardware timestamp if set.512 513 5143. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP515=======================================================================516 517Hardware time stamping must also be initialized for each device driver518that is expected to do hardware time stamping. The parameter is defined in519include/uapi/linux/net_tstamp.h as::520 521 struct hwtstamp_config {522 int flags; /* no flags defined right now, must be zero */523 int tx_type; /* HWTSTAMP_TX_* */524 int rx_filter; /* HWTSTAMP_FILTER_* */525 };526 527Desired behavior is passed into the kernel and to a specific device by528calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose529ifr_data points to a struct hwtstamp_config. The tx_type and530rx_filter are hints to the driver what it is expected to do. If531the requested fine-grained filtering for incoming packets is not532supported, the driver may time stamp more than just the requested types533of packets.534 535Drivers are free to use a more permissive configuration than the requested536configuration. It is expected that drivers should only implement directly the537most generic mode that can be supported. For example if the hardware can538support HWTSTAMP_FILTER_PTP_V2_EVENT, then it should generally always upscale539HWTSTAMP_FILTER_PTP_V2_L2_SYNC, and so forth, as HWTSTAMP_FILTER_PTP_V2_EVENT540is more generic (and more useful to applications).541 542A driver which supports hardware time stamping shall update the struct543with the actual, possibly more permissive configuration. If the544requested packets cannot be time stamped, then nothing should be545changed and ERANGE shall be returned (in contrast to EINVAL, which546indicates that SIOCSHWTSTAMP is not supported at all).547 548Only a processes with admin rights may change the configuration. User549space is responsible to ensure that multiple processes don't interfere550with each other and that the settings are reset.551 552Any process can read the actual configuration by passing this553structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has554not been implemented in all drivers.555 556::557 558 /* possible values for hwtstamp_config->tx_type */559 enum {560 /*561 * no outgoing packet will need hardware time stamping;562 * should a packet arrive which asks for it, no hardware563 * time stamping will be done564 */565 HWTSTAMP_TX_OFF,566 567 /*568 * enables hardware time stamping for outgoing packets;569 * the sender of the packet decides which are to be570 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE571 * before sending the packet572 */573 HWTSTAMP_TX_ON,574 };575 576 /* possible values for hwtstamp_config->rx_filter */577 enum {578 /* time stamp no incoming packet at all */579 HWTSTAMP_FILTER_NONE,580 581 /* time stamp any incoming packet */582 HWTSTAMP_FILTER_ALL,583 584 /* return value: time stamp all packets requested plus some others */585 HWTSTAMP_FILTER_SOME,586 587 /* PTP v1, UDP, any kind of event packet */588 HWTSTAMP_FILTER_PTP_V1_L4_EVENT,589 590 /* for the complete list of values, please check591 * the include file include/uapi/linux/net_tstamp.h592 */593 };594 5953.1 Hardware Timestamping Implementation: Device Drivers596--------------------------------------------------------597 598A driver which supports hardware time stamping must support the599SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with600the actual values as described in the section on SIOCSHWTSTAMP. It601should also support SIOCGHWTSTAMP.602 603Time stamps for received packets must be stored in the skb. To get a pointer604to the shared time stamp structure of the skb call skb_hwtstamps(). Then605set the time stamps in the structure::606 607 struct skb_shared_hwtstamps {608 /* hardware time stamp transformed into duration609 * since arbitrary point in time610 */611 ktime_t hwtstamp;612 };613 614Time stamps for outgoing packets are to be generated as follows:615 616- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)617 is set no-zero. If yes, then the driver is expected to do hardware time618 stamping.619- If this is possible for the skb and requested, then declare620 that the driver is doing the time stamping by setting the flag621 SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with::622 623 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;624 625 You might want to keep a pointer to the associated skb for the next step626 and not free the skb. A driver not supporting hardware time stamping doesn't627 do that. A driver must never touch sk_buff::tstamp! It is used to store628 software generated time stamps by the network subsystem.629- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware630 as possible. skb_tx_timestamp() provides a software time stamp if requested631 and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).632- As soon as the driver has sent the packet and/or obtained a633 hardware time stamp for it, it passes the time stamp back by634 calling skb_tstamp_tx() with the original skb, the raw635 hardware time stamp. skb_tstamp_tx() clones the original skb and636 adds the timestamps, therefore the original skb has to be freed now.637 If obtaining the hardware time stamp somehow fails, then the driver638 should not fall back to software time stamping. The rationale is that639 this would occur at a later time in the processing pipeline than other640 software time stamping and therefore could lead to unexpected deltas641 between time stamps.642 6433.2 Special considerations for stacked PTP Hardware Clocks644----------------------------------------------------------645 646There are situations when there may be more than one PHC (PTP Hardware Clock)647in the data path of a packet. The kernel has no explicit mechanism to allow the648user to select which PHC to use for timestamping Ethernet frames. Instead, the649assumption is that the outermost PHC is always the most preferable, and that650kernel drivers collaborate towards achieving that goal. Currently there are 3651cases of stacked PHCs, detailed below:652 6533.2.1 DSA (Distributed Switch Architecture) switches654^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^655 656These are Ethernet switches which have one of their ports connected to an657(otherwise completely unaware) host Ethernet interface, and perform the role of658a port multiplier with optional forwarding acceleration features. Each DSA659switch port is visible to the user as a standalone (virtual) network interface,660and its network I/O is performed, under the hood, indirectly through the host661interface (redirecting to the host port on TX, and intercepting frames on RX).662 663When a DSA switch is attached to a host port, PTP synchronization has to664suffer, since the switch's variable queuing delay introduces a path delay665jitter between the host port and its PTP partner. For this reason, some DSA666switches include a timestamping clock of their own, and have the ability to667perform network timestamping on their own MAC, such that path delays only668measure wire and PHY propagation latencies. Timestamping DSA switches are669supported in Linux and expose the same ABI as any other network interface (save670for the fact that the DSA interfaces are in fact virtual in terms of network671I/O, they do have their own PHC). It is typical, but not mandatory, for all672interfaces of a DSA switch to share the same PHC.673 674By design, PTP timestamping with a DSA switch does not need any special675handling in the driver for the host port it is attached to. However, when the676host port also supports PTP timestamping, DSA will take care of intercepting677the ``.ndo_eth_ioctl`` calls towards the host port, and block attempts to enable678hardware timestamping on it. This is because the SO_TIMESTAMPING API does not679allow the delivery of multiple hardware timestamps for the same packet, so680anybody else except for the DSA switch port must be prevented from doing so.681 682In the generic layer, DSA provides the following infrastructure for PTP683timestamping:684 685- ``.port_txtstamp()``: a hook called prior to the transmission of686 packets with a hardware TX timestamping request from user space.687 This is required for two-step timestamping, since the hardware688 timestamp becomes available after the actual MAC transmission, so the689 driver must be prepared to correlate the timestamp with the original690 packet so that it can re-enqueue the packet back into the socket's691 error queue. To save the packet for when the timestamp becomes692 available, the driver can call ``skb_clone_sk`` , save the clone pointer693 in skb->cb and enqueue a tx skb queue. Typically, a switch will have a694 PTP TX timestamp register (or sometimes a FIFO) where the timestamp695 becomes available. In case of a FIFO, the hardware might store696 key-value pairs of PTP sequence ID/message type/domain number and the697 actual timestamp. To perform the correlation correctly between the698 packets in a queue waiting for timestamping and the actual timestamps,699 drivers can use a BPF classifier (``ptp_classify_raw``) to identify700 the PTP transport type, and ``ptp_parse_header`` to interpret the PTP701 header fields. There may be an IRQ that is raised upon this702 timestamp's availability, or the driver might have to poll after703 invoking ``dev_queue_xmit()`` towards the host interface.704 One-step TX timestamping do not require packet cloning, since there is705 no follow-up message required by the PTP protocol (because the706 TX timestamp is embedded into the packet by the MAC), and therefore707 user space does not expect the packet annotated with the TX timestamp708 to be re-enqueued into its socket's error queue.709 710- ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to711 identify PTP event messages (any other packets, including PTP general712 messages, are not timestamped). The original (and only) timestampable713 skb is provided to the driver, for it to annotate it with a timestamp,714 if that is immediately available, or defer to later. On reception,715 timestamps might either be available in-band (through metadata in the716 DSA header, or attached in other ways to the packet), or out-of-band717 (through another RX timestamping FIFO). Deferral on RX is typically718 necessary when retrieving the timestamp needs a sleepable context. In719 that case, it is the responsibility of the DSA driver to call720 ``netif_rx()`` on the freshly timestamped skb.721 7223.2.2 Ethernet PHYs723^^^^^^^^^^^^^^^^^^^724 725These are devices that typically fulfill a Layer 1 role in the network stack,726hence they do not have a representation in terms of a network interface as DSA727switches do. However, PHYs may be able to detect and timestamp PTP packets, for728performance reasons: timestamps taken as close as possible to the wire have the729potential to yield a more stable and precise synchronization.730 731A PHY driver that supports PTP timestamping must create a ``struct732mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence733of this pointer will be checked by the networking stack.734 735Since PHYs do not have network interface representations, the timestamping and736ethtool ioctl operations for them need to be mediated by their respective MAC737driver. Therefore, as opposed to DSA switches, modifications need to be done738to each individual MAC driver for PHY timestamping support. This entails:739 740- Checking, in ``.ndo_eth_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)``741 is true or not. If it is, then the MAC driver should not process this request742 but instead pass it on to the PHY using ``phy_mii_ioctl()``.743 744- On RX, special intervention may or may not be needed, depending on the745 function used to deliver skb's up the network stack. In the case of plain746 ``netif_rx()`` and similar, MAC drivers must check whether747 ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't748 call ``netif_rx()`` at all. If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is749 enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook750 will be called now, to determine, using logic very similar to DSA, whether751 deferral for RX timestamping is necessary. Again like DSA, it becomes the752 responsibility of the PHY driver to send the packet up the stack when the753 timestamp is available.754 755 For other skb receive functions, such as ``napi_gro_receive`` and756 ``netif_receive_skb``, the stack automatically checks whether757 ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside758 the driver.759 760- On TX, again, special intervention might or might not be needed. The761 function that calls the ``mii_ts->txtstamp()`` hook is named762 ``skb_clone_tx_timestamp()``. This function can either be called directly763 (case in which explicit MAC driver support is indeed needed), but the764 function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC765 drivers already perform for software timestamping purposes. Therefore, if a766 MAC supports software timestamping, it does not need to do anything further767 at this stage.768 7693.2.3 MII bus snooping devices770^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^771 772These perform the same role as timestamping Ethernet PHYs, save for the fact773that they are discrete devices and can therefore be used in conjunction with774any PHY even if it doesn't support timestamping. In Linux, they are775discoverable and attachable to a ``struct phy_device`` through Device Tree, and776for the rest, they use the same mii_ts infrastructure as those. See777Documentation/devicetree/bindings/ptp/timestamper.txt for more details.778 7793.2.4 Other caveats for MAC drivers780^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^781 782Stacked PHCs, especially DSA (but not only) - since that doesn't require any783modification to MAC drivers, so it is more difficult to ensure correctness of784all possible code paths - is that they uncover bugs which were impossible to785trigger before the existence of stacked PTP clocks. One example has to do with786this line of code, already presented earlier::787 788 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;789 790Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY791driver or a MII bus snooping device driver, should set this flag.792But a MAC driver that is unaware of PHC stacking might get tripped up by793somebody other than itself setting this flag, and deliver a duplicate794timestamp.795For example, a typical driver design for TX timestamping might be to split the796transmission part into 2 portions:797 7981. "TX": checks whether PTP timestamping has been previously enabled through799 the ``.ndo_eth_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the800 current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags &801 SKBTX_HW_TSTAMP``"). If this is true, it sets the802 "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as803 described above, in the case of a stacked PHC system, this condition should804 never trigger, as this MAC is certainly not the outermost PHC. But this is805 not where the typical issue is. Transmission proceeds with this packet.806 8072. "TX confirmation": Transmission has finished. The driver checks whether it808 is necessary to collect any TX timestamp for it. Here is where the typical809 issues are: the MAC driver takes a shortcut and only checks whether810 "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked811 PHC system, this is incorrect because this MAC driver is not the only entity812 in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first813 place.814 815The correct solution for this problem is for MAC drivers to have a compound816check in their "TX confirmation" portion, not only for817"``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for818"``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures819that PTP timestamping is not enabled for anything other than the outermost PHC,820this enhanced check will avoid delivering a duplicated TX timestamp to user821space.822