337 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3======================4 USB4 and Thunderbolt5======================6USB4 is the public specification based on Thunderbolt 3 protocol with7some differences at the register level among other things. Connection8manager is an entity running on the host router (host controller)9responsible for enumerating routers and establishing tunnels. A10connection manager can be implemented either in firmware or software.11Typically PCs come with a firmware connection manager for Thunderbolt 312and early USB4 capable systems. Apple systems on the other hand use13software connection manager and the later USB4 compliant devices follow14the suit.15 16The Linux Thunderbolt driver supports both and can detect at runtime which17connection manager implementation is to be used. To be on the safe side the18software connection manager in Linux also advertises security level19``user`` which means PCIe tunneling is disabled by default. The20documentation below applies to both implementations with the exception that21the software connection manager only supports ``user`` security level and22is expected to be accompanied with an IOMMU based DMA protection.23 24Security levels and how to use them25-----------------------------------26The interface presented here is not meant for end users. Instead there27should be a userspace tool that handles all the low-level details, keeps28a database of the authorized devices and prompts users for new connections.29 30More details about the sysfs interface for Thunderbolt devices can be31found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.32 33Those users who just want to connect any device without any sort of34manual work can add following line to35``/etc/udev/rules.d/99-local.rules``::36 37 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"38 39This will authorize all devices automatically when they appear. However,40keep in mind that this bypasses the security levels and makes the system41vulnerable to DMA attacks.42 43Starting with Intel Falcon Ridge Thunderbolt controller there are 444security levels available. Intel Titan Ridge added one more security level45(usbonly). The reason for these is the fact that the connected devices can46be DMA masters and thus read contents of the host memory without CPU and OS47knowing about it. There are ways to prevent this by setting up an IOMMU but48it is not always available for various reasons.49 50Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is51treated as another security level (nopcie).52 53The security levels are as follows:54 55 none56 All devices are automatically connected by the firmware. No user57 approval is needed. In BIOS settings this is typically called58 *Legacy mode*.59 60 user61 User is asked whether the device is allowed to be connected.62 Based on the device identification information available through63 ``/sys/bus/thunderbolt/devices``, the user then can make the decision.64 In BIOS settings this is typically called *Unique ID*.65 66 secure67 User is asked whether the device is allowed to be connected. In68 addition to UUID the device (if it supports secure connect) is sent69 a challenge that should match the expected one based on a random key70 written to the ``key`` sysfs attribute. In BIOS settings this is71 typically called *One time saved key*.72 73 dponly74 The firmware automatically creates tunnels for Display Port and75 USB. No PCIe tunneling is done. In BIOS settings this is76 typically called *Display Port Only*.77 78 usbonly79 The firmware automatically creates tunnels for the USB controller and80 Display Port in a dock. All PCIe links downstream of the dock are81 removed.82 83 nopcie84 PCIe tunneling is disabled/forbidden from the BIOS. Available in some85 USB4 systems.86 87The current security level can be read from88``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is89the Thunderbolt domain the host controller manages. There is typically90one domain per Thunderbolt host controller.91 92If the security level reads as ``user`` or ``secure`` the connected93device must be authorized by the user before PCIe tunnels are created94(e.g the PCIe device appears).95 96Each Thunderbolt device plugged in will appear in sysfs under97``/sys/bus/thunderbolt/devices``. The device directory carries98information that can be used to identify the particular device,99including its name and UUID.100 101Authorizing devices when security level is ``user`` or ``secure``102-----------------------------------------------------------------103When a device is plugged in it will appear in sysfs as follows::104 105 /sys/bus/thunderbolt/devices/0-1/authorized - 0106 /sys/bus/thunderbolt/devices/0-1/device - 0x8004107 /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter108 /sys/bus/thunderbolt/devices/0-1/vendor - 0x1109 /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc.110 /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff111 112The ``authorized`` attribute reads 0 which means no PCIe tunnels are113created yet. The user can authorize the device by simply entering::114 115 # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized116 117This will create the PCIe tunnels and the device is now connected.118 119If the device supports secure connect, and the domain security level is120set to ``secure``, it has an additional attribute ``key`` which can hold121a random 32-byte value used for authorization and challenging the device in122future connects::123 124 /sys/bus/thunderbolt/devices/0-3/authorized - 0125 /sys/bus/thunderbolt/devices/0-3/device - 0x305126 /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box127 /sys/bus/thunderbolt/devices/0-3/key -128 /sys/bus/thunderbolt/devices/0-3/vendor - 0x41129 /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron130 /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16131 132Notice the key is empty by default.133 134If the user does not want to use secure connect they can just ``echo 1``135to the ``authorized`` attribute and the PCIe tunnels will be created in136the same way as in the ``user`` security level.137 138If the user wants to use secure connect, the first time the device is139plugged a key needs to be created and sent to the device::140 141 # key=$(openssl rand -hex 32)142 # echo $key > /sys/bus/thunderbolt/devices/0-3/key143 # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized144 145Now the device is connected (PCIe tunnels are created) and in addition146the key is stored on the device NVM.147 148Next time the device is plugged in the user can verify (challenge) the149device using the same key::150 151 # echo $key > /sys/bus/thunderbolt/devices/0-3/key152 # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized153 154If the challenge the device returns back matches the one we expect based155on the key, the device is connected and the PCIe tunnels are created.156However, if the challenge fails no tunnels are created and error is157returned to the user.158 159If the user still wants to connect the device they can either approve160the device without a key or write a new key and write 1 to the161``authorized`` file to get the new key stored on the device NVM.162 163De-authorizing devices164----------------------165It is possible to de-authorize devices by writing ``0`` to their166``authorized`` attribute. This requires support from the connection167manager implementation and can be checked by reading domain168``deauthorization`` attribute. If it reads ``1`` then the feature is169supported.170 171When a device is de-authorized the PCIe tunnel from the parent device172PCIe downstream (or root) port to the device PCIe upstream port is torn173down. This is essentially the same thing as PCIe hot-remove and the PCIe174toplogy in question will not be accessible anymore until the device is175authorized again. If there is storage such as NVMe or similar involved,176there is a risk for data loss if the filesystem on that storage is not177properly shut down. You have been warned!178 179DMA protection utilizing IOMMU180------------------------------181Recent systems from 2018 and forward with Thunderbolt ports may natively182support IOMMU. This means that Thunderbolt security is handled by an IOMMU183so connected devices cannot access memory regions outside of what is184allocated for them by drivers. When Linux is running on such system it185automatically enables IOMMU if not enabled by the user already. These186systems can be identified by reading ``1`` from187``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute.188 189The driver does not do anything special in this case but because DMA190protection is handled by the IOMMU, security levels (if set) are191redundant. For this reason some systems ship with security level set to192``none``. Other systems have security level set to ``user`` in order to193support downgrade to older OS, so users who want to automatically194authorize devices when IOMMU DMA protection is enabled can use the195following ``udev`` rule::196 197 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1"198 199Upgrading NVM on Thunderbolt device, host or retimer200----------------------------------------------------201Since most of the functionality is handled in firmware running on a202host controller or a device, it is important that the firmware can be203upgraded to the latest where possible bugs in it have been fixed.204Typically OEMs provide this firmware from their support site.205 206There is also a central site which has links where to download firmware207for some machines:208 209 `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_210 211Before you upgrade firmware on a device, host or retimer, please make212sure it is a suitable upgrade. Failing to do that may render the device213in a state where it cannot be used properly anymore without special214tools!215 216Host NVM upgrade on Apple Macs is not supported.217 218Once the NVM image has been downloaded, you need to plug in a219Thunderbolt device so that the host controller appears. It does not220matter which device is connected (unless you are upgrading NVM on a221device - then you need to connect that particular device).222 223Note an OEM-specific method to power the controller up ("force power") may224be available for your system in which case there is no need to plug in a225Thunderbolt device.226 227After that we can write the firmware to the non-active parts of the NVM228of the host or device. As an example here is how Intel NUC6i7KYK (Skull229Canyon) Thunderbolt controller NVM is upgraded::230 231 # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem232 233Once the operation completes we can trigger NVM authentication and234upgrade process as follows::235 236 # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate237 238If no errors are returned, the host controller shortly disappears. Once239it comes back the driver notices it and initiates a full power cycle.240After a while the host controller appears again and this time it should241be fully functional.242 243We can verify that the new NVM firmware is active by running the following244commands::245 246 # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate247 0x0248 # cat /sys/bus/thunderbolt/devices/0-0/nvm_version249 18.0250 251If ``nvm_authenticate`` contains anything other than 0x0 it is the error252code from the last authentication cycle, which means the authentication253of the NVM image failed.254 255Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``256depend on the order they are registered in the NVMem subsystem. N in257the name is the identifier added by the NVMem subsystem.258 259Upgrading on-board retimer NVM when there is no cable connected260---------------------------------------------------------------261If the platform supports, it may be possible to upgrade the retimer NVM262firmware even when there is nothing connected to the USB4263ports. When this is the case the ``usb4_portX`` devices have two special264attributes: ``offline`` and ``rescan``. The way to upgrade the firmware265is to first put the USB4 port into offline mode::266 267 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline268 269This step makes sure the port does not respond to any hotplug events,270and also ensures the retimers are powered on. The next step is to scan271for the retimers::272 273 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan274 275This enumerates and adds the on-board retimers. Now retimer NVM can be276upgraded in the same way than with cable connected (see previous277section). However, the retimer is not disconnected as we are offline278mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for2795 or more seconds before running rescan again::280 281 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan282 283This point if everything went fine, the port can be put back to284functional state again::285 286 # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline287 288Upgrading NVM when host controller is in safe mode289--------------------------------------------------290If the existing NVM is not properly authenticated (or is missing) the291host controller goes into safe mode which means that the only available292functionality is flashing a new NVM image. When in this mode, reading293``nvm_version`` fails with ``ENODATA`` and the device identification294information is missing.295 296To recover from this mode, one needs to flash a valid NVM image to the297host controller in the same way it is done in the previous chapter.298 299Networking over Thunderbolt cable300---------------------------------301Thunderbolt technology allows software communication between two hosts302connected by a Thunderbolt cable.303 304It is possible to tunnel any kind of traffic over a Thunderbolt link but305currently we only support Apple ThunderboltIP protocol.306 307If the other host is running Windows or macOS, the only thing you need to308do is to connect a Thunderbolt cable between the two hosts; the309``thunderbolt-net`` driver is loaded automatically. If the other host is310also Linux you should load ``thunderbolt-net`` manually on one host (it311does not matter which one)::312 313 # modprobe thunderbolt-net314 315This triggers module load on the other host automatically. If the driver316is built-in to the kernel image, there is no need to do anything.317 318The driver will create one virtual ethernet interface per Thunderbolt319port which are named like ``thunderbolt0`` and so on. From this point320you can either use standard userspace tools like ``ifconfig`` to321configure the interface or let your GUI handle it automatically.322 323Forcing power324-------------325Many OEMs include a method that can be used to force the power of a326Thunderbolt controller to an "On" state even if nothing is connected.327If supported by your machine this will be exposed by the WMI bus with328a sysfs attribute called "force_power".329 330For example the intel-wmi-thunderbolt driver exposes this attribute in:331 /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power332 333 To force the power to on, write 1 to this attribute file.334 To disable force power, write 0 to this attribute file.335 336Note: it's currently not possible to query the force power state of a platform.337