616 lines · plain
1L1TF - L1 Terminal Fault2========================3 4L1 Terminal Fault is a hardware vulnerability which allows unprivileged5speculative access to data which is available in the Level 1 Data Cache6when the page table entry controlling the virtual address, which is used7for the access, has the Present bit cleared or other reserved bits set.8 9Affected processors10-------------------11 12This vulnerability affects a wide range of Intel processors. The13vulnerability is not present on:14 15 - Processors from AMD, Centaur and other non Intel vendors16 17 - Older processor models, where the CPU family is < 618 19 - A range of Intel ATOM processors (Cedarview, Cloverview, Lincroft,20 Penwell, Pineview, Silvermont, Airmont, Merrifield)21 22 - The Intel XEON PHI family23 24 - Intel processors which have the ARCH_CAP_RDCL_NO bit set in the25 IA32_ARCH_CAPABILITIES MSR. If the bit is set the CPU is not affected26 by the Meltdown vulnerability either. These CPUs should become27 available by end of 2018.28 29Whether a processor is affected or not can be read out from the L1TF30vulnerability file in sysfs. See :ref:`l1tf_sys_info`.31 32Related CVEs33------------34 35The following CVE entries are related to the L1TF vulnerability:36 37 ============= ================= ==============================38 CVE-2018-3615 L1 Terminal Fault SGX related aspects39 CVE-2018-3620 L1 Terminal Fault OS, SMM related aspects40 CVE-2018-3646 L1 Terminal Fault Virtualization related aspects41 ============= ================= ==============================42 43Problem44-------45 46If an instruction accesses a virtual address for which the relevant page47table entry (PTE) has the Present bit cleared or other reserved bits set,48then speculative execution ignores the invalid PTE and loads the referenced49data if it is present in the Level 1 Data Cache, as if the page referenced50by the address bits in the PTE was still present and accessible.51 52While this is a purely speculative mechanism and the instruction will raise53a page fault when it is retired eventually, the pure act of loading the54data and making it available to other speculative instructions opens up the55opportunity for side channel attacks to unprivileged malicious code,56similar to the Meltdown attack.57 58While Meltdown breaks the user space to kernel space protection, L1TF59allows to attack any physical memory address in the system and the attack60works across all protection domains. It allows an attack of SGX and also61works from inside virtual machines because the speculation bypasses the62extended page table (EPT) protection mechanism.63 64 65Attack scenarios66----------------67 681. Malicious user space69^^^^^^^^^^^^^^^^^^^^^^^70 71 Operating Systems store arbitrary information in the address bits of a72 PTE which is marked non present. This allows a malicious user space73 application to attack the physical memory to which these PTEs resolve.74 In some cases user-space can maliciously influence the information75 encoded in the address bits of the PTE, thus making attacks more76 deterministic and more practical.77 78 The Linux kernel contains a mitigation for this attack vector, PTE79 inversion, which is permanently enabled and has no performance80 impact. The kernel ensures that the address bits of PTEs, which are not81 marked present, never point to cacheable physical memory space.82 83 A system with an up to date kernel is protected against attacks from84 malicious user space applications.85 862. Malicious guest in a virtual machine87^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^88 89 The fact that L1TF breaks all domain protections allows malicious guest90 OSes, which can control the PTEs directly, and malicious guest user91 space applications, which run on an unprotected guest kernel lacking the92 PTE inversion mitigation for L1TF, to attack physical host memory.93 94 A special aspect of L1TF in the context of virtualization is symmetric95 multi threading (SMT). The Intel implementation of SMT is called96 HyperThreading. The fact that Hyperthreads on the affected processors97 share the L1 Data Cache (L1D) is important for this. As the flaw allows98 only to attack data which is present in L1D, a malicious guest running99 on one Hyperthread can attack the data which is brought into the L1D by100 the context which runs on the sibling Hyperthread of the same physical101 core. This context can be host OS, host user space or a different guest.102 103 If the processor does not support Extended Page Tables, the attack is104 only possible, when the hypervisor does not sanitize the content of the105 effective (shadow) page tables.106 107 While solutions exist to mitigate these attack vectors fully, these108 mitigations are not enabled by default in the Linux kernel because they109 can affect performance significantly. The kernel provides several110 mechanisms which can be utilized to address the problem depending on the111 deployment scenario. The mitigations, their protection scope and impact112 are described in the next sections.113 114 The default mitigations and the rationale for choosing them are explained115 at the end of this document. See :ref:`default_mitigations`.116 117.. _l1tf_sys_info:118 119L1TF system information120-----------------------121 122The Linux kernel provides a sysfs interface to enumerate the current L1TF123status of the system: whether the system is vulnerable, and which124mitigations are active. The relevant sysfs file is:125 126/sys/devices/system/cpu/vulnerabilities/l1tf127 128The possible values in this file are:129 130 =========================== ===============================131 'Not affected' The processor is not vulnerable132 'Mitigation: PTE Inversion' The host protection is active133 =========================== ===============================134 135If KVM/VMX is enabled and the processor is vulnerable then the following136information is appended to the 'Mitigation: PTE Inversion' part:137 138 - SMT status:139 140 ===================== ================141 'VMX: SMT vulnerable' SMT is enabled142 'VMX: SMT disabled' SMT is disabled143 ===================== ================144 145 - L1D Flush mode:146 147 ================================ ====================================148 'L1D vulnerable' L1D flushing is disabled149 150 'L1D conditional cache flushes' L1D flush is conditionally enabled151 152 'L1D cache flushes' L1D flush is unconditionally enabled153 ================================ ====================================154 155The resulting grade of protection is discussed in the following sections.156 157 158Host mitigation mechanism159-------------------------160 161The kernel is unconditionally protected against L1TF attacks from malicious162user space running on the host.163 164 165Guest mitigation mechanisms166---------------------------167 168.. _l1d_flush:169 1701. L1D flush on VMENTER171^^^^^^^^^^^^^^^^^^^^^^^172 173 To make sure that a guest cannot attack data which is present in the L1D174 the hypervisor flushes the L1D before entering the guest.175 176 Flushing the L1D evicts not only the data which should not be accessed177 by a potentially malicious guest, it also flushes the guest178 data. Flushing the L1D has a performance impact as the processor has to179 bring the flushed guest data back into the L1D. Depending on the180 frequency of VMEXIT/VMENTER and the type of computations in the guest181 performance degradation in the range of 1% to 50% has been observed. For182 scenarios where guest VMEXIT/VMENTER are rare the performance impact is183 minimal. Virtio and mechanisms like posted interrupts are designed to184 confine the VMEXITs to a bare minimum, but specific configurations and185 application scenarios might still suffer from a high VMEXIT rate.186 187 The kernel provides two L1D flush modes:188 - conditional ('cond')189 - unconditional ('always')190 191 The conditional mode avoids L1D flushing after VMEXITs which execute192 only audited code paths before the corresponding VMENTER. These code193 paths have been verified that they cannot expose secrets or other194 interesting data to an attacker, but they can leak information about the195 address space layout of the hypervisor.196 197 Unconditional mode flushes L1D on all VMENTER invocations and provides198 maximum protection. It has a higher overhead than the conditional199 mode. The overhead cannot be quantified correctly as it depends on the200 workload scenario and the resulting number of VMEXITs.201 202 The general recommendation is to enable L1D flush on VMENTER. The kernel203 defaults to conditional mode on affected processors.204 205 **Note**, that L1D flush does not prevent the SMT problem because the206 sibling thread will also bring back its data into the L1D which makes it207 attackable again.208 209 L1D flush can be controlled by the administrator via the kernel command210 line and sysfs control files. See :ref:`mitigation_control_command_line`211 and :ref:`mitigation_control_kvm`.212 213.. _guest_confinement:214 2152. Guest VCPU confinement to dedicated physical cores216^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^217 218 To address the SMT problem, it is possible to make a guest or a group of219 guests affine to one or more physical cores. The proper mechanism for220 that is to utilize exclusive cpusets to ensure that no other guest or221 host tasks can run on these cores.222 223 If only a single guest or related guests run on sibling SMT threads on224 the same physical core then they can only attack their own memory and225 restricted parts of the host memory.226 227 Host memory is attackable, when one of the sibling SMT threads runs in228 host OS (hypervisor) context and the other in guest context. The amount229 of valuable information from the host OS context depends on the context230 which the host OS executes, i.e. interrupts, soft interrupts and kernel231 threads. The amount of valuable data from these contexts cannot be232 declared as non-interesting for an attacker without deep inspection of233 the code.234 235 **Note**, that assigning guests to a fixed set of physical cores affects236 the ability of the scheduler to do load balancing and might have237 negative effects on CPU utilization depending on the hosting238 scenario. Disabling SMT might be a viable alternative for particular239 scenarios.240 241 For further information about confining guests to a single or to a group242 of cores consult the cpusets documentation:243 244 https://www.kernel.org/doc/Documentation/admin-guide/cgroup-v1/cpusets.rst245 246.. _interrupt_isolation:247 2483. Interrupt affinity249^^^^^^^^^^^^^^^^^^^^^250 251 Interrupts can be made affine to logical CPUs. This is not universally252 true because there are types of interrupts which are truly per CPU253 interrupts, e.g. the local timer interrupt. Aside of that multi queue254 devices affine their interrupts to single CPUs or groups of CPUs per255 queue without allowing the administrator to control the affinities.256 257 Moving the interrupts, which can be affinity controlled, away from CPUs258 which run untrusted guests, reduces the attack vector space.259 260 Whether the interrupts with are affine to CPUs, which run untrusted261 guests, provide interesting data for an attacker depends on the system262 configuration and the scenarios which run on the system. While for some263 of the interrupts it can be assumed that they won't expose interesting264 information beyond exposing hints about the host OS memory layout, there265 is no way to make general assumptions.266 267 Interrupt affinity can be controlled by the administrator via the268 /proc/irq/$NR/smp_affinity[_list] files. Limited documentation is269 available at:270 271 https://www.kernel.org/doc/Documentation/core-api/irq/irq-affinity.rst272 273.. _smt_control:274 2754. SMT control276^^^^^^^^^^^^^^277 278 To prevent the SMT issues of L1TF it might be necessary to disable SMT279 completely. Disabling SMT can have a significant performance impact, but280 the impact depends on the hosting scenario and the type of workloads.281 The impact of disabling SMT needs also to be weighted against the impact282 of other mitigation solutions like confining guests to dedicated cores.283 284 The kernel provides a sysfs interface to retrieve the status of SMT and285 to control it. It also provides a kernel command line interface to286 control SMT.287 288 The kernel command line interface consists of the following options:289 290 =========== ==========================================================291 nosmt Affects the bring up of the secondary CPUs during boot. The292 kernel tries to bring all present CPUs online during the293 boot process. "nosmt" makes sure that from each physical294 core only one - the so called primary (hyper) thread is295 activated. Due to a design flaw of Intel processors related296 to Machine Check Exceptions the non primary siblings have297 to be brought up at least partially and are then shut down298 again. "nosmt" can be undone via the sysfs interface.299 300 nosmt=force Has the same effect as "nosmt" but it does not allow to301 undo the SMT disable via the sysfs interface.302 =========== ==========================================================303 304 The sysfs interface provides two files:305 306 - /sys/devices/system/cpu/smt/control307 - /sys/devices/system/cpu/smt/active308 309 /sys/devices/system/cpu/smt/control:310 311 This file allows to read out the SMT control state and provides the312 ability to disable or (re)enable SMT. The possible states are:313 314 ============== ===================================================315 on SMT is supported by the CPU and enabled. All316 logical CPUs can be onlined and offlined without317 restrictions.318 319 off SMT is supported by the CPU and disabled. Only320 the so called primary SMT threads can be onlined321 and offlined without restrictions. An attempt to322 online a non-primary sibling is rejected323 324 forceoff Same as 'off' but the state cannot be controlled.325 Attempts to write to the control file are rejected.326 327 notsupported The processor does not support SMT. It's therefore328 not affected by the SMT implications of L1TF.329 Attempts to write to the control file are rejected.330 ============== ===================================================331 332 The possible states which can be written into this file to control SMT333 state are:334 335 - on336 - off337 - forceoff338 339 /sys/devices/system/cpu/smt/active:340 341 This file reports whether SMT is enabled and active, i.e. if on any342 physical core two or more sibling threads are online.343 344 SMT control is also possible at boot time via the l1tf kernel command345 line parameter in combination with L1D flush control. See346 :ref:`mitigation_control_command_line`.347 3485. Disabling EPT349^^^^^^^^^^^^^^^^350 351 Disabling EPT for virtual machines provides full mitigation for L1TF even352 with SMT enabled, because the effective page tables for guests are353 managed and sanitized by the hypervisor. Though disabling EPT has a354 significant performance impact especially when the Meltdown mitigation355 KPTI is enabled.356 357 EPT can be disabled in the hypervisor via the 'kvm-intel.ept' parameter.358 359There is ongoing research and development for new mitigation mechanisms to360address the performance impact of disabling SMT or EPT.361 362.. _mitigation_control_command_line:363 364Mitigation control on the kernel command line365---------------------------------------------366 367The kernel command line allows to control the L1TF mitigations at boot368time with the option "l1tf=". The valid arguments for this option are:369 370 ============ =============================================================371 full Provides all available mitigations for the L1TF372 vulnerability. Disables SMT and enables all mitigations in373 the hypervisors, i.e. unconditional L1D flushing374 375 SMT control and L1D flush control via the sysfs interface376 is still possible after boot. Hypervisors will issue a377 warning when the first VM is started in a potentially378 insecure configuration, i.e. SMT enabled or L1D flush379 disabled.380 381 full,force Same as 'full', but disables SMT and L1D flush runtime382 control. Implies the 'nosmt=force' command line option.383 (i.e. sysfs control of SMT is disabled.)384 385 flush Leaves SMT enabled and enables the default hypervisor386 mitigation, i.e. conditional L1D flushing387 388 SMT control and L1D flush control via the sysfs interface389 is still possible after boot. Hypervisors will issue a390 warning when the first VM is started in a potentially391 insecure configuration, i.e. SMT enabled or L1D flush392 disabled.393 394 flush,nosmt Disables SMT and enables the default hypervisor mitigation,395 i.e. conditional L1D flushing.396 397 SMT control and L1D flush control via the sysfs interface398 is still possible after boot. Hypervisors will issue a399 warning when the first VM is started in a potentially400 insecure configuration, i.e. SMT enabled or L1D flush401 disabled.402 403 flush,nowarn Same as 'flush', but hypervisors will not warn when a VM is404 started in a potentially insecure configuration.405 406 off Disables hypervisor mitigations and doesn't emit any407 warnings.408 It also drops the swap size and available RAM limit restrictions409 on both hypervisor and bare metal.410 411 ============ =============================================================412 413The default is 'flush'. For details about L1D flushing see :ref:`l1d_flush`.414 415 416.. _mitigation_control_kvm:417 418Mitigation control for KVM - module parameter419-------------------------------------------------------------420 421The KVM hypervisor mitigation mechanism, flushing the L1D cache when422entering a guest, can be controlled with a module parameter.423 424The option/parameter is "kvm-intel.vmentry_l1d_flush=". It takes the425following arguments:426 427 ============ ==============================================================428 always L1D cache flush on every VMENTER.429 430 cond Flush L1D on VMENTER only when the code between VMEXIT and431 VMENTER can leak host memory which is considered432 interesting for an attacker. This still can leak host memory433 which allows e.g. to determine the hosts address space layout.434 435 never Disables the mitigation436 ============ ==============================================================437 438The parameter can be provided on the kernel command line, as a module439parameter when loading the modules and at runtime modified via the sysfs440file:441 442/sys/module/kvm_intel/parameters/vmentry_l1d_flush443 444The default is 'cond'. If 'l1tf=full,force' is given on the kernel command445line, then 'always' is enforced and the kvm-intel.vmentry_l1d_flush446module parameter is ignored and writes to the sysfs file are rejected.447 448.. _mitigation_selection:449 450Mitigation selection guide451--------------------------452 4531. No virtualization in use454^^^^^^^^^^^^^^^^^^^^^^^^^^^455 456 The system is protected by the kernel unconditionally and no further457 action is required.458 4592. Virtualization with trusted guests460^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^461 462 If the guest comes from a trusted source and the guest OS kernel is463 guaranteed to have the L1TF mitigations in place the system is fully464 protected against L1TF and no further action is required.465 466 To avoid the overhead of the default L1D flushing on VMENTER the467 administrator can disable the flushing via the kernel command line and468 sysfs control files. See :ref:`mitigation_control_command_line` and469 :ref:`mitigation_control_kvm`.470 471 4723. Virtualization with untrusted guests473^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^474 4753.1. SMT not supported or disabled476""""""""""""""""""""""""""""""""""477 478 If SMT is not supported by the processor or disabled in the BIOS or by479 the kernel, it's only required to enforce L1D flushing on VMENTER.480 481 Conditional L1D flushing is the default behaviour and can be tuned. See482 :ref:`mitigation_control_command_line` and :ref:`mitigation_control_kvm`.483 4843.2. EPT not supported or disabled485""""""""""""""""""""""""""""""""""486 487 If EPT is not supported by the processor or disabled in the hypervisor,488 the system is fully protected. SMT can stay enabled and L1D flushing on489 VMENTER is not required.490 491 EPT can be disabled in the hypervisor via the 'kvm-intel.ept' parameter.492 4933.3. SMT and EPT supported and active494"""""""""""""""""""""""""""""""""""""495 496 If SMT and EPT are supported and active then various degrees of497 mitigations can be employed:498 499 - L1D flushing on VMENTER:500 501 L1D flushing on VMENTER is the minimal protection requirement, but it502 is only potent in combination with other mitigation methods.503 504 Conditional L1D flushing is the default behaviour and can be tuned. See505 :ref:`mitigation_control_command_line` and :ref:`mitigation_control_kvm`.506 507 - Guest confinement:508 509 Confinement of guests to a single or a group of physical cores which510 are not running any other processes, can reduce the attack surface511 significantly, but interrupts, soft interrupts and kernel threads can512 still expose valuable data to a potential attacker. See513 :ref:`guest_confinement`.514 515 - Interrupt isolation:516 517 Isolating the guest CPUs from interrupts can reduce the attack surface518 further, but still allows a malicious guest to explore a limited amount519 of host physical memory. This can at least be used to gain knowledge520 about the host address space layout. The interrupts which have a fixed521 affinity to the CPUs which run the untrusted guests can depending on522 the scenario still trigger soft interrupts and schedule kernel threads523 which might expose valuable information. See524 :ref:`interrupt_isolation`.525 526The above three mitigation methods combined can provide protection to a527certain degree, but the risk of the remaining attack surface has to be528carefully analyzed. For full protection the following methods are529available:530 531 - Disabling SMT:532 533 Disabling SMT and enforcing the L1D flushing provides the maximum534 amount of protection. This mitigation is not depending on any of the535 above mitigation methods.536 537 SMT control and L1D flushing can be tuned by the command line538 parameters 'nosmt', 'l1tf', 'kvm-intel.vmentry_l1d_flush' and at run539 time with the matching sysfs control files. See :ref:`smt_control`,540 :ref:`mitigation_control_command_line` and541 :ref:`mitigation_control_kvm`.542 543 - Disabling EPT:544 545 Disabling EPT provides the maximum amount of protection as well. It is546 not depending on any of the above mitigation methods. SMT can stay547 enabled and L1D flushing is not required, but the performance impact is548 significant.549 550 EPT can be disabled in the hypervisor via the 'kvm-intel.ept'551 parameter.552 5533.4. Nested virtual machines554""""""""""""""""""""""""""""555 556When nested virtualization is in use, three operating systems are involved:557the bare metal hypervisor, the nested hypervisor and the nested virtual558machine. VMENTER operations from the nested hypervisor into the nested559guest will always be processed by the bare metal hypervisor. If KVM is the560bare metal hypervisor it will:561 562 - Flush the L1D cache on every switch from the nested hypervisor to the563 nested virtual machine, so that the nested hypervisor's secrets are not564 exposed to the nested virtual machine;565 566 - Flush the L1D cache on every switch from the nested virtual machine to567 the nested hypervisor; this is a complex operation, and flushing the L1D568 cache avoids that the bare metal hypervisor's secrets are exposed to the569 nested virtual machine;570 571 - Instruct the nested hypervisor to not perform any L1D cache flush. This572 is an optimization to avoid double L1D flushing.573 574 575.. _default_mitigations:576 577Default mitigations578-------------------579 580 The kernel default mitigations for vulnerable processors are:581 582 - PTE inversion to protect against malicious user space. This is done583 unconditionally and cannot be controlled. The swap storage is limited584 to ~16TB.585 586 - L1D conditional flushing on VMENTER when EPT is enabled for587 a guest.588 589 The kernel does not by default enforce the disabling of SMT, which leaves590 SMT systems vulnerable when running untrusted guests with EPT enabled.591 592 The rationale for this choice is:593 594 - Force disabling SMT can break existing setups, especially with595 unattended updates.596 597 - If regular users run untrusted guests on their machine, then L1TF is598 just an add on to other malware which might be embedded in an untrusted599 guest, e.g. spam-bots or attacks on the local network.600 601 There is no technical way to prevent a user from running untrusted code602 on their machines blindly.603 604 - It's technically extremely unlikely and from today's knowledge even605 impossible that L1TF can be exploited via the most popular attack606 mechanisms like JavaScript because these mechanisms have no way to607 control PTEs. If this would be possible and not other mitigation would608 be possible, then the default might be different.609 610 - The administrators of cloud and hosting setups have to carefully611 analyze the risk for their scenarios and make the appropriate612 mitigation choices, which might even vary across their deployed613 machines and also result in other changes of their overall setup.614 There is no way for the kernel to provide a sensible default for this615 kind of scenarios.616