156 lines · plain
1======================2Asymmetric 32-bit SoCs3======================4 5Author: Will Deacon <will@kernel.org>6 7This document describes the impact of asymmetric 32-bit SoCs on the8execution of 32-bit (``AArch32``) applications.9 10Date: 2021-05-1711 12Introduction13============14 15Some Armv9 SoCs suffer from a big.LITTLE misfeature where only a subset16of the CPUs are capable of executing 32-bit user applications. On such17a system, Linux by default treats the asymmetry as a "mismatch" and18disables support for both the ``PER_LINUX32`` personality and19``execve(2)`` of 32-bit ELF binaries, with the latter returning20``-ENOEXEC``. If the mismatch is detected during late onlining of a2164-bit-only CPU, then the onlining operation fails and the new CPU is22unavailable for scheduling.23 24Surprisingly, these SoCs have been produced with the intention of25running legacy 32-bit binaries. Unsurprisingly, that doesn't work very26well with the default behaviour of Linux.27 28It seems inevitable that future SoCs will drop 32-bit support29altogether, so if you're stuck in the unenviable position of needing to30run 32-bit code on one of these transitionary platforms then you would31be wise to consider alternatives such as recompilation, emulation or32retirement. If neither of those options are practical, then read on.33 34Enabling kernel support35=======================36 37Since the kernel support is not completely transparent to userspace,38allowing 32-bit tasks to run on an asymmetric 32-bit system requires an39explicit "opt-in" and can be enabled by passing the40``allow_mismatched_32bit_el0`` parameter on the kernel command-line.41 42For the remainder of this document we will refer to an *asymmetric43system* to mean an asymmetric 32-bit SoC running Linux with this kernel44command-line option enabled.45 46Userspace impact47================48 4932-bit tasks running on an asymmetric system behave in mostly the same50way as on a homogeneous system, with a few key differences relating to51CPU affinity.52 53sysfs54-----55 56The subset of CPUs capable of running 32-bit tasks is described in57``/sys/devices/system/cpu/aarch32_el0`` and is documented further in58``Documentation/ABI/testing/sysfs-devices-system-cpu``.59 60**Note:** CPUs are advertised by this file as they are detected and so61late-onlining of 32-bit-capable CPUs can result in the file contents62being modified by the kernel at runtime. Once advertised, CPUs are never63removed from the file.64 65``execve(2)``66-------------67 68On a homogeneous system, the CPU affinity of a task is preserved across69``execve(2)``. This is not always possible on an asymmetric system,70specifically when the new program being executed is 32-bit yet the71affinity mask contains 64-bit-only CPUs. In this situation, the kernel72determines the new affinity mask as follows:73 74 1. If the 32-bit-capable subset of the affinity mask is not empty,75 then the affinity is restricted to that subset and the old affinity76 mask is saved. This saved mask is inherited over ``fork(2)`` and77 preserved across ``execve(2)`` of 32-bit programs.78 79 **Note:** This step does not apply to ``SCHED_DEADLINE`` tasks.80 See `SCHED_DEADLINE`_.81 82 2. Otherwise, the cpuset hierarchy of the task is walked until an83 ancestor is found containing at least one 32-bit-capable CPU. The84 affinity of the task is then changed to match the 32-bit-capable85 subset of the cpuset determined by the walk.86 87 3. On failure (i.e. out of memory), the affinity is changed to the set88 of all 32-bit-capable CPUs of which the kernel is aware.89 90A subsequent ``execve(2)`` of a 64-bit program by the 32-bit task will91invalidate the affinity mask saved in (1) and attempt to restore the CPU92affinity of the task using the saved mask if it was previously valid.93This restoration may fail due to intervening changes to the deadline94policy or cpuset hierarchy, in which case the ``execve(2)`` continues95with the affinity unchanged.96 97Calls to ``sched_setaffinity(2)`` for a 32-bit task will consider only98the 32-bit-capable CPUs of the requested affinity mask. On success, the99affinity for the task is updated and any saved mask from a prior100``execve(2)`` is invalidated.101 102``SCHED_DEADLINE``103------------------104 105Explicit admission of a 32-bit deadline task to the default root domain106(e.g. by calling ``sched_setattr(2)``) is rejected on an asymmetric10732-bit system unless admission control is disabled by writing -1 to108``/proc/sys/kernel/sched_rt_runtime_us``.109 110``execve(2)`` of a 32-bit program from a 64-bit deadline task will111return ``-ENOEXEC`` if the root domain for the task contains any11264-bit-only CPUs and admission control is enabled. Concurrent offlining113of 32-bit-capable CPUs may still necessitate the procedure described in114`execve(2)`_, in which case step (1) is skipped and a warning is115emitted on the console.116 117**Note:** It is recommended that a set of 32-bit-capable CPUs are placed118into a separate root domain if ``SCHED_DEADLINE`` is to be used with11932-bit tasks on an asymmetric system. Failure to do so is likely to120result in missed deadlines.121 122Cpusets123-------124 125The affinity of a 32-bit task on an asymmetric system may include CPUs126that are not explicitly allowed by the cpuset to which it is attached.127This can occur as a result of the following two situations:128 129 - A 64-bit task attached to a cpuset which allows only 64-bit CPUs130 executes a 32-bit program.131 132 - All of the 32-bit-capable CPUs allowed by a cpuset containing a133 32-bit task are offlined.134 135In both of these cases, the new affinity is calculated according to step136(2) of the process described in `execve(2)`_ and the cpuset hierarchy is137unchanged irrespective of the cgroup version.138 139CPU hotplug140-----------141 142On an asymmetric system, the first detected 32-bit-capable CPU is143prevented from being offlined by userspace and any such attempt will144return ``-EPERM``. Note that suspend is still permitted even if the145primary CPU (i.e. CPU 0) is 64-bit-only.146 147KVM148---149 150Although KVM will not advertise 32-bit EL0 support to any vCPUs on an151asymmetric system, a broken guest at EL1 could still attempt to execute15232-bit code at EL0. In this case, an exit from a vCPU thread in 32-bit153mode will return to host userspace with an ``exit_reason`` of154``KVM_EXIT_FAIL_ENTRY`` and will remain non-runnable until successfully155re-initialised by a subsequent ``KVM_ARM_VCPU_INIT`` operation.156