331 lines · plain
1===================================================================2How to Cross Compile Compiler-rt Builtins For Arm3===================================================================4 5Introduction6============7 8This document contains information about building and testing the builtins part9of compiler-rt for an Arm target, from an x86_64 Linux machine.10 11While this document concentrates on Arm and Linux, the general principles should12apply to other targets supported by compiler-rt. Further contributions for other13targets are welcome.14 15The instructions in this document depend on libraries and programs external to16LLVM. There are many ways to install and configure these dependencies, so you17may need to adapt the instructions here to fit your own situation.18 19Prerequisites20=============21 22In this use case, we will be using cmake on a Debian-based Linux system,23cross-compiling from an x86_64 host to a hard-float Armv7-A target. We will be24using as many of the LLVM tools as we can, but it is possible to use GNU25equivalents.26 27You will need:28 * A build of LLVM for the llvm-tools and LLVM CMake files.29 * A clang executable with support for the ``ARM`` target.30 * ``compiler-rt`` sources.31 * The ``qemu-arm`` user mode emulator.32 * An ``arm-linux-gnueabihf`` sysroot.33 34.. note::35 An existing sysroot is required because some of the builtins include C library36 headers and a sysroot is the easiest way to get those.37 38In this example, we will be using ``ninja`` as the build tool.39 40See https://compiler-rt.llvm.org/ for information about the dependencies41on clang and LLVM.42 43See https://llvm.org/docs/GettingStarted.html for information about obtaining44the source for LLVM and compiler-rt.45 46``qemu-arm`` should be available as a package for your Linux distribution.47 48The most complicated of the prerequisites to satisfy is the ``arm-linux-gnueabihf``49sysroot. In theory, it is possible to use the Linux distributions multiarch50support to fulfill the dependencies for building but unfortunately due to51``/usr/local/include`` being added some host includes are selected.52 53The easiest way to supply a sysroot is to download an ``arm-linux-gnueabihf``54toolchain from https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads.55 56Building compiler-rt builtins for Arm57=====================================58 59We will be doing a standalone build of compiler-rt. The command is shown below.60Shell variables are used to simplify some of the options::61 62 LLVM_TOOLCHAIN=<path-to-llvm-install>/63 TARGET_TRIPLE=arm-none-linux-gnueabihf64 GCC_TOOLCHAIN=<path-to-gcc-toolchain>65 SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc66 COMPILE_FLAGS="-march=armv7-a"67 68 cmake ../llvm-project/compiler-rt \69 -G Ninja \70 -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \71 -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \72 -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \73 -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \74 -DCMAKE_SYSROOT="${SYSROOT}" \75 -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \76 -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \77 -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \78 -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \79 -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \80 -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \81 -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \82 -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \83 -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \84 -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \85 -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \86 -DCOMPILER_RT_BUILD_BUILTINS=ON \87 -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \88 -DCOMPILER_RT_BUILD_MEMPROF=OFF \89 -DCOMPILER_RT_BUILD_PROFILE=OFF \90 -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \91 -DCOMPILER_RT_BUILD_SANITIZERS=OFF \92 -DCOMPILER_RT_BUILD_XRAY=OFF \93 -DCOMPILER_RT_BUILD_ORC=OFF \94 -DCOMPILER_RT_BUILD_CRT=OFF \95 -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \96 -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \97 -DCOMPILER_RT_INCLUDE_TESTS=ON \98 -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \99 -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} ${COMPILE_FLAGS} --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"100 101.. note::102 The command above also enables tests. Enabling tests is not required, more details103 in the testing section.104 105``CMAKE_<LANGUAGE>_<OPTION>`` options are set so that the correct ``--target``,106``--sysroot``, ``--gcc-toolchain`` and ``-march`` options will be given to the107compilers.108 109The combination of these settings needs to be enough to pass CMake's compiler110checks, compile compiler-rt and build the test cases.111 112The flags need to select:113 * The Arm target (``--target arm-none-linux-gnueabihf``)114 * The Arm architecture level (``-march=armv7-a``)115 * Whether to generate Arm (``-marm``, the default) or Thumb (``-mthumb``) instructions.116 117It is possible to pass all these flags to CMake using ``CMAKE_<LANGUAGE>_FLAGS``,118but the command above uses standard CMake options instead. If you need to119add flags that CMake cannot generate automatically, add them to120``CMAKE_<LANGUAGE>_FLAGS``.121 122When CMake has finished, build with Ninja::123 124 ninja builtins125 126Testing compiler-rt builtins using qemu-arm127===========================================128 129The following options are required to enable tests::130 131 -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \132 -DCOMPILER_RT_INCLUDE_TESTS=ON \133 -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \134 -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"135 136This tells compiler-rt that we want to run tests on ``qemu-arm``. If you do not137want to run tests, remove these options from the CMake command.138 139Note that ``COMPILER_RT_TEST_COMPILER_CFLAGS`` contains the equivalent of the140options CMake generated for us with the first command. We must pass them141manually here because standard options like ``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN``142do not apply here.143 144When CMake has finished, run the tests::145 146 ninja check-builtins147 148Troubleshooting149===============150 151The cmake try compile stage fails152---------------------------------153At an early stage cmake will attempt to compile and link a simple C program to154test if the toolchain is working.155 156This stage can often fail at link time if the ``--sysroot=``, ``--target``, or157``--gcc-toolchain=`` options are not passed to the compiler. Check the158``CMAKE_<LANGUAGE>_FLAGS`` and ``CMAKE_<LANGAUGE>_COMPILER_TARGET`` flags along159with any of the specific CMake sysroot and toolchain options.160 161It can be useful to build a simple example outside of cmake with your toolchain162to make sure it is working. For example::163 164 clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c165 166Clang uses the host header files167--------------------------------168On Debian-based systems, it is possible to install multiarch support for169``arm-linux-gnueabi`` and ``arm-linux-gnueabihf``. In many cases clang can successfully170use this multiarch support when ``--gcc-toolchain=`` and ``--sysroot=`` are not supplied.171Unfortunately clang adds ``/usr/local/include`` before172``/usr/include/arm-linux-gnueabihf`` leading to errors when compiling the hosts173header files.174 175The multiarch support is not sufficient to build the builtins you will need to176use a separate ``arm-linux-gnueabihf`` toolchain.177 178No target passed to clang179-------------------------180If clang is not given a target, it will typically use the host target. This will181not understand the Arm assembly language files, resulting in error messages such182as ``error: unknown directive .syntax unified``.183 184You can check the clang invocation in the error message to see if there is no185``--target`` or if it is set incorrectly. The cause is usually186``CMAKE_ASM_FLAGS`` not containing ``--target`` or ``CMAKE_ASM_COMPILER_TARGET``187not being present.188 189Arm architecture not given190--------------------------191The ``--target=arm-linux-gnueabihf`` will default to Arm architecture v4t which192cannot assemble the barrier instructions used in the ``synch_and_fetch`` source193files.194 195The cause is usually a missing ``-march=armv7a`` from the ``CMAKE_ASM_FLAGS``.196 197Compiler-rt builds but the tests fail to build198----------------------------------------------199The flags used to build the tests are not the same as those used to build the200builtins. The c flags are provided by ``COMPILER_RT_TEST_COMPILE_CFLAGS`` and201the ``CMAKE_C_COMPILER_TARGET``, ``CMAKE_ASM_COMPILER_TARGET``,202``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` and ``CMAKE_SYSROOT`` flags are not203applied to tests.204 205Make sure that ``COMPILER_RT_TEST_COMPILE_CFLAGS`` contains all the necessary206flags.207 208 209Modifications for other Targets210===============================211 212Arm Soft-Float Target213---------------------214The instructions for the Arm hard-float target can be used for the soft-float215target by substituting soft-float equivalents for the sysroot and target. The216target to use is:217 218* ``-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi``219 220Depending on whether you want to use floating point instructions or not, you221may need extra c-flags such as ``-mfloat-abi=softfp`` for use of floating-point222instructions, and ``-mfloat-abi=soft -mfpu=none`` for software floating-point223emulation.224 225You will need to use an ``arm-linux-gnueabi`` GNU toolchain for soft-float.226 227AArch64 Target228--------------229The instructions for Arm can be used for AArch64 by substituting AArch64230equivalents for the sysroot, emulator and target::231 232 -DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu233 -DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot234 235You will also have to update any use of the target triple in compiler flags.236For instance in ``CMAKE_C_FLAGS`` and ``COMPILER_RT_TEST_COMPILER_CFLAGS``.237 238Armv6-m, Armv7-m and Armv7E-M targets239-------------------------------------240To build and test the libraries using a similar method to Armv7-A is possible241but more difficult. The main problems are:242 243* There is not a ``qemu-arm`` user-mode emulator for bare-metal systems.244 ``qemu-system-arm`` can be used, but this is significantly more difficult245 to setup. This document does not explain how to do this.246* The targets to compile compiler-rt have the suffix ``-none-eabi``. This uses247 the BareMetal driver in clang and by default will not find the libraries248 needed to pass the cmake compiler check.249 250As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions251that are supported on Armv7-A we can still get most of the value of running the252tests using the same ``qemu-arm`` that we used for Armv7-A by building and253running the test cases for Armv7-A but using the builtins compiled for254Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked255into a binary and execute the tests correctly, but it will not catch if the256builtins use instructions that are supported on Armv7-A but not on Armv6-M,257Armv7-M and Armv7E-M.258 259This requires a second ``arm-none-eabi`` toolchain for building the builtins.260Using a bare-metal toolchain ensures that the target and C library details are261specific to bare-metal instead of using Linux settings. This means that some262tests may behave differently compared to real hardware, but at least the content263of the builtins library is correct.264 265Below is an example that builds the builtins for Armv7-M, but runs the tests266as Armv7-A. It is presented in full, but is very similar to the earlier267command for Armv7-A build and test::268 269 LLVM_TOOLCHAIN=<path to llvm install>/270 271 # For the builtins.272 TARGET_TRIPLE=arm-none-eabi273 GCC_TOOLCHAIN=<path to arm-none-eabi toolchain>/274 SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc275 COMPILE_FLAGS="-march=armv7-m -mfpu=vfpv2"276 277 # For the test cases.278 A_PROFILE_TARGET_TRIPLE=arm-none-linux-gnueabihf279 A_PROFILE_GCC_TOOLCHAIN=<path to arm-none-linux-gnueabihf toolchain>/280 A_PROFILE_SYSROOT=${A_PROFILE_GCC_TOOLCHAIN}/${A_PROFILE_TARGET_TRIPLE}/libc281 282 cmake ../llvm-project/compiler-rt \283 -G Ninja \284 -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \285 -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \286 -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \287 -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \288 -DCMAKE_SYSROOT="${SYSROOT}" \289 -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \290 -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \291 -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \292 -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \293 -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \294 -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \295 -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \296 -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \297 -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \298 -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \299 -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \300 -DCOMPILER_RT_BUILD_BUILTINS=ON \301 -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \302 -DCOMPILER_RT_BUILD_MEMPROF=OFF \303 -DCOMPILER_RT_BUILD_PROFILE=OFF \304 -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \305 -DCOMPILER_RT_BUILD_SANITIZERS=OFF \306 -DCOMPILER_RT_BUILD_XRAY=OFF \307 -DCOMPILER_RT_BUILD_ORC=OFF \308 -DCOMPILER_RT_BUILD_CRT=OFF \309 -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \310 -DCOMPILER_RT_EMULATOR="qemu-arm -L ${A_PROFILE_SYSROOT}" \311 -DCOMPILER_RT_INCLUDE_TESTS=ON \312 -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \313 -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${A_PROFILE_TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${A_PROFILE_GCC_TOOLCHAIN} --sysroot=${A_PROFILE_SYSROOT} -fuse-ld=lld" \314 -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \315 -DCOMPILER_RT_OS_DIR="baremetal" \316 -DCOMPILER_RT_BAREMETAL_BUILD=ON317 318.. note::319 The sysroot used for compiling the tests is ``arm-linux-gnueabihf``, not320 ``arm-none-eabi`` which is used when compiling the builtins.321 322The Armv6-M builtins will use the soft-float ABI. When compiling the tests for323Armv7-A we must include ``"-mthumb -mfloat-abi=soft -mfpu=none"`` in the324test-c-flags. We must use an Armv7-A soft-float ABI sysroot for ``qemu-arm``.325 326Depending on the linker used for the test cases, you may encounter BuildAttribute327mismatches between the M-profile objects from compiler-rt and the A-profile328objects from the test. The lld linker does not check the profile329BuildAttribute so it can be used to link the tests by adding ``-fuse-ld=lld`` to the330``COMPILER_RT_TEST_COMPILER_CFLAGS``.331