305 lines · plain
1=============================2Advanced Build Configurations3=============================4 5.. contents::6 :local:7 8Introduction9============10 11`CMake <http://www.cmake.org/>`_ is a cross-platform build-generator tool. CMake12does not build the project, it generates the files needed by your build tool13(GNU make, Visual Studio, etc.) for building LLVM.14 15If **you are a new contributor**, please start with the :doc:`GettingStarted` or16:doc:`CMake` pages. This page is intended for users doing more complex builds.17 18Many of the examples below are written assuming specific CMake Generators.19Unless explicitly stated otherwise, these commands should work with any CMake20generator.21 22Many of the build configurations mentioned on this documentation page can be23utilized by using a CMake cache. A CMake cache is essentially a configuration24file that sets the necessary flags for a specific build configuration. The caches25for Clang are located in :code:`/clang/cmake/caches` within the monorepo. They26can be passed to CMake using the :code:`-C` flag as demonstrated in the examples27below along with additional configuration flags.28 29Bootstrap Builds30================31 32The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a33high level, a multi-stage build is a chain of builds that pass data from one34stage into the next. The most common and simple version of this is a traditional35bootstrap build.36 37In a simple two-stage bootstrap build, we build clang using the system compiler,38then use that just-built clang to build clang again. In CMake this simplest form39of a bootstrap build can be configured with a single option,40``CLANG_ENABLE_BOOTSTRAP``.41 42.. code-block:: console43 44 $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \45 -DCLANG_ENABLE_BOOTSTRAP=On \46 -DLLVM_ENABLE_PROJECTS="clang" \47 <path to source>/llvm48 $ ninja stage249 50This command itself isn't terribly useful because it assumes default51configurations for each stage. The next series of examples utilize CMake cache52scripts to provide more complex options.53 54By default, only a few CMake options will be passed between stages.55The list, called _BOOTSTRAP_DEFAULT_PASSTHROUGH, is defined in ``clang/CMakeLists.txt``.56To force the passing of the variables between stages, use the ``-DCLANG_BOOTSTRAP_PASSTHROUGH``57CMake option, each variable separated by a ";". For example:58 59.. code-block:: console60 61 $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \62 -DCLANG_ENABLE_BOOTSTRAP=On \63 -DCLANG_BOOTSTRAP_PASSTHROUGH="CMAKE_INSTALL_PREFIX;CMAKE_VERBOSE_MAKEFILE" \64 -DLLVM_ENABLE_PROJECTS="clang" \65 <path to source>/llvm66 $ ninja stage267 68CMake options starting with ``BOOTSTRAP_`` will be passed only to the stage2 build.69This gives the opportunity to use Clang-specific build flags.70For example, the following CMake call will enable ``-fno-addrsig`` only during71the stage2 build for C and C++.72 73.. code-block:: console74 75 $ cmake [..] -DBOOTSTRAP_CMAKE_CXX_FLAGS='-fno-addrsig' -DBOOTSTRAP_CMAKE_C_FLAGS='-fno-addrsig' [..]76 77The clang build system refers to builds as stages. A stage1 build is a standard78build using the compiler installed on the host, and a stage2 build is built79using the stage1 compiler. This nomenclature holds up to more stages too. In80general, a stage*n* build is built using the output from stage*n-1*.81 82Apple Clang Builds (A More Complex Bootstrap)83=============================================84 85Apple's Clang builds are a slightly more complicated example of the simple86bootstrapping scenario. Apple Clang is built using a 2-stage build.87 88The stage1 compiler is a host-only compiler with some options set. The stage189compiler is a balance of optimization vs build time because it is a throwaway.90The stage2 compiler is the fully optimized compiler intended to ship to users.91 92Setting up these compilers requires a lot of options. To simplify the93configuration, the Apple Clang build settings are contained in CMake Cache files.94You can build an Apple Clang compiler using the following commands:95 96.. code-block:: console97 98 $ cmake -G Ninja -C <path to source>/clang/cmake/caches/Apple-stage1.cmake <path to source>/llvm99 $ ninja stage2-distribution100 101This CMake invocation configures the stage1 host compiler, and sets102``CLANG_BOOTSTRAP_CMAKE_ARGS`` to pass the Apple-stage2.cmake cache script to the103stage2 configuration step.104 105When you build the stage2-distribution target it builds the minimal stage1106compiler and required tools, then configures and builds the stage2 compiler107based on the settings in Apple-stage2.cmake.108 109This pattern of using cache scripts to set complex settings, and specifically to110make later stage builds include cache scripts is common in our more advanced111build configurations.112 113Multi-stage PGO114===============115 116Profile-Guided Optimization (PGO) is a really great way to optimize the code117clang generates. Our multi-stage PGO builds are a workflow for generating PGO118profiles that can be used to optimize clang.119 120At a high level, the way PGO works is that you build an instrumented compiler,121then you run the instrumented compiler against sample source files. While the122instrumented compiler runs it will output a bunch of files containing123performance counters (``.profraw`` files). After generating all the profraw files124you use llvm-profdata to merge the files into a single profdata file that you125can feed into the ``LLVM_PROFDATA_FILE`` option.126 127Our ``PGO.cmake`` cache automates that whole process. You can use it for128configuration with CMake with the following command:129 130.. code-block:: console131 132 $ cmake -G Ninja -C <path to source>/clang/cmake/caches/PGO.cmake \133 <path to source>/llvm134 135There are several additional options that the cache file also accepts to modify136the build, particularly the ``PGO_INSTRUMENT_LTO`` option. Setting this option to137Thin or Full will enable ThinLTO or full LTO respectively, further enhancing138the performance gains from a PGO build by enabling interprocedural139optimizations. For example, to run a CMake configuration for a PGO build140that also enables ThinLTO, use the following command:141 142.. code-block:: console143 144 $ cmake -G Ninja -C <path to source>/clang/cmake/caches/PGO.cmake \145 -DPGO_INSTRUMENT_LTO=Thin \146 <path to source>/llvm147 148By default, clang will generate profile data by compiling a simple149hello world program. You can also tell clang to use an external150project for generating profile data that may be a better fit for your151use case. The project you specify must either be a lit test suite152(use the ``CLANG_PGO_TRAINING_DATA`` option) or a CMake project (use the153``CLANG_PERF_TRAINING_DATA_SOURCE_DIR`` option).154 155For example, if you wanted to use the156`LLVM Test Suite <https://github.com/llvm/llvm-test-suite/>`_ to generate157profile data you would use the following command:158 159.. code-block:: console160 161 $ cmake -G Ninja -C <path to source>/clang/cmake/caches/PGO.cmake \162 -DBOOTSTRAP_CLANG_PGO_TRAINING_DATA_SOURCE_DIR=<path to llvm-test-suite> \163 -DBOOTSTRAP_CLANG_PGO_TRAINING_DEPS=runtimes164 165The ``BOOTSTRAP\_`` prefix tells CMake to pass the variables on to the instrumented166stage two build. And the ``CLANG_PGO_TRAINING_DEPS`` option lets you specify167additional build targets to build before building the external project. The168LLVM Test Suite requires compiler-rt to build, so we need to add the169`runtimes` target as a dependency.170 171After configuration, building the stage2-instrumented-generate-profdata target172will automatically build the stage1 compiler, build the instrumented compiler173with the stage1 compiler, and then run the instrumented compiler against the174perf training data:175 176.. code-block:: console177 178 $ ninja stage2-instrumented-generate-profdata179 180If you let that run for a few hours or so, it will place a profdata file in your181build directory. This takes a really long time because it builds clang twice,182and you *must* have compiler-rt in your build tree.183 184This process uses any source files under the perf-training directory as training185data as long as the source files are marked up with LIT-style ``RUN`` lines.186 187After it finishes you can use :code:`find . -name clang.profdata` to find it, but it188should be at a path something like:189 190.. code-block:: console191 192 <build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata193 194You can feed that file into the ``LLVM_PROFDATA_FILE`` option when you build your195optimized compiler.196 197It may be necessary to build additional targets before running perf training, such as198builtins and runtime libraries. You can use the :code:`CLANG_PGO_TRAINING_DEPS` CMake199variable for that purpose:200 201.. code-block:: cmake202 203 set(CLANG_PGO_TRAINING_DEPS builtins runtimes CACHE STRING "")204 205The PGO cache has a slightly different stage naming scheme than other206multi-stage builds. It generates three stages: stage1, stage2-instrumented, and207stage2. Both of the stage2 builds are built using the stage1 compiler.208 209The PGO cache generates the following additional targets:210 211**stage2-instrumented**212 Builds a stage1 compiler, runtime, and required tools (llvm-config,213 llvm-profdata) then uses that compiler to build an instrumented stage2 compiler.214 215**stage2-instrumented-generate-profdata**216 Depends on stage2-instrumented and will use the instrumented compiler to217 generate profdata based on the training files in ``clang/utils/perf-training``218 219**stage2**220 Depends on stage2-instrumented-generate-profdata and will use the stage1221 compiler with the stage2 profdata to build a PGO-optimized compiler.222 223**stage2-check-llvm**224 Depends on stage2 and runs check-llvm using the stage2 compiler.225 226**stage2-check-clang**227 Depends on stage2 and runs check-clang using the stage2 compiler.228 229**stage2-check-all**230 Depends on stage2 and runs check-all using the stage2 compiler.231 232**stage2-test-suite**233 Depends on stage2 and runs the test-suite using the stage2 compiler (requires234 in-tree test-suite).235 236BOLT237====238 239`BOLT <https://github.com/llvm/llvm-project/blob/main/bolt/README.md>`_240(Binary Optimization and Layout Tool) is a tool that optimizes binaries241post-link by profiling them at runtime and then using that information to242optimize the layout of the final binary among other optimizations performed243at the binary level. There are also CMake caches available to build244LLVM/Clang with BOLT.245 246To configure a single-stage build that builds LLVM/Clang and then optimizes247it with BOLT, use the following CMake configuration:248 249.. code-block:: console250 251 $ cmake <path to source>/llvm -C <path to source>/clang/cmake/caches/BOLT.cmake252 253Then, build the BOLT-optimized binary by running the following ninja command:254 255.. code-block:: console256 257 $ ninja clang-bolt258 259If you're seeing errors in the build process, try building with a recent260version of Clang/LLVM by setting the ``CMAKE_C_COMPILER`` and261``CMAKE_CXX_COMPILER`` flags to the appropriate values.262 263It is also possible to use BOLT on top of PGO and (Thin)LTO for an even more264significant runtime speedup. To configure a three-stage PGO build with ThinLTO265that optimizes the resulting binary with BOLT, use the following CMake266configuration command:267 268.. code-block:: console269 270 $ cmake -G Ninja <path to source>/llvm \271 -C <path to source>/clang/cmake/caches/BOLT-PGO.cmake \272 -DBOOTSTRAP_LLVM_ENABLE_LLD=ON \273 -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LLD=ON \274 -DPGO_INSTRUMENT_LTO=Thin275 276Then, to build the final optimized binary, build the stage2-clang-bolt target:277 278.. code-block:: console279 280 $ ninja stage2-clang-bolt281 2823-Stage Non-Determinism283=======================284 285In the ancient lore of compilers, non-determinism is like the multi-headed hydra.286Whenever its head pops up, terror and chaos ensue.287 288Historically, one of the tests to verify that a compiler was deterministic would289be a three-stage build. The idea of a three-stage build is that you take your sources290and build a compiler (stage1), then use that compiler to rebuild the sources291(stage2), then you use that compiler to rebuild the sources a third time292(stage3) with a configuration identical to the stage2 build. At the end of293this, you have a stage2 and stage3 compiler that should be bit-for-bit294identical.295 296You can perform one of these 3-stage builds with LLVM and clang using the297following commands:298 299.. code-block:: console300 301 $ cmake -G Ninja -C <path to source>/clang/cmake/caches/3-stage.cmake <path to source>/llvm302 $ ninja stage3303 304After the build, you can compare the stage2 and stage3 compilers.305