187 lines · plain
1===================2DTLTO3===================4.. contents::5 :local:6 :depth: 27 8.. toctree::9 :maxdepth: 110 11Distributed ThinLTO (DTLTO)12===========================13 14Distributed ThinLTO (DTLTO) enables the distribution of backend ThinLTO15compilations via external distribution systems, such as Incredibuild, during the16link step.17 18DTLTO extends the existing ThinLTO distribution support which uses separate19*thin-link*, *backend compilation*, and *link* steps. This method is documented20here:21 22 https://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html23 24Using the *separate thin-link* approach requires a build system capable of25handling the dynamic dependencies specified in the individual summary index26files, such as Bazel. DTLTO removes this requirement, allowing it to be used27with any build process that supports in-process ThinLTO.28 29The following commands show the steps used for the *separate thin-link*30approach for a basic example:31 32.. code-block:: console33 34 1. clang -flto=thin -O2 t1.c t2.c -c35 2. clang -flto=thin -O2 t1.o t2.o -fuse-ld=lld -Wl,--thinlto-index-only36 3. clang -O2 -o t1.native.o t1.o -c -fthinlto-index=t1.o.thinlto.bc37 4. clang -O2 -o t2.native.o t2.o -c -fthinlto-index=t2.o.thinlto.bc38 5. clang t1.native.o t2.native.o -o a.out -fuse-ld=lld39 40With DTLTO, steps 2-5 are performed internally as part of the link step. The41equivalent DTLTO commands for the above are:42 43.. code-block:: console44 45 clang -flto=thin -O2 t1.c t2.c -c46 clang -flto=thin -O2 t1.o t2.o -fuse-ld=lld -fthinlto-distributor=<distributor_process>47 48For DTLTO, LLD prepares the following for each ThinLTO backend compilation job:49 50- An individual index file and a list of input and output files (corresponds to51 step 2 above).52- A Clang command line to perform the ThinLTO backend compilations.53 54This information is supplied, via a JSON file, to ``distributor_process``, which55executes the backend compilations using a distribution system (corresponds to56steps 3 and 4 above). Upon completion, LLD integrates the compiled native object57files into the link process and completes the link (corresponds to step 558above).59 60This design keeps the details of distribution systems out of the LLVM source61code.62 63An example distributor that performs all work on the local system is included in64the LLVM source tree. To run an example with that distributor, a command line65such as the following can be used:66 67.. code-block:: console68 69 clang -flto=thin -fuse-ld=lld -O2 t1.o t2.o -fthinlto-distributor=$(which python3) \70 -Xthinlto-distributor=$LLVMSRC/llvm/utils/dtlto/local.py71 72Distributors73------------74 75Distributors are programs responsible for:76 771. Consuming the JSON backend compilations job description file.782. Translating job descriptions into requests for the distribution system.793. Blocking execution until all backend compilations are complete.80 81Distributors must return a non-zero exit code on failure. They can be82implemented as platform native executables or in a scripting language, such as83Python.84 85Clang and LLD provide options to specify a distributor program for managing86backend compilations. Distributor options and backend compilation options can87also be specified. Such options are transparently forwarded.88 89The backend compilations are currently performed by invoking Clang. For further90details, refer to:91 92* Clang documentation: https://clang.llvm.org/docs/ThinLTO.html93* LLD documentation: https://lld.llvm.org/DTLTO.html94 95When invoked with a distributor, LLD generates a JSON file describing the96backend compilation jobs and executes the distributor, passing it this file.97 98JSON Schema99-----------100 101The JSON format is explained by reference to the following example, which102describes the backend compilation of the modules ``t1.o`` and ``t2.o``:103 104.. code-block:: json105 106 {107 "common": {108 "linker_output": "dtlto.elf",109 "args": ["/usr/bin/clang", "-O2", "-c", "-fprofile-sample-use=my.prof"],110 "inputs": ["my.prof"]111 },112 "jobs": [113 {114 "args": ["t1.o", "-fthinlto-index=t1.o.thinlto.bc", "-o", "t1.native.o", "-fproc-stat-report=t1.stats.txt"],115 "inputs": ["t1.o", "t1.o.thinlto.bc"],116 "outputs": ["t1.native.o", "t1.stats.txt"]117 },118 {119 "args": ["t2.o", "-fthinlto-index=t2.o.thinlto.bc", "-o", "t2.native.o", "-fproc-stat-report=t2.stats.txt"],120 "inputs": ["t2.o", "t2.o.thinlto.bc"],121 "outputs": ["t2.native.o", "t2.stats.txt"]122 }123 ]124 }125 126Each entry in the ``jobs`` array represents a single backend compilation job.127Each job object records its own command-line arguments and input/output files.128Shared arguments and inputs are defined once in the ``common`` object.129 130Reserved Entries:131 132- The first entry in the ``common.args`` array specifies the compiler133 executable to invoke.134- The first entry in each job's ``inputs`` array is the bitcode file for the135 module being compiled.136- The second entry in each job's ``inputs`` array is the corresponding137 individual summary index file.138- The first entry in each job's ``outputs`` array is the primary output object139 file.140 141For the ``outputs`` array, only the first entry is reserved for the primary142output file; there is no guaranteed order for the remaining entries. The primary143output file is specified in a reserved entry because some distribution systems144rely on this path - for example, to provide a meaningful user label for145compilation jobs. Initially, the DTLTO implementation will not produce more than146one output file. However, in the future, if LTO options are added that imply147additional output files, those files will also be included in this array.148 149Command-line arguments and input/output files are stored separately to allow150the remote compiler to be changed without updating the distributors, as the151distributors do not need to understand the details of the compiler command152line.153 154To generate the backend compilation commands, the common and job-specific155arguments are concatenated.156 157When consuming the example JSON above, a distributor is expected to issue the158following backend compilation commands with maximum parallelism:159 160.. code-block:: console161 162 /usr/bin/clang -O2 -c -fprofile-sample-use=my.prof t1.o -fthinlto-index=t1.o.thinlto.bc -o t1.native.o \163 -fproc-stat-report=t1.stats.txt164 165 /usr/bin/clang -O2 -c -fprofile-sample-use=my.prof t2.o -fthinlto-index=t2.o.thinlto.bc -o t2.native.o \166 -fproc-stat-report=t2.stats.txt167 168TODOs169-----170 171The following features are planned for DTLTO but not yet implemented:172 173- Support for the ThinLTO in-process cache.174- Support for platforms other than ELF and COFF.175- Support for archives with bitcode members.176- Support for more LTO configurations; only a very limited set of LTO177 configurations is supported currently, e.g., support for basic block sections178 is not currently available.179 180Constraints181-----------182 183- Matching versions of Clang and LLD should be used.184- The distributor used must support the JSON schema generated by the version of185 LLD in use.186 187