401 lines · plain
1=========================2Driver Design & Internals3=========================4 5.. contents::6 :local:7 8Introduction9============10 11This document describes the Clang driver. The purpose of this document12is to describe both the motivation and design goals for the driver, as13well as details of the internal implementation.14 15Features and Goals16==================17 18The Clang driver is intended to be a production quality compiler driver19providing access to the Clang compiler and tools, with a command line20interface which is compatible with the gcc driver.21 22Although the driver is part of and driven by the Clang project, it is23logically a separate tool which shares many of the same goals as Clang:24 25.. contents:: Features26 :local:27 28GCC Compatibility29-----------------30 31The number one goal of the driver is to ease the adoption of Clang by32allowing users to drop Clang into a build system which was designed to33call GCC. Although this makes the driver much more complicated than34might otherwise be necessary, we decided that being very compatible with35the gcc command line interface was worth it in order to allow users to36quickly test clang on their projects.37 38Flexible39--------40 41The driver was designed to be flexible and easily accommodate new uses42as we grow the clang and LLVM infrastructure. As one example, the driver43can easily support the introduction of tools which have an integrated44assembler; something we hope to add to LLVM in the future.45 46Similarly, most of the driver functionality is kept in a library which47can be used to build other tools which want to implement or accept a gcc48like interface.49 50Low Overhead51------------52 53The driver should have as little overhead as possible. In practice, we54found that the gcc driver by itself incurred a small but meaningful55overhead when compiling many small files. The driver doesn't do much56work compared to a compilation, but we have tried to keep it as57efficient as possible by following a few simple principles:58 59- Avoid memory allocation and string copying when possible.60- Don't parse arguments more than once.61- Provide a few simple interfaces for efficiently searching arguments.62 63Simple64------65 66Finally, the driver was designed to be "as simple as possible", given67the other goals. Notably, trying to be completely compatible with the68gcc driver adds a significant amount of complexity. However, the design69of the driver attempts to mitigate this complexity by dividing the70process into a number of independent stages instead of a single71monolithic task.72 73Internal Design and Implementation74==================================75 76.. contents::77 :local:78 :depth: 179 80Internals Introduction81----------------------82 83In order to satisfy the stated goals, the driver was designed to84completely subsume the functionality of the gcc executable; that is, the85driver should not need to delegate to gcc to perform subtasks. On86Darwin, this implies that the Clang driver also subsumes the gcc87driver-driver, which is used to implement support for building universal88images (binaries and object files). This also implies that the driver89should be able to call the language specific compilers (e.g. cc1)90directly, which means that it must have enough information to forward91command line arguments to child processes correctly.92 93Design Overview94---------------95 96The diagram below shows the significant components of the driver97architecture and how they relate to one another. The orange components98represent concrete data structures built by the driver, the green99components indicate conceptually distinct stages which manipulate these100data structures, and the blue components are important helper classes.101 102.. image:: DriverArchitecture.png103 :align: center104 :alt: Driver Architecture Diagram105 106Driver Stages107-------------108 109The driver functionality is conceptually divided into five stages:110 111#. **Parse: Option Parsing**112 113 The command line argument strings are decomposed into arguments114 (``Arg`` instances). The driver expects to understand all available115 options, although there is some facility for just passing certain116 classes of options through (like ``-Wl,``).117 118 Each argument corresponds to exactly one abstract ``Option``119 definition, which describes how the option is parsed along with some120 additional metadata. The Arg instances themselves are lightweight and121 merely contain enough information for clients to determine which122 option they correspond to and their values (if they have additional123 parameters).124 125 For example, a command line like "-Ifoo -I foo" would parse to two126 Arg instances (a JoinedArg and a SeparateArg instance), but each127 would refer to the same Option.128 129 Options are lazily created in order to avoid populating all Option130 classes when the driver is loaded. Most of the driver code only needs131 to deal with options by their unique ID (e.g., ``options::OPT_I``),132 133 Arg instances themselves do not generally store the values of134 parameters. In many cases, this would simply result in creating135 unnecessary string copies. Instead, Arg instances are always embedded136 inside an ArgList structure, which contains the original vector of137 argument strings. Each Arg itself only needs to contain an index into138 this vector instead of storing its values directly.139 140 The clang driver can dump the results of this stage using the141 ``-###`` flag (which must precede any actual command142 line arguments). For example:143 144 .. code-block:: console145 146 $ clang -### -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c147 Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"}148 Option 1 - Name: "-Wa,", Values: {"-fast"}149 Option 2 - Name: "-I", Values: {"foo"}150 Option 3 - Name: "-I", Values: {"foo"}151 Option 4 - Name: "<input>", Values: {"t.c"}152 153 After this stage is complete the command line should be broken down154 into well defined option objects with their appropriate parameters.155 Subsequent stages should rarely, if ever, need to do any string156 processing.157 158#. **Pipeline: Compilation Action Construction**159 160 Once the arguments are parsed, the tree of subprocess jobs needed for161 the desired compilation sequence are constructed. This involves162 determining the input files and their types, what work is to be done163 on them (preprocess, compile, assemble, link, etc.), and constructing164 a list of Action instances for each task. The result is a list of one165 or more top-level actions, each of which generally corresponds to a166 single output (for example, an object or linked executable).167 168 The majority of Actions correspond to actual tasks, however there are169 two special Actions. The first is InputAction, which simply serves to170 adapt an input argument for use as an input to other Actions. The171 second is BindArchAction, which conceptually alters the architecture172 to be used for all of its input Actions.173 174 The clang driver can dump the results of this stage using the175 ``-ccc-print-phases`` flag. For example:176 177 .. code-block:: console178 179 $ clang -ccc-print-phases -x c t.c -x assembler t.s180 0: input, "t.c", c181 1: preprocessor, {0}, cpp-output182 2: compiler, {1}, assembler183 3: assembler, {2}, object184 4: input, "t.s", assembler185 5: assembler, {4}, object186 6: linker, {3, 5}, image187 188 Here the driver is constructing seven distinct actions, four to189 compile the "t.c" input into an object file, two to assemble the190 "t.s" input, and one to link them together.191 192 A rather different compilation pipeline is shown here; in this193 example there are two top level actions to compile the input files194 into two separate object files, where each object file is built using195 ``lipo`` to merge results built for two separate architectures.196 197 .. code-block:: console198 199 $ clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c200 0: input, "t0.c", c201 1: preprocessor, {0}, cpp-output202 2: compiler, {1}, assembler203 3: assembler, {2}, object204 4: bind-arch, "i386", {3}, object205 5: bind-arch, "x86_64", {3}, object206 6: lipo, {4, 5}, object207 7: input, "t1.c", c208 8: preprocessor, {7}, cpp-output209 9: compiler, {8}, assembler210 10: assembler, {9}, object211 11: bind-arch, "i386", {10}, object212 12: bind-arch, "x86_64", {10}, object213 13: lipo, {11, 12}, object214 215 After this stage is complete the compilation process is divided into216 a simple set of actions which need to be performed to produce217 intermediate or final outputs (in some cases, like ``-fsyntax-only``,218 there is no "real" final output). Phases are well known compilation219 steps, such as "preprocess", "compile", "assemble", "link", etc.220 221#. **Bind: Tool & Filename Selection**222 223 This stage (in conjunction with the Translate stage) turns the tree224 of Actions into a list of actual subprocess to run. Conceptually, the225 driver performs a top down matching to assign Action(s) to Tools. The226 ToolChain is responsible for selecting the tool to perform a227 particular action; once selected the driver interacts with the tool228 to see if it can match additional actions (for example, by having an229 integrated preprocessor).230 231 Once Tools have been selected for all actions, the driver determines232 how the tools should be connected (for example, using an inprocess233 module, pipes, temporary files, or user provided filenames). If an234 output file is required, the driver also computes the appropriate235 file name (the suffix and file location depend on the input types and236 options such as ``-save-temps``).237 238 The driver interacts with a ToolChain to perform the Tool bindings.239 Each ToolChain contains information about all the tools needed for240 compilation for a particular architecture, platform, and operating241 system. A single driver invocation may query multiple ToolChains242 during one compilation in order to interact with tools for separate243 architectures.244 245 The results of this stage are not computed directly, but the driver246 can print the results via the ``-ccc-print-bindings`` option. For247 example:248 249 .. code-block:: console250 251 $ clang -ccc-print-bindings -arch i386 -arch ppc t0.c252 # "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s"253 # "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o"254 # "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out"255 # "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s"256 # "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o"257 # "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out"258 # "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out"259 260 This shows the tool chain, tool, inputs and outputs which have been261 bound for this compilation sequence. Here clang is being used to262 compile t0.c on the i386 architecture and darwin specific versions of263 the tools are being used to assemble and link the result, but generic264 gcc versions of the tools are being used on PowerPC.265 266#. **Translate: Tool Specific Argument Translation**267 268 Once a Tool has been selected to perform a particular Action, the269 Tool must construct concrete Commands which will be executed during270 compilation. The main work is in translating from the gcc style271 command line options to whatever options the subprocess expects.272 273 Some tools, such as the assembler, only interact with a handful of274 arguments and just determine the path of the executable to call and275 pass on their input and output arguments. Others, like the compiler276 or the linker, may translate a large number of arguments in addition.277 278 The ArgList class provides a number of simple helper methods to279 assist with translating arguments; for example, to pass on only the280 last of arguments corresponding to some option, or all arguments for281 an option.282 283 The result of this stage is a list of Commands (executable paths and284 argument strings) to execute.285 286#. **Execute**287 288 Finally, the compilation pipeline is executed. This is mostly289 straightforward, although there is some interaction with options like290 ``-pipe``, ``-pass-exit-codes`` and ``-time``.291 292Additional Notes293----------------294 295The Compilation Object296^^^^^^^^^^^^^^^^^^^^^^297 298The driver constructs a Compilation object for each set of command line299arguments. The Driver itself is intended to be invariant during300construction of a Compilation; an IDE should be able to construct a301single long lived driver instance to use for an entire build, for302example.303 304The Compilation object holds information that is particular to each305compilation sequence. For example, the list of used temporary files306(which must be removed once compilation is finished) and result files307(which should be removed if compilation fails).308 309Unified Parsing & Pipelining310^^^^^^^^^^^^^^^^^^^^^^^^^^^^311 312Parsing and pipelining both occur without reference to a Compilation313instance. This is by design; the driver expects that both of these314phases are platform neutral, with a few very well defined exceptions315such as whether the platform uses a driver driver.316 317ToolChain Argument Translation318^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^319 320In order to match gcc very closely, the clang driver currently allows321tool chains to perform their own translation of the argument list (into322a new ArgList data structure). Although this allows the clang driver to323match gcc easily, it also makes the driver operation much harder to324understand (since the Tools stop seeing some arguments the user325provided, and see new ones instead).326 327For example, on Darwin ``-gfull`` gets translated into two separate328arguments, ``-g`` and ``-fno-eliminate-unused-debug-symbols``. Trying to329write Tool logic to do something with ``-gfull`` will not work, because330Tool argument translation is done after the arguments have been331translated.332 333A long term goal is to remove this tool chain specific translation, and334instead force each tool to change its own logic to do the right thing on335the untranslated original arguments.336 337Unused Argument Warnings338^^^^^^^^^^^^^^^^^^^^^^^^339 340The driver operates by parsing all arguments but giving Tools the341opportunity to choose which arguments to pass on. One downside of this342infrastructure is that if the user misspells some option, or is confused343about which options to use, some command line arguments the user really344cared about may go unused. This problem is particularly important when345using clang as a compiler, since the clang compiler does not support346anywhere near all the options that gcc does, and we want to make sure347users know which ones are being used.348 349To support this, the driver maintains a bit associated with each350argument of whether it has been used (at all) during the compilation.351This bit usually doesn't need to be set by hand, as the key ArgList352accessors will set it automatically.353 354When a compilation is successful (there are no errors), the driver355checks the bit and emits an "unused argument" warning for any arguments356which were never accessed. This is conservative (the argument may not357have been used to do what the user wanted) but still catches the most358obvious cases.359 360Relation to GCC Driver Concepts361-------------------------------362 363For those familiar with the gcc driver, this section provides a brief364overview of how things from the gcc driver map to the clang driver.365 366- **Driver Driver**367 368 The driver driver is fully integrated into the clang driver. The369 driver simply constructs additional Actions to bind the architecture370 during the *Pipeline* phase. The tool chain specific argument371 translation is responsible for handling ``-Xarch_``.372 373 The one caveat is that this approach requires ``-Xarch_`` not be used374 to alter the compilation itself (for example, one cannot provide375 ``-S`` as an ``-Xarch_`` argument). The driver attempts to reject376 such invocations, and overall there isn't a good reason to abuse377 ``-Xarch_`` to that end in practice.378 379 The upside is that the clang driver is more efficient and does little380 extra work to support universal builds. It also provides better error381 reporting and UI consistency.382 383- **Specs**384 385 The clang driver has no direct correspondent for "specs". The386 majority of the functionality that is embedded in specs is in the387 Tool specific argument translation routines. The parts of specs which388 control the compilation pipeline are generally part of the *Pipeline*389 stage.390 391- **Toolchains**392 393 The gcc driver has no direct understanding of tool chains. Each gcc394 binary roughly corresponds to the information which is embedded395 inside a single ToolChain.396 397 The clang driver is intended to be portable and support complex398 compilation environments. All platform and tool chain specific code399 should be protected behind either abstract or well defined interfaces400 (such as whether the platform supports use as a driver driver).401