546 lines · plain
1=====================2Clang Offload Bundler3=====================4 5.. contents::6 :local:7 8.. _clang-offload-bundler:9 10Introduction11============12 13For heterogeneous single source programming languages, use one or more14``--offload-arch=<target-id>`` Clang options to specify the target IDs of the15code to generate for the offload code regions.16 17The tool chain may perform multiple compilations of a translation unit to18produce separate code objects for the host and potentially multiple offloaded19devices. The ``clang-offload-bundler`` tool may be used as part of the tool20chain to combine these multiple code objects into a single bundled code object.21 22The tool chain may use a bundled code object as an intermediate step so that23each tool chain step consumes and produces a single file as in traditional24non-heterogeneous tool chains. The bundled code object contains the code objects25for the host and all the offload devices.26 27A bundled code object may also be used to bundle just the offloaded code28objects, and embedded as data into the host code object. The host compilation29includes an ``init`` function that will use the runtime corresponding to the30offload kind (see :ref:`clang-offload-kind-table`) to load the offload code31objects appropriate for the devices present when the host program is executed.32 33:program:`clang-offload-bundler` is located in34`clang/tools/clang-offload-bundler`.35 36.. code-block:: console37 38 $ clang-offload-bundler -help39 OVERVIEW: A tool to bundle several input files of the specified type <type>40 referring to the same source file but different targets into a single41 one. The resulting file can also be unbundled into different files by42 this tool if -unbundle is provided.43 44 USAGE: clang-offload-bundler [options]45 46 OPTIONS:47 48 Generic Options:49 50 --help - Display available options (--help-hidden for more)51 --help-list - Display list of available options (--help-list-hidden for more)52 --version - Display the version of this program53 54 clang-offload-bundler options:55 56 --### - Print any external commands that are to be executed instead of actually executing them - for testing purposes.57 --allow-missing-bundles - Create empty files if bundles are missing when unbundling.58 --bundle-align=<uint> - Alignment of bundle for binary files59 --check-input-archive - Check if input heterogeneous archive is valid in terms of TargetID rules.60 --inputs=<string> - [<input file>,...]61 --list - List bundle IDs in the bundled file.62 --outputs=<string> - [<output file>,...]63 --targets=<string> - [<offload kind>-<target triple>,...]64 --type=<string> - Type of the files to be bundled/unbundled.65 Current supported types are:66 i - cpp-output67 ii - c++-cpp-output68 cui - cuda/hip-output69 d - dependency70 ll - llvm71 bc - llvm-bc72 s - assembler73 o - object74 a - archive of bundled files75 gch - precompiled-header76 ast - clang AST file77 --unbundle - Unbundle bundled file into several output files.78 79Usage80=====81 82This tool can be used as follows for bundling:83 84::85 86 clang-offload-bundler -targets=triple1,triple2 -type=ii -inputs=a.triple1.ii,a.triple2.ii -outputs=a.ii87 88or, it can be used as follows for unbundling:89 90::91 92 clang-offload-bundler -targets=triple1,triple2 -type=ii -outputs=a.triple1.ii,a.triple2.ii -inputs=a.ii -unbundle93 94 95Supported File Formats96======================97 98Multiple text and binary file formats are supported for bundling/unbundling. See99:ref:`supported-file-formats-table` for a list of currently supported input100formats. Use the ``File Type`` column to determine the value to pass to the101``--type`` option based on the type of input files while bundling/unbundling.102 103 .. table:: Supported File Formats104 :name: supported-file-formats-table105 106 +--------------------------+----------------+-------------+107 | File Format | File Type | Text/Binary |108 +==========================+================+=============+109 | CPP output | i | Text |110 +--------------------------+----------------+-------------+111 | C++ CPP output | ii | Text |112 +--------------------------+----------------+-------------+113 | CUDA/HIP output | cui | Text |114 +--------------------------+----------------+-------------+115 | Dependency | d | Text |116 +--------------------------+----------------+-------------+117 | LLVM | ll | Text |118 +--------------------------+----------------+-------------+119 | LLVM Bitcode | bc | Binary |120 +--------------------------+----------------+-------------+121 | Assembler | s | Text |122 +--------------------------+----------------+-------------+123 | Object | o | Binary |124 +--------------------------+----------------+-------------+125 | Archive of bundled files | a | Binary |126 +--------------------------+----------------+-------------+127 | Precompiled header | gch | Binary |128 +--------------------------+----------------+-------------+129 | Clang AST file | ast | Binary |130 +--------------------------+----------------+-------------+131 132.. _clang-bundled-code-object-layout-text:133 134Bundled Text File Layout135========================136 137The text file formats are concatenated with comments that have a magic string138and bundle entry ID in between. The BNF syntax to represent a code object139bundle file is:140 141::142 143 <file> ::== <bundle> | <bundle> <file>144 <bundle> ::== <comment> <start> <bundle_id> <eol> <bundle> <eol>145 <comment> end <bundle_id> <eol>146 <start> ::== OFFLOAD_BUNDLER_MAGIC_STR__START__147 <end> ::== OFFLOAD_BUNDLER_MAGIC_STR__END__148 149**comment**150 The symbol used for starting a single-line comment in the file type of151 constituting bundles. E.g. it is ";" for ll ``File Type`` and "#" for "s"152 ``File Type``.153 154**bundle_id**155 The :ref:`clang-bundle-entry-id` for the enclosing bundle.156 157**eol**158 The end-of-line character.159 160**bundle**161 The code object stored in one of the supported text file formats.162 163**OFFLOAD_BUNDLER_MAGIC_STR__**164 The magic string that marks the existence of offloading data i.e.165 "__CLANG_OFFLOAD_BUNDLE__".166 167.. _clang-bundled-code-object-layout:168 169Bundled Binary File Layout170==========================171 172The layout of a bundled code object is defined by the following table:173 174 .. table:: Bundled Code Object Layout175 :name: bundled-code-object-layout-table176 177 =================================== ======= ================ ===============================178 Field Type Size in Bytes Description179 =================================== ======= ================ ===============================180 Magic String string 24 ``__CLANG_OFFLOAD_BUNDLE__``181 Number Of Bundle Entries integer 8 Number of bundle entries.182 1st Bundle Entry Code Object Offset integer 8 Byte offset from beginning of183 bundled code object to 1st code184 object.185 1st Bundle Entry Code Object Size integer 8 Byte size of 1st code object.186 1st Bundle Entry ID Length integer 8 Character length of bundle187 entry ID of 1st code object.188 1st Bundle Entry ID string 1st Bundle Entry Bundle entry ID of 1st code189 ID Length object. This is not NUL190 terminated. See191 :ref:`clang-bundle-entry-id`.192 \...193 Nth Bundle Entry Code Object Offset integer 8194 Nth Bundle Entry Code Object Size integer 8195 Nth Bundle Entry ID Length integer 8196 Nth Bundle Entry ID string 1st Bundle Entry197 ID Length198 1st Bundle Entry Code Object bytes 1st Bundle Entry199 Code Object Size200 \...201 Nth Bundle Entry Code Object bytes Nth Bundle Entry202 Code Object Size203 =================================== ======= ================ ===============================204 205.. _clang-bundle-entry-id:206 207Bundle Entry ID208===============209 210Each entry in a bundled code object (see :ref:`clang-bundled-code-object-layout-text`211and :ref:`clang-bundled-code-object-layout`) has a bundle entry ID that indicates212the kind of the entry's code object and the runtime that manages it.213 214Bundle entry ID syntax is defined by the following BNF syntax:215 216.. code::217 218 <bundle-entry-id> ::== <offload-kind> "-" <target-triple> [ "-" <target-id> ]219 220Where:221 222**offload-kind**223 The runtime responsible for managing the bundled entry code object. See224 :ref:`clang-offload-kind-table`.225 226 .. table:: Bundled Code Object Offload Kind227 :name: clang-offload-kind-table228 229 ============= ==============================================================230 Offload Kind Description231 ============= ==============================================================232 host Host code object. ``clang-offload-bundler`` always includes233 this entry as the first bundled code object entry. For an234 embedded bundled code object, this entry is not used by the235 runtime and so is generally an empty code object.236 237 hip Offload code object for the HIP language. Used for all238 HIP language offload code objects when the239 ``clang-offload-bundler`` is used to bundle code objects as240 intermediate steps of the tool chain. Also used for AMD GPU241 code objects before ABI version V4 when the242 ``clang-offload-bundler`` is used to create a *fat binary*243 to be loaded by the HIP runtime. The fat binary can be244 loaded directly from a file, or be embedded in the host code245 object as a data section with the name ``.hip_fatbin``.246 247 hipv4 Offload code object for the HIP language. Used for AMD GPU248 code objects with at least ABI version V4 and above when the249 ``clang-offload-bundler`` is used to create a *fat binary*250 to be loaded by the HIP runtime. The fat binary can be251 loaded directly from a file, or be embedded in the host code252 object as a data section with the name ``.hip_fatbin``.253 254 openmp Offload code object for the OpenMP language extension.255 ============= ==============================================================256 257Note: The distinction between the `hip` and `hipv4` offload kinds is historically based.258Originally, these designations might have indicated different versions of the259code object ABI. However, as the system has evolved, the ABI version is now embedded260directly within the code object itself, making these historical distinctions irrelevant261during the unbundling process. Consequently, `hip` and `hipv4` are treated as compatible262in current implementations, facilitating interchangeable handling of code objects263without differentiation based on offload kind.264 265**target-triple**266 The target triple of the code object. See `Target Triple267 <https://clang.llvm.org/docs/CrossCompilation.html#target-triple>`_.268 269 LLVM target triples can be with or without the optional environment field:270 271 ``<arch><sub>-<vendor>-<sys>``, or272 ``<arch><sub>-<vendor>-<sys>-<env>``273 274 However, in order to standardize outputs for tools that consume bitcode bundles275 and to parse a target ID containing dashes, the bundler only accepts target276 triples in the 4-field format:277 278 ``<arch><sub>-<vendor>-<sys>-<env>``279 280**target-id**281 The canonical target ID of the code object. Present only if the target282 supports a target ID. See :ref:`clang-target-id`.283 284.. _code-object-composition:285 286Bundled Code Object Composition287-------------------------------288 289 * Each entry of a bundled code object must have a different bundle entry ID.290 * There can be multiple entries for the same processor provided they differ291 in target feature settings.292 * If there is an entry with a target feature specified as *Any*, then all293 entries must specify that target feature as *Any* for the same processor.294 295There may be additional target-specific restrictions.296 297.. _compatibility-bundle-entry-id:298 299Compatibility Rules for Bundle Entry ID300---------------------------------------301 302 A code object, specified using its Bundle Entry ID, can be loaded and303 executed on a target processor if:304 305 * Their offload kinds are the same or compatible.306 * Their target triples are compatible.307 * Their Target IDs are compatible as defined in :ref:`compatibility-target-id`.308 309.. _clang-target-id:310 311Target ID312=========313 314A target ID is used to indicate the processor and optionally its configuration,315expressed by a set of target features, that affect ISA generation. It is target316specific if a target ID is supported, or if the target triple alone is317sufficient to specify the ISA generation.318 319It is used with the ``-mcpu=<target-id>`` and ``--offload-arch=<target-id>``320Clang compilation options to specify the kind of code to generate.321 322It is also used as part of the bundle entry ID to identify the code object. See323:ref:`clang-bundle-entry-id`.324 325Target ID syntax is defined by the following BNF syntax:326 327.. code::328 329 <target-id> ::== <processor> ( ":" <target-feature> ( "+" | "-" ) )*330 331Where:332 333**processor**334 Is the target-specific processor or any alternative processor name.335 336**target-feature**337 Is a target feature name that is supported by the processor. Each target338 feature must appear at most once in a target ID and can have one of three339 values:340 341 *Any*342 Specified by omitting the target feature from the target ID.343 A code object compiled with a target ID specifying the default344 value of a target feature can be loaded and executed on a processor345 configured with the target feature on or off.346 347 *On*348 Specified by ``+``, indicating the target feature is enabled. A code349 object compiled with a target ID specifying a target feature on350 can only be loaded on a processor configured with the target feature on.351 352 *Off*353 Specified by ``-``, indicating the target feature is disabled. A code354 object compiled with a target ID specifying a target feature off355 can only be loaded on a processor configured with the target feature off.356 357.. _compatibility-target-id:358 359Compatibility Rules for Target ID360---------------------------------361 362 A code object compiled for a Target ID is considered compatible for a363 target if:364 365 * Their processor is the same.366 * Their feature set is compatible as defined above.367 368There are two forms of target ID:369 370*Non-Canonical Form*371 The non-canonical form is used as the input to user commands to allow the user372 greater convenience. It allows both the primary and alternative processor name373 to be used and the target features may be specified in any order.374 375*Canonical Form*376 The canonical form is used for all generated output to allow greater377 convenience for tools that consume the information. It is also used for378 internal passing of information between tools. Only the primary and not379 alternative processor name is used and the target features are specified in380 alphabetic order. Command line tools convert non-canonical form to canonical381 form.382 383Target-Specific information384===========================385 386Target-specific information is available for the following:387 388*AMD GPU*389 AMD GPU supports target ID and target features. See `User Guide for AMDGPU Backend390 <https://llvm.org/docs/AMDGPUUsage.html>`_ which defines the `processors391 <https://llvm.org/docs/AMDGPUUsage.html#amdgpu-processors>`_ and `target392 features <https://llvm.org/docs/AMDGPUUsage.html#amdgpu-target-features>`_393 supported.394 395Most other targets do not support target IDs.396 397Archive Unbundling398==================399 400Unbundling of a heterogeneous device archive (HDA) is done to create device-specific401archives. HDA is in a format compatible with GNU ``ar`` utility and contains a402collection of bundled device binaries where each bundle file will contain403device binaries for a host and one or more targets. The output device-specific404archive is in a format compatible with GNU ``ar`` utility and contains a405collection of device binaries for a specific target.406 407::408 409 Heterogeneous Device Archive, HDA = {F1.X, F2.X, ..., FN.Y}410 where, Fi = Bundle{Host-DeviceBinary, T1-DeviceBinary, T2-DeviceBinary, ...,411 Tm-DeviceBinary},412 Ti = {Target i, qualified using Bundle Entry ID},413 X/Y = \*.bc for AMDGPU and \*.cubin for NVPTX414 415 Device Specific Archive, DSA(Tk) = {F1-Tk-DeviceBinary.X, F2-Tk-DeviceBinary.X, ...416 FN-Tk-DeviceBinary.Y}417 where, Fi-Tj-DeviceBinary.X represents device binary of i-th bundled device418 binary file for target Tj.419 420The clang-offload-bundler extracts compatible device binaries for a given target421from the bundled device binaries in a heterogeneous device archive and creates422a target-specific device archive without bundling.423 424The clang-offload-bundler determines whether a device binary is compatible425with a target by comparing bundle IDs. Two bundle IDs are considered426compatible if:427 428 * Their offload kinds are the same429 * Their target triples are the same430 * Their Target IDs are the same431 432Creating a Heterogeneous Device Archive433---------------------------------------434 4351. Compile source file(s) to generate object file(s)436 437 ::438 439 clang -O2 -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa,\440 nvptx64-nvidia-cuda, nvptx64-nvidia-cuda \441 -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906:sramecc-:xnack+ \442 -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906:sramecc+:xnack+ \443 -Xopenmp-target=nvptx64-nvidia-cuda -march=sm_70 \444 -Xopenmp-target=nvptx64-nvidia-cuda -march=sm_80 \445 -c func_1.c -o func_1.o446 447 clang -O2 -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa,448 nvptx64-nvidia-cuda, nvptx64-nvidia-cuda \449 -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906:sramecc-:xnack+ \450 -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906:sramecc+:xnack+ \451 -Xopenmp-target=nvptx64-nvidia-cuda -march=sm_70 \452 -Xopenmp-target=nvptx64-nvidia-cuda -march=sm_80 \453 -c func_2.c -o func_2.o454 4552. Create a heterogeneous device archive by combining all the object file(s)456 457 ::458 459 llvm-ar cr libFatArchive.a func_1.o func_2.o460 461Extracting a Device Specific Archive462------------------------------------463 464UnbundleArchive takes a heterogeneous device archive file (".a") as input465containing bundled device binary files, and a list of offload targets (not466host), and extracts the device binaries into a new archive file for each467offload target. Each resulting archive file contains all device binaries468compatible with that particular offload target. Compatibility between a469device binary in HDA and a target is based on the compatibility between their470bundle entry IDs as defined in :ref:`compatibility-bundle-entry-id`.471 472The following cases may arise during compatibility testing:473 474* A binary is compatible with one or more targets: Insert the binary into the475 device-specific archive of each compatible target.476* A binary is not compatible with any target: Skip the binary.477* One or more binaries are compatible with a target: Insert all binaries into478 the device-specific archive of the target. The insertion need not be ordered.479* No binary is compatible with a target: If ``allow-missing-bundles`` option is480 present then create an empty archive for the target. Otherwise, produce an481 error without creating an archive.482 483The created archive file does not contain an index of the symbols and device484binary files are named as <<Parent Bundle Name>-<DeviceBinary's TargetID>>,485with ':' replaced with '_'.486 487Usage488-----489 490::491 492 clang-offload-bundler --unbundle --inputs=libFatArchive.a -type=a \493 -targets=openmp-amdgcn-amdhsa-gfx906:sramecc+:xnack+, \494 openmp-amdgcn-amdhsa-gfx908:sramecc-:xnack+ \495 -outputs=devicelib-gfx906.a,deviceLib-gfx908.a496 497.. _additional-options-archive-unbundling:498 499Additional Options while Archive Unbundling500-------------------------------------------501 502**-allow-missing-bundles**503 Create an empty archive file if no compatible device binary is found.504 505**-check-input-archive**506 Check if input heterogeneous device archive follows rules for composition507 as defined in :ref:`code-object-composition` before creating device-specific508 archive(s).509 510**-debug-only=CodeObjectCompatibility**511 Verbose printing of matched/unmatched comparisons between bundle entry id of512 a device binary from HDA and bundle entry ID of a given target processor513 (see :ref:`compatibility-bundle-entry-id`).514 515Compression and Decompression516=============================517 518``clang-offload-bundler`` provides features to compress and decompress the full519bundle, leveraging inherent redundancies within the bundle entries. Use the520``-compress`` command-line option to enable this compression capability.521 522The compressed offload bundle begins with a header followed by the compressed binary data:523 524- **Magic Number (4 bytes)**:525 This is a unique identifier to distinguish compressed offload bundles. The value is the string 'CCOB' (Compressed Clang Offload Bundle).526 527- **Version Number (16-bit unsigned int)**:528 This denotes the version of the compressed offload bundle format. The current version is `3`.529 530- **Compression Method (16-bit unsigned int)**:531 This field indicates the compression method used. The value corresponds to either `zlib` or `zstd`, represented as a 16-bit unsigned integer cast from the LLVM compression enumeration.532 533- **Total File Size (unsigned int, 32-bit in v2, 64-bit in v3)**:534 This is the total size (in bytes) of the file, including the header. Available in version 2 and above.535 536- **Uncompressed Binary Size (unsigned int, 32-bit in v2, 64-bit in v3)**:537 This is the size (in bytes) of the binary data before it was compressed.538 539- **Hash (64-bit unsigned int)**:540 This is a 64-bit truncated MD5 hash of the uncompressed binary data. It serves for verification and caching purposes.541 542- **Compressed Data**:543 The actual compressed binary data follows the header. Its size can be inferred from the total size of the file minus the header size.544 545 > **Note**: Version 3 is now the default format. For backward compatibility with older HIP runtimes that support version 2 only, set the environment variable ``COMPRESSED_BUNDLE_FORMAT_VERSION=2``.546