547 lines · plain
1==========================2Source-based Code Coverage3==========================4 5.. contents::6 :local:7 8Introduction9============10 11This document explains how to use clang's source-based code coverage feature.12It's called "source-based" because it operates on AST and preprocessor13information directly. This allows it to generate very precise coverage data.14 15Clang ships two other code coverage implementations:16 17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the18 various sanitizers. It can provide up to edge-level coverage.19 20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.21 This is enabled by ``-ftest-coverage`` or ``--coverage``.22 23From this point onwards "code coverage" will refer to the source-based kind.24 25The code coverage workflow26==========================27 28The code coverage workflow consists of three main steps:29 30* Compiling with coverage enabled.31 32* Running the instrumented program.33 34* Creating coverage reports.35 36The next few sections work through a complete, copy-'n-paste friendly example37based on this program:38 39.. code-block:: cpp40 41 % cat <<EOF > foo.cc42 #define BAR(x) ((x) || (x))43 template <typename T> void foo(T x) {44 for (unsigned I = 0; I < 10; ++I) { BAR(I); }45 }46 int main() {47 foo<int>(0);48 foo<float>(0);49 return 0;50 }51 EOF52 53Compiling with coverage enabled54===============================55 56To compile code with coverage enabled, pass ``-fprofile-instr-generate57-fcoverage-mapping`` to the compiler:58 59.. code-block:: console60 61 # Step 1: Compile with coverage enabled.62 % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo63 64Note that linking together code with and without coverage instrumentation is65supported. Uninstrumented code simply won't be accounted for in reports.66 67To compile code with Modified Condition/Decision Coverage (MC/DC) enabled,68pass ``-fcoverage-mcdc`` in addition to the clang options specified above.69MC/DC is an advanced form of code coverage most applicable to the embedded70space.71 72Running the instrumented program73================================74 75The next step is to run the instrumented program. When the program exits, it76will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``77environment variable. If that variable does not exist, the profile is written78to ``default.profraw`` in the current directory of the program. If79``LLVM_PROFILE_FILE`` specifies a path to a non-existent directory, the missing80directory structure will be created. Additionally, the following special81**pattern strings** are rewritten:82 83* "%p" expands out to the process ID.84 85* "%h" expands out to the hostname of the machine running the program.86 87* "%t" expands out to the value of the ``TMPDIR`` environment variable. On88 Darwin, this is typically set to a temporary scratch directory.89 90* "%Nm" expands out to the instrumented binary's signature. When this pattern91 is specified, the runtime creates a pool of N raw profiles which are used for92 on-line profile merging. The runtime takes care of selecting a raw profile93 from the pool, locking it, and updating it before the program exits. If N is94 not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. The95 merge pool specifier can only occur once per filename pattern.96 97* "%b" expands out to the binary ID (build ID). It can be used with "%Nm" to98 avoid binary signature collisions. To use it, the program should be compiled99 with the build ID linker option (``--build-id`` for GNU ld or LLD,100 ``/build-id`` for lld-link on Windows). Linux, Windows, and AIX are supported.101 102* "%c" expands out to nothing, but enables a mode in which profile counter103 updates are continuously synced to a file. This means that if the104 instrumented program crashes, or is killed by a signal, perfect coverage105 information can still be recovered. Continuous mode does not support value106 profiling for PGO, and is only supported on Darwin at the moment. Support for107 Linux may be mostly complete but requires testing, and support for Windows108 may require more extensive changes: please get involved if you are interested109 in porting this feature.110 111.. code-block:: console112 113 # Step 2: Run the program.114 % LLVM_PROFILE_FILE="foo.profraw" ./foo115 116Note that continuous mode is also used on Fuchsia where it's the only supported117mode, but the implementation is different. The Darwin and Linux implementation118relies on padding and the ability to map a file over the existing memory119mapping which is generally only available on POSIX systems and isn't suitable120for other platforms.121 122On Fuchsia, we rely on the ability to relocate counters at runtime using a123level of indirection. On every counter access, we add a bias to the counter124address. This bias is stored in ``__llvm_profile_counter_bias`` symbol that's125provided by the profile runtime and is initially set to zero, meaning no126relocation. The runtime can map the profile into memory at arbitrary locations,127and set bias to the offset between the original and the new counter location,128at which point every subsequent counter access will be to the new location,129which allows updating profile directly akin to the continuous mode.130 131The advantage of this approach is that it doesn't require any special OS support.132The disadvantage is the extra overhead due to additional instructions required133for each counter access (overhead both in terms of binary size and performance)134plus duplication of counters (i.e. one copy in the binary itself and another135copy that's mapped into memory). This implementation can be also enabled for136other platforms by passing the ``-runtime-counter-relocation`` option to the137backend during compilation.138 139For a program such as the `Lit <https://llvm.org/docs/CommandGuide/lit.html>`_140testing tool, which invokes other programs, it may be necessary to set141``LLVM_PROFILE_FILE`` for each invocation. The pattern strings "%p" or "%Nm"142may help to avoid corruption due to concurrency. Note that "%p" is also a Lit143token and needs to be escaped as "%%p".144 145.. code-block:: console146 147 % clang++ -fprofile-instr-generate -fcoverage-mapping -mllvm -runtime-counter-relocation foo.cc -o foo148 149Creating coverage reports150=========================151 152Raw profiles must be **indexed** before they can be used to generate153coverage reports. This is done using the "merge" tool in ``llvm-profdata``154(which can combine multiple raw profiles and index them at the same time):155 156.. code-block:: console157 158 # Step 3(a): Index the raw profile.159 % llvm-profdata merge -sparse foo.profraw -o foo.profdata160 161For an example of merging multiple profiles created by testing,162see the LLVM `coverage build script <https://github.com/llvm/llvm-zorg/blob/main/zorg/jenkins/jobs/jobs/llvm-coverage>`_.163 164There are multiple different ways to render coverage reports. The simplest165option is to generate a line-oriented report:166 167.. code-block:: console168 169 # Step 3(b): Create a line-oriented coverage report.170 % llvm-cov show ./foo -instr-profile=foo.profdata171 172This report includes a summary view as well as dedicated sub-views for173templated functions and their instantiations. For our example program, we get174distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If175``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line176region counts (even in macro expansions):177 178.. code-block:: none179 180 1| 20|#define BAR(x) ((x) || (x))181 ^20 ^2182 2| 2|template <typename T> void foo(T x) {183 3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); }184 ^22 ^20 ^20^20185 4| 2|}186 ------------------187 | void foo<int>(int):188 | 2| 1|template <typename T> void foo(T x) {189 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); }190 | ^11 ^10 ^10^10191 | 4| 1|}192 ------------------193 | void foo<float>(int):194 | 2| 1|template <typename T> void foo(T x) {195 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); }196 | ^11 ^10 ^10^10197 | 4| 1|}198 ------------------199 200If ``--show-branches=count`` and ``--show-expansions`` are also enabled, the201sub-views will show detailed branch coverage information in addition to the202region counts:203 204.. code-block:: none205 206 ------------------207 | void foo<float>(int):208 | 2| 1|template <typename T> void foo(T x) {209 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); }210 | ^11 ^10 ^10^10211 | ------------------212 | | | 1| 10|#define BAR(x) ((x) || (x))213 | | | ^10 ^1214 | | | ------------------215 | | | | Branch (1:17): [True: 9, False: 1]216 | | | | Branch (1:24): [True: 0, False: 1]217 | | | ------------------218 | ------------------219 | | Branch (3:23): [True: 10, False: 1]220 | ------------------221 | 4| 1|}222 ------------------223 224If the application was instrumented for Modified Condition/Decision Coverage225(MC/DC) using the clang option ``-fcoverage-mcdc``, an MC/DC subview can be226enabled using ``--show-mcdc`` that will show detailed MC/DC information for227each complex condition boolean expression containing at most six conditions.228 229To generate a file-level summary of coverage statistics instead of a230line-oriented report, try:231 232.. code-block:: console233 234 # Step 3(c): Create a coverage summary.235 % llvm-cov report ./foo -instr-profile=foo.profdata236 Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover237 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------238 /tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33%239 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------240 TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33%241 242The ``llvm-cov`` tool supports specifying a custom demangler, writing out243reports in a directory structure, and generating HTML reports. For the full244list of options, please refer to the `command guide245<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_.246 247A few final notes:248 249* The ``-sparse`` flag is optional but can produce dramatically smaller250 indexed profiles. This option should not be used if the indexed profile will251 be reused for PGO.252 253* Raw profiles can be discarded after they are indexed. Advanced use of the254 profile runtime library allows an instrumented program to merge profiling255 information directly into an existing raw profile on disk. The details are256 out of scope.257 258* The ``llvm-profdata`` tool can be used to merge multiple raw or259 indexed profiles. To combine profiling data from multiple runs of a program,260 try e.g:261 262 .. code-block:: console263 264 % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata265 266Exporting coverage data267=======================268 269Coverage data can be exported into JSON using the ``llvm-cov export``270sub-command. There is a comprehensive reference which defines the structure of271the exported data at a high level in the llvm-cov source code.272 273Interpreting reports274====================275 276There are six statistics tracked in a coverage summary:277 278* Function coverage is the percentage of functions which have been executed at279 least once. A function is considered to be executed if any of its280 instantiations are executed.281 282* Instantiation coverage is the percentage of function instantiations which283 have been executed at least once. Template functions and static inline284 functions from headers are two kinds of functions which may have multiple285 instantiations. This statistic is hidden by default in reports, but can be286 enabled via the ``-show-instantiation-summary`` option.287 288* Line coverage is the percentage of code lines which have been executed at289 least once. Only executable lines within function bodies are considered to be290 code lines.291 292* Region coverage is the percentage of code regions which have been executed at293 least once. A code region may span multiple lines (e.g in a large function294 body with no control flow). However, it's also possible for a single line to295 contain multiple code regions (e.g in "return x || y && z").296 297* Branch coverage is the percentage of "true" and "false" branches that have298 been taken at least once. Each branch is tied to individual conditions in the299 source code that may each evaluate to either "true" or "false". These300 conditions may comprise larger boolean expressions linked by boolean logical301 operators. For example, "x = (y == 2) || (z < 10)" is a boolean expression302 comprised of two individual conditions, each of which evaluates to303 either true or false, producing four total branch outcomes.304 305* Modified Condition/Decision Coverage (MC/DC) is the percentage of individual306 branch conditions that have been shown to independently affect the decision307 outcome of the boolean expression they comprise. This is accomplished using308 the analysis of executed control flow through the expression (i.e. test309 vectors) to show that as a condition's outcome is varied between "true" and310 false", the decision's outcome also varies between "true" and false", while311 the outcome of all other conditions is held fixed (or they are masked out as312 unevaluatable, as happens in languages whose logical operators have313 short-circuit semantics). MC/DC builds on top of branch coverage and314 requires that all code blocks and all execution paths have been tested. This315 statistic is hidden by default in reports, but it can be enabled via the316 ``-show-mcdc-summary`` option as long as code was also compiled using the317 clang option ``-fcoverage-mcdc``.318 319 * Boolean expressions comprised of only one condition (and therefore320 have no logical operators) are not included in MC/DC analysis and are321 trivially deducible using branch coverage.322 323Of these six statistics, function coverage is usually the least granular while324branch coverage (with MC/DC) is the most granular. 100% branch coverage for a325function implies 100% region coverage for a function. The project-wide totals326for each statistic are listed in the summary.327 328Format compatibility guarantees329===============================330 331* There are no backwards or forwards compatibility guarantees for the raw332 profile format. Raw profiles may be dependent on the specific compiler333 revision used to generate them. It's inadvisable to store raw profiles for334 long periods of time.335 336* Tools must retain **backwards** compatibility with indexed profile formats.337 These formats are not forwards-compatible: i.e, a tool which uses format338 version X will not be able to understand format version (X+k).339 340* Tools must also retain **backwards** compatibility with the format of the341 coverage mappings emitted into instrumented binaries. These formats are not342 forwards-compatible.343 344* The JSON coverage export format has a (major, minor, patch) version triple.345 Only a major version increment indicates a backwards-incompatible change. A346 minor version increment is for added functionality, and patch version347 increments are for bugfixes.348 349Impact of llvm optimizations on coverage reports350================================================351 352llvm optimizations (such as inlining or CFG simplification) should have no353impact on coverage report quality. This is due to the fact that the mapping354from source regions to profile counters is immutable, and is generated before355the llvm optimizer kicks in. The optimizer can't prove that profile counter356instrumentation is safe to delete (because it's not: it affects the profile the357program emits), and so leaves it alone.358 359Note that this coverage feature does not rely on information that can degrade360during the course of optimization, such as debug info line tables.361 362Using the profiling runtime without static initializers363=======================================================364 365By default the compiler runtime uses a static initializer to determine the366profile output path and to register a writer function. To collect profiles367without using static initializers, do this manually:368 369* Export an ``int __llvm_profile_runtime`` symbol from each instrumented shared370 library and executable. When the linker finds a definition of this symbol, it371 knows to skip loading the object which contains the profiling runtime's372 static initializer.373 374* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it375 once from each instrumented executable. This function parses376 ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files377 at that path. To get the same behavior without truncating existing files,378 pass a filename pattern string to ``void __llvm_profile_set_filename(char379 *)``. These calls can be placed anywhere so long as they precede all calls380 to ``__llvm_profile_write_file``.381 382* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write383 out a profile. This function returns 0 on success, and a non-zero value384 otherwise. Calling this function multiple times appends profile data to an385 existing on-disk raw profile.386 387In C++ files, declare these as ``extern "C"``.388 389Using the profiling runtime without a filesystem390------------------------------------------------391 392The profiling runtime also supports freestanding environments that lack a393filesystem. The runtime ships as a static archive that's structured to make394dependencies on a hosted environment optional, depending on what features395the client application uses.396 397The first step is to export ``__llvm_profile_runtime``, as above, to disable398the default static initializers. Instead of calling the ``*_file()`` APIs399described above, use the following to save the profile directly to a buffer400under your control:401 402* Forward-declare ``uint64_t __llvm_profile_get_size_for_buffer(void)`` and403 call it to determine the size of the profile. You'll need to allocate a404 buffer of this size.405 406* Forward-declare ``int __llvm_profile_write_buffer(char *Buffer)`` and call it407 to copy the current counters to ``Buffer``, which is expected to already be408 allocated and big enough for the profile.409 410* Optionally, forward-declare ``void __llvm_profile_reset_counters(void)`` and411 call it to reset the counters before entering a specific section to be412 profiled. This is only useful if there is some setup that should be excluded413 from the profile.414 415In C++ files, declare these as ``extern "C"``.416 417Collecting coverage reports for the llvm project418================================================419 420To prepare a coverage report for llvm (and any of its sub-projects), add421``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the CMake configuration. Raw422profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html423report, run ``llvm/utils/prepare-code-coverage-artifact.py``.424 425To specify an alternate directory for raw profiles, use426``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use427``-DLLVM_PROFILE_MERGE_POOL_SIZE``.428 429Drawbacks and limitations430=========================431 432* Prior to version 2.26, the GNU binutils BFD linker cannot link programs433 compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode. Possible434 workarounds include disabling ``--gc-sections``, upgrading to a newer version435 of BFD, or using the Gold linker.436 437* Code coverage does not handle unpredictable changes in control flow or stack438 unwinding in the presence of exceptions precisely. Consider the following439 function:440 441 .. code-block:: cpp442 443 int f() {444 may_throw();445 return 0;446 }447 448 If the call to ``may_throw()`` propagates an exception into ``f``, the code449 coverage tool may mark the ``return`` statement as executed even though it is450 not. A call to ``longjmp()`` can have similar effects.451 452Clang implementation details453============================454 455This section may be of interest to those wishing to understand or improve456the clang code coverage implementation.457 458Gap regions459-----------460 461Gap regions are source regions with counts. A reporting tool cannot set a line462execution count to the count from a gap region unless that region is the only463one on a line.464 465Gap regions are used to eliminate unnatural artifacts in coverage reports, such466as red "unexecuted" highlights present at the end of an otherwise covered line,467or blue "executed" highlights present at the start of a line that is otherwise468not executed.469 470Branch regions471--------------472When viewing branch coverage details in source-based file-level sub-views using473``--show-branches``, it is recommended that users show all macro expansions474(using option ``--show-expansions``) since macros may contain hidden branch475conditions. The coverage summary report will always include these macro-based476boolean expressions in the overall branch coverage count for a function or477source file.478 479Branch coverage is not tracked for constant folded branch conditions since480branches are not generated for these cases. In the source-based file-level481sub-view, these branches will simply be shown as ``[Folded - Ignored]`` so that482users are informed about what happened.483 484Branch coverage is tied directly to branch-generating conditions in the source485code. Users should not see hidden branches that aren't actually tied to the486source code.487 488MC/DC Instrumentation489---------------------490 491When instrumenting for Modified Condition/Decision Coverage (MC/DC) using the492clang option ``-fcoverage-mcdc``, there are two hard limits.493 494The maximum number of terms is limited to 32767, which is practical for495handwritten expressions. To be more restrictive in order to enforce coding rules,496use ``-Xclang -fmcdc-max-conditions=n``. Expressions with exceeded condition497counts ``n`` will generate warnings and will be excluded in the MC/DC coverage.498 499The number of test vectors (the maximum number of possible combinations of500expressions) is limited to 2,147,483,646. In this case, approximately501256MiB (==2GiB/8) is used to record test vectors.502 503To reduce memory usage, users can limit the maximum number of test vectors per504expression with ``-Xclang -fmcdc-max-test-vectors=m``.505If the number of test vectors resulting from the analysis of an expression506exceeds ``m``, a warning will be issued and the expression will be excluded507from the MC/DC coverage.508 509The number of test vectors ``m``, for ``n`` terms in an expression, can be510``m <= 2^n`` in the theoretical worst case, but is usually much smaller.511In simple cases, such as expressions consisting of a sequence of single512operators, ``m == n+1``. For example, ``(a && b && c && d && e && f && g)``513requires 8 test vectors.514 515Expressions such as ``((a0 && b0) || (a1 && b1) || ...)`` can cause the516number of test vectors to increase exponentially.517 518Also, if a boolean expression is embedded in the nest of another boolean519expression but separated by a non-logical operator, this is also not supported.520For example, in ``x = (a && b && c && func(d && f))``, the ``d && f`` case521starts a new boolean expression that is separated from the other conditions by522the operator ``func()``. When this is encountered, a warning will be generated523and the boolean expression will not be instrumented.524 525Switch statements526-----------------527 528The region mapping for a switch body consists of a gap region that covers the529entire body (starting from the '{' in 'switch (...) {', and terminating where the530last case ends). This gap region has a zero count: this causes "gap" areas in531between case statements, which contain no executable code, to appear uncovered.532 533When a switch case is visited, the parent region is extended: if the parent534region has no start location, its start location becomes the start of the case.535This is used to support switch statements without a ``CompoundStmt`` body, in536which the switch body and the single case share a count.537 538For switches with ``CompoundStmt`` bodies, a new region is created at the start539of each switch case.540 541Branch regions are also generated for each switch case, including the default542case. If there is no explicitly defined default case in the source code, a543branch region is generated to correspond to the implicit default case that is544generated by the compiler. The implicit branch region is tied to the line and545column number of the switch statement condition since no source code for the546implicit case exists.547