1104 lines · plain
1=================================2LLVM Testing Infrastructure Guide3=================================4 5.. contents::6 :local:7 8.. toctree::9 :hidden:10 11 TestSuiteGuide12 TestSuiteMakefileGuide13 14Overview15========16 17This document is the reference manual for the LLVM testing18infrastructure. It documents the structure of the LLVM testing19infrastructure, the tools needed to use it, and how to add and run20tests.21 22Requirements23============24 25In order to use the LLVM testing infrastructure, you will need all of the26software required to build LLVM, as well as `Python <http://python.org>`_ 3.8 or27later.28 29LLVM Testing Infrastructure Organization30========================================31 32The LLVM testing infrastructure contains three major categories of tests:33unit tests, regression tests, and whole programs. The unit tests and regression34tests are contained inside the LLVM repository itself under ``llvm/unittests``35and ``llvm/test`` respectively and are expected to always pass. They should be36run before every commit.37 38The whole-program tests are referred to as the "LLVM test suite" (or39"test-suite") and are in the ``test-suite``40`repository on GitHub <https://github.com/llvm/llvm-test-suite.git>`_.41For historical reasons, these tests are also referred to as the "nightly42tests" in places, which is less ambiguous than "test-suite" and remains43in use although we run them much more often than nightly.44 45Unit tests46----------47 48Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/docs/primer.md>`_49and `Google Mock <https://github.com/google/googletest/blob/master/docs/gmock_for_dummies.md>`_50and are located in the ``llvm/unittests`` directory.51In general, unit tests are reserved for targeting the support library and other52generic data structure. We prefer relying on regression tests for testing53transformations and analysis on the IR.54 55Regression tests56----------------57 58The regression tests are small pieces of code that test a specific59feature of LLVM or trigger a specific bug in LLVM. The language they are60written in depends on the part of LLVM being tested. These tests are driven by61the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and62are located in the ``llvm/test`` directory.63 64Typically, when a bug is found in LLVM, a regression test containing just65enough code to reproduce the problem should be written and placed66somewhere underneath this directory. For example, it can be a small67piece of LLVM IR distilled from an actual application or benchmark.68 69Testing Analysis70----------------71 72An analysis is a pass to infer properties on some part of the IR without73transforming it. They are tested in general using the same infrastructure as the74regression tests, by creating a separate "Printer" pass to consume the analysis75result and print it on the standard output in a textual format suitable for76FileCheck.77See `llvm/test/Analysis/BranchProbabilityInfo/loop.ll <https://github.com/llvm/llvm-project/blob/main/llvm/test/Analysis/BranchProbabilityInfo/loop.ll>`_78for an example of such test.79 80``test-suite``81--------------82 83The test suite contains whole programs, which are pieces of code which84can be compiled and linked into a stand-alone program that can be85executed. These programs are generally written in high-level languages,86such as C and C++.87 88These programs are compiled using a user-specified compiler and set of89flags, and then executed to capture the program output and timing90information. The output of these programs is compared to a reference91output to ensure that the program is being compiled correctly.92 93In addition to compiling and executing programs, whole-program tests94serve as a way of benchmarking LLVM performance, both in terms of the95efficiency of the programs generated as well as the speed with which96LLVM compiles, optimizes, and generates code.97 98The test-suite is located in the ``test-suite``99`repository on GitHub <https://github.com/llvm/llvm-test-suite.git>`_.100 101See the :doc:`TestSuiteGuide` for details.102 103Debugging Information tests104---------------------------105 106The test suite contains tests to check the quality of debugging information.107The tests are written in C-based languages or in LLVM assembly language.108 109These tests are compiled and run under a debugger. The debugger output110is checked to validate the debugging information. See ``README.txt`` in the111test suite for more information. This test suite is located in the112``cross-project-tests/debuginfo-tests`` directory.113 114Quick start115===========116 117The tests are located in two separate repositories. The unit and118regression tests are in the main "llvm"/ directory under the directories119``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the120main LLVM tree). Use ``make check-all`` to run the unit and regression tests121after building LLVM.122 123The ``test-suite`` module contains more comprehensive tests including whole C124and C++ programs. See the :doc:`TestSuiteGuide` for details.125 126Unit and Regression tests127-------------------------128 129To run all of the LLVM unit tests, use the ``check-llvm-unit`` target:130 131.. code-block:: bash132 133 % make check-llvm-unit134 135To run all of the LLVM regression tests, use the ``check-llvm`` target:136 137.. code-block:: bash138 139 % make check-llvm140 141In order to get reasonable testing performance, build LLVM and subprojects142in release mode, i.e.,143 144.. code-block:: bash145 146 % cmake -DCMAKE_BUILD_TYPE="Release" -DLLVM_ENABLE_ASSERTIONS=On147 148If you have `Clang <https://clang.llvm.org/>`_ checked out and built, you149can run the LLVM and Clang tests simultaneously using:150 151.. code-block:: bash152 153 % make check-all154 155To run the tests with Valgrind (Memcheck by default), use the ``LIT_OPTS`` make156variable to pass the required options to lit. For example, you can use:157 158.. code-block:: bash159 160 % make check LIT_OPTS="-v --vg --vg-leak"161 162to enable testing with Valgrind and with leak checking enabled.163 164To run individual tests or subsets of tests, you can use the ``llvm-lit``165script which is built as part of LLVM. For example, to run the166``Integer/BitPacked.ll`` test by itself, you can run:167 168.. code-block:: bash169 170 % llvm-lit <path to llvm-project>/llvm/test/Integer/BitPacked.ll171 172.. note::173 The test files are in the ``llvm-project`` directory, not the directory you174 are building LLVM in.175 176Or you can run a whole folder of tests. To run all of the ARM CodeGen tests:177 178.. code-block:: bash179 180 % llvm-lit <path to llvm-project>/llvm/test/CodeGen/ARM181 182The regression tests will use the Python psutil module only if installed in a183**non-user** location. Under Linux, install with sudo or within a virtual184environment. Under Windows, install Python for all users and then run185``pip install psutil`` in an elevated command prompt.186 187For more information on using the :program:`lit` tool, see ``llvm-lit --help``188or the :doc:`lit man page <CommandGuide/lit>`.189 190Debugging Information tests191---------------------------192 193To run debugging information tests simply add the ``cross-project-tests``194project to your ``LLVM_ENABLE_PROJECTS`` define on the cmake195command-line.196 197Regression test structure198=========================199 200The LLVM regression tests are driven by :program:`lit` and are located in the201``llvm/test`` directory.202 203This directory contains a large array of small tests that exercise204various features of LLVM and to ensure that regressions do not occur.205The directory is broken into several subdirectories, each focused on a206particular area of LLVM.207 208Writing new regression tests209----------------------------210 211The regression test structure is very simple but does require some212information to be set. This information is gathered via ``cmake``213and is written to a file, ``test/lit.site.cfg.py`` in the build directory.214The ``llvm/test`` Makefile does this work for you.215 216In order for the regression tests to work, each directory of tests must217have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine218how to run the tests. This file is just Python code and thus is very219flexible, but we've standardized it for the LLVM regression tests. If220you're adding a directory of tests, just copy ``lit.local.cfg`` from221another directory to get running. The standard ``lit.local.cfg`` simply222specifies which files to look in for tests. Any directory that contains223only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit224documentation <CommandGuide/lit>` for more information.225 226Each test file must contain lines starting with "RUN:" that tell :program:`lit`227how to run it. If there are no ``RUN`` lines, :program:`lit` will issue an error228while running a test.229 230``RUN`` lines are specified in the comments of the test program using the231keyword ``RUN`` followed by a colon, and lastly the command (pipeline)232to execute. Together, these lines form the "script" that :program:`lit`233executes to run the test case. The syntax of the ``RUN`` lines is similar to a234shell's syntax for pipelines including I/O redirection and variable235substitution. However, even though these lines may *look* like a shell236script, they are not. ``RUN`` lines are interpreted by :program:`lit`.237Consequently, the syntax differs from shell in a few ways. You can specify238as many ``RUN`` lines as needed.239 240:program:`lit` performs substitution on each ``RUN`` line to replace LLVM tool names241with the full paths to the executable built for each tool (in242``$(LLVM_OBJ_ROOT)/bin``). This ensures that :program:`lit` does243not invoke any stray LLVM tools in the user's path during testing.244 245Each ``RUN`` line is executed on its own, distinct from other lines unless246its last character is ``\``. This continuation character causes the ``RUN``247line to be concatenated with the next one. In this way, you can build up248long pipelines of commands without making huge line lengths. The lines249ending in ``\`` are concatenated until a ``RUN`` line that doesn't end in250``\`` is found. This concatenated set of ``RUN`` lines then constitutes one251execution. :program:`lit` will substitute variables and arrange for the pipeline252to be executed. If any process in the pipeline fails, the entire line (and253test case) fails too.254 255Below is an example of legal ``RUN`` lines in a ``.ll`` file:256 257.. code-block:: llvm258 259 ; RUN: llvm-as < %s | llvm-dis > %t1260 ; RUN: llvm-dis < %s.bc-13 > %t2261 ; RUN: diff %t1 %t2262 263As with a Unix shell, the ``RUN`` lines permit pipelines and I/O264redirection to be used.265 266There are some quoting rules that you must pay attention to when writing267your ``RUN`` lines. In general, nothing needs to be quoted. :program:`lit` won't268strip off any quote characters, so they will get passed to the invoked program.269To avoid this use curly braces to tell :program:`lit` that it should treat270everything enclosed as one value.271 272In general, you should strive to keep your ``RUN`` lines as simple as possible,273using them only to run tools that generate textual output you can then examine.274The recommended way to examine output to figure out if the test passes is using275the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in ``RUN``276lines is deprecated - please do not send or commit patches that use it.]*277 278Put related tests into a single file rather than having a separate file per279test. Check if there are files already covering your feature and consider280adding your code there instead of creating a new file.281 282Generating assertions in regression tests283-----------------------------------------284 285Some regression test cases are very large and complex to write/update by hand.286In that case, to reduce the manual work, we can use the scripts available in287``llvm/utils/`` to generate the assertions.288 289For example, to generate assertions in an :program:`llc`-based test, after290adding one or more ``RUN`` lines, use:291 292 .. code-block:: bash293 294 % llvm/utils/update_llc_test_checks.py --llc-binary build/bin/llc test.ll295 296This will generate FileCheck assertions, and insert a ``NOTE:`` line at the297top to indicate that assertions were automatically generated.298 299If you want to update assertions in an existing test case, pass the `-u` option300which first checks the ``NOTE:`` line exists and matches the script name.301 302Sometimes, a test absolutely depends on hand-written assertions and should not303have assertions automatically generated. In that case, add the text ``NOTE: Do304not autogenerate`` to the first line, and the scripts will skip that test. It305is a good idea to explain why generated assertions will not work for the test306so future developers will understand what is going on.307 308These are the most common scripts and their purposes/applications in generating309assertions:310 311.. code-block:: none312 313 update_analyze_test_checks.py314 opt -passes='print<cost-model>'315 316 update_cc_test_checks.py317 C/C++, or clang/clang++ (IR checks)318 319 update_llc_test_checks.py320 llc (assembly checks)321 322 update_mca_test_checks.py323 llvm-mca324 325 update_mir_test_checks.py326 llc (MIR checks)327 328 update_test_checks.py329 opt330 331Precommit workflow for tests332----------------------------333 334If the test does not crash, assert, or infinite loop, commit the test with335baseline check-lines first. That is, the test will show a miscompile or336missing optimization. Add a "TODO" or "FIXME" comment to indicate that337something is expected to change in a test.338 339A follow-up patch with code changes to the compiler will then show check-line340differences to the tests, so it is easier to see the effect of the patch.341Remove TODO/FIXME comments added in the previous step if a problem is solved.342 343Baseline tests (no-functional-change or NFC patch) may be pushed to main344without pre-commit review if you have commit access.345 346Best practices for regression tests347-----------------------------------348 349- Use auto-generated check lines (produced by the scripts mentioned above)350 whenever feasible.351- Include comments about what is tested/expected in a particular test. If there352 are relevant issues in the bug tracker, add references to those bug reports353 (for example, "See PR999 for more details").354- Avoid undefined behavior and poison/undef values unless necessary. For355 example, do not use patterns like ``br i1 undef``, which are likely to break356 as a result of future optimizations.357- Minimize tests by removing unnecessary instructions, metadata, attributes,358 etc. Tools like ``llvm-reduce`` can help automate this.359- Outside PhaseOrdering tests, only run a minimal set of passes. For example,360 prefer ``opt -S -passes=instcombine`` over ``opt -S -O3``.361- Avoid unnamed instructions/blocks (such as ``%0`` or ``1:``), because they may362 require renumbering on future test modifications. These can be removed by363 running the test through ``opt -S -passes=instnamer``.364- Try to give values (including variables, blocks and functions) meaningful365 names, and avoid retaining complex names generated by the optimization366 pipeline (such as ``%foo.0.0.0.0.0.0``).367 368Extra files369-----------370 371If your test requires extra files besides the file containing the ``RUN:`` lines,372and the extra files are small, consider specifying them in the same file and373using ``split-file`` to extract them. For example,374 375.. code-block:: llvm376 377 ; RUN: split-file %s %t378 ; RUN: llvm-link -S %t/a.ll %t/b.ll | FileCheck %s379 380 ; CHECK: ...381 382 ;--- a.ll383 ...384 ;--- b.ll385 ...386 387The parts are separated by the regex ``^(.|//)--- <part>``.388 389If you want to test relative line numbers like ``[[#@LINE+1]]``, specify390``--leading-lines`` to add leading empty lines to preserve line numbers.391 392If the extra files are large, the idiomatic place to put them is in a subdirectory ``Inputs``.393You can then refer to the extra files as ``%S/Inputs/foo.bar``.394 395For example, consider ``test/Linker/ident.ll``. The directory structure is396as follows::397 398 test/399 Linker/400 ident.ll401 Inputs/402 ident.a.ll403 ident.b.ll404 405For convenience, these are the contents:406 407.. code-block:: llvm408 409 ;;;;; ident.ll:410 411 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s412 413 ; Verify that multiple input llvm.ident metadata are linked together.414 415 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}416 ; CHECK-DAG: "Compiler V1"417 ; CHECK-DAG: "Compiler V2"418 ; CHECK-DAG: "Compiler V3"419 420 ;;;;; Inputs/ident.a.ll:421 422 !llvm.ident = !{!0, !1}423 !0 = metadata !{metadata !"Compiler V1"}424 !1 = metadata !{metadata !"Compiler V2"}425 426 ;;;;; Inputs/ident.b.ll:427 428 !llvm.ident = !{!0}429 !0 = metadata !{metadata !"Compiler V3"}430 431For symmetry, ``ident.ll`` is just a dummy file that doesn't432actually participate in the test besides holding the ``RUN:`` lines.433 434.. note::435 436 Some existing tests use ``RUN: true`` in extra files instead of just437 putting the extra files in an ``Inputs/`` directory. This pattern is438 deprecated.439 440Elaborated tests441----------------442 443Generally, IR and assembly test files benefit from being cleaned to remove444unnecessary details. However, for tests requiring elaborate IR or assembly445files where cleanup is less practical (e.g., a large amount of debug information446output from Clang), you can include generation instructions within447``split-file`` part called ``gen``. Then, run448``llvm/utils/update_test_body.py`` on the test file to generate the needed449content.450 451.. code-block:: none452 453 ; RUN: rm -rf %t && split-file %s %t && cd %t454 ; RUN: opt -S a.ll ... | FileCheck %s455 456 ; CHECK: hello457 458 ;--- a.cc459 int va;460 ;--- gen461 clang --target=x86_64-linux -S -emit-llvm -g a.cc -o -462 463 ;--- a.ll464 # content generated by the script 'gen'465 466.. code-block:: bash467 468 PATH=/path/to/clang_build/bin:$PATH llvm/utils/update_test_body.py path/to/test.ll469 470The script will prepare extra files with ``split-file``, invoke ``gen``, and471then rewrite the part after ``gen`` with its stdout.472 473For convenience, if the test needs a single assembly file, you can also wrap474``gen`` and its required files with ``.ifdef`` and ``.endif``. Then you can475skip ``split-file`` in ``RUN`` lines.476 477.. code-block:: none478 479 # RUN: llvm-mc -filetype=obj -triple=x86_64 %s -o a.o480 # RUN: ... | FileCheck %s481 482 # CHECK: hello483 484 .ifdef GEN485 #--- a.cc486 int va;487 #--- gen488 clang --target=x86_64-linux -S -g a.cc -o -489 .endif490 # content generated by the script 'gen'491 492.. note::493 494 Consider specifying an explicit target triple to avoid differences when495 regeneration is needed on another machine.496 497 ``gen`` is invoked with ``PWD`` set to ``/proc/self/cwd``. Clang commands498 don't need ``-fdebug-compilation-dir=`` since its default value is ``PWD``.499 500 Check prefixes should be placed before ``.endif`` since the part after501 ``.endif`` is replaced.502 503If the test body contains multiple files, you can print ``---`` separators and504utilize ``split-file`` in ``RUN`` lines.505 506.. code-block:: none507 508 # RUN: rm -rf %t && split-file %s %t && cd %t509 ...510 511 #--- a.cc512 int va;513 #--- b.cc514 int vb;515 #--- gen516 clang --target=x86_64-linux -S -O1 -g a.cc -o -517 echo '#--- b.s'518 clang --target=x86_64-linux -S -O1 -g b.cc -o -519 #--- a.s520 521Fragile tests522-------------523 524It is easy to write a fragile test that could fail spuriously if the tool being525tested outputs a full path to the input file. For example, :program:`opt` by526default outputs a ``ModuleID``:527 528.. code-block:: console529 530 $ cat example.ll531 define i32 @main() nounwind {532 ret i32 0533 }534 535 $ opt -S /path/to/example.ll536 ; ModuleID = '/path/to/example.ll'537 538 define i32 @main() nounwind {539 ret i32 0540 }541 542``ModuleID`` can unexpectedly match against ``CHECK`` lines. For example:543 544.. code-block:: llvm545 546 ; RUN: opt -S %s | FileCheck547 548 define i32 @main() nounwind {549 ; CHECK-NOT: load550 ret i32 0551 }552 553This test will fail if placed into a ``download`` directory.554 555To make your tests robust, always use ``opt ... < %s`` in the ``RUN`` line.556:program:`opt` does not output a ``ModuleID`` when input comes from stdin.557 558Platform-Specific Tests559-----------------------560 561Whenever adding tests that require the knowledge of a specific platform,562either related to code generated, specific output or back-end features,563you must isolate the features, so that buildbots that564run on different architectures (and don't even compile all back-ends),565don't fail.566 567The first problem is to check for target-specific output, for example sizes568of structures, paths and architecture names, for example:569 570* Tests containing Windows paths will fail on Linux and vice versa.571* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.572* Tests where the debug information calculates the size of types and structures.573 574Also, if the test relies on any behaviour that is coded in any back-end, it must575go in its own directory. So, for instance, code generator tests for ARM go576into ``test/CodeGen/ARM`` and so on. Those directories contain a special577``lit`` configuration file that ensures all tests in that directory will578only run if a specific back-end is compiled and available.579 580For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:581 582.. code-block:: python583 584 config.suffixes = ['.ll', '.c', '.cpp', '.test']585 if not 'ARM' in config.root.targets:586 config.unsupported = True587 588Other platform-specific tests are those that depend on a specific feature589of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.590 591For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture592variants:593 594.. code-block:: llvm595 596 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2597 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1598 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2599 600And the checks are different:601 602.. code-block:: llvm603 604 ; SSE2: @test1605 ; SSE2: psubusw LCPI0_0(%rip), %xmm0606 ; AVX1: @test1607 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0608 ; AVX2: @test1609 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0610 611So, if you're testing for a behaviour that you know is platform-specific or612depends on special features of sub-architectures, you must add the specific613triple, test with the specific FileCheck and put it into the specific614directory that will filter out all other architectures.615 616 617Constraining test execution618---------------------------619 620Some tests can be run only in specific configurations, such as621with debug builds or on particular platforms. Use ``REQUIRES``622and ``UNSUPPORTED`` to control when the test is enabled.623 624Some tests are expected to fail. For example, there may be a known bug625that the test detects. Use ``XFAIL`` to mark a test as an expected failure.626An ``XFAIL`` test will be successful if its execution fails, and627will be a failure if its execution succeeds.628 629.. code-block:: llvm630 631 ; This test will be only enabled in the build with asserts.632 ; REQUIRES: asserts633 ; This test is disabled when running on Linux.634 ; UNSUPPORTED: system-linux635 ; This test is expected to fail when targeting PowerPC.636 ; XFAIL: target=powerpc{{.*}}637 638``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated639list of boolean expressions. The values in each expression may be:640 641- Features added to ``config.available_features`` by configuration files such as ``lit.cfg``.642 String comparison of features is case-sensitive. Furthermore, a boolean expression can643 contain any Python regular expression enclosed in ``{{ }}``, in which case the boolean644 expression is satisfied if any feature matches the regular expression. Regular645 expressions can appear inside an identifier, so for example ``he{{l+}}o`` would match646 ``helo``, ``hello``, ``helllo``, and so on.647- The default target triple, preceded by the string ``target=`` (for example,648 ``target=x86_64-pc-windows-msvc``). Typically, regular expressions are used649 to match parts of the triple (for example, ``target={{.*}}-windows{{.*}}``650 to match any Windows target triple).651 652| ``REQUIRES`` enables the test if all expressions are true.653| ``UNSUPPORTED`` disables the test if any expression is true.654| ``XFAIL`` expects the test to fail if any expression is true.655 656Use, ``XFAIL: *`` if the test is expected to fail everywhere. Similarly, use657``UNSUPPORTED: target={{.*}}`` to disable the test everywhere.658 659.. code-block:: llvm660 661 ; This test is disabled when running on Windows,662 ; and is disabled when targeting Linux, except for Android Linux.663 ; UNSUPPORTED: system-windows, target={{.*linux.*}} && !target={{.*android.*}}664 ; This test is expected to fail when targeting PowerPC or running on Darwin.665 ; XFAIL: target=powerpc{{.*}}, system-darwin666 667 668Tips for writing constraints669----------------------------670 671**``REQUIRES`` and ``UNSUPPORTED``**672 673These are logical inverses. In principle, ``UNSUPPORTED`` isn't absolutely674necessary (the logical negation could be used with ``REQUIRES`` to get675exactly the same effect), but it can make these clauses easier to read and676understand. Generally, people use ``REQUIRES`` to state things that the test677depends on to operate correctly, and ``UNSUPPORTED`` to exclude cases where678the test is expected never to work.679 680**``UNSUPPORTED`` and ``XFAIL``**681 682Both of these indicate that the test isn't expected to work; however, they683have different effects. ``UNSUPPORTED`` causes the test to be skipped;684this saves execution time, but then you'll never know whether the test685actually would start working. Conversely, ``XFAIL`` actually runs the test686but expects a failure output, taking extra execution time but alerting you687if/when the test begins to behave correctly (an ``XPASS`` test result). You688need to decide which is more appropriate in each case.689 690**Using ``target=...``**691 692Checking the target triple can be tricky; it's easy to mis-specify. For693example, ``target=mips{{.*}}`` will match not only mips, but also mipsel,694mips64, and mips64el. ``target={{.*}}-linux-gnu`` will match695x86_64-unknown-linux-gnu, but not armv8l-unknown-linux-gnueabihf.696Prefer to use hyphens to delimit triple components (``target=mips-{{.*}}``)697and it's generally a good idea to use a trailing wildcard to allow for698unexpected suffixes.699 700Also, it's generally better to write regular expressions that use entire701triple components than to do something clever to shorten them. For702example, to match both freebsd and netbsd in an expression, you could write703``target={{.*(free|net)bsd.*}}`` and that would work. However, it would704prevent a ``grep freebsd`` from finding this test. Better to use:705``target={{.+-freebsd.*}} || target={{.+-netbsd.*}}``706 707 708Substitutions709-------------710 711Besides replacing LLVM tool names, the following substitutions are performed in712``RUN`` lines:713 714``%%``715 Replaced by a single ``%``. This allows escaping other substitutions.716 717``%s``718 File path to the test case's source. This is suitable for passing on the719 command line as the input to an LLVM tool.720 721 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``722 723``%S``724 Directory path to the test case's source.725 726 Example: ``/home/user/llvm/test/MC/ELF``727 728``%t``729 File path to a temporary file name that can be used for this test case.730 The file name won't conflict with other test cases. You can append to it731 if you need multiple temporaries. This is useful as the destination of732 some redirected output.733 734 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``735 736``%T``737 Directory of ``%t``. Deprecated. Shouldn't be used, because it can be easily738 misused and cause race conditions between tests.739 740 Use ``rm -rf %t && mkdir %t`` instead if a temporary directory is necessary.741 742 Example: ``/home/user/llvm.build/test/MC/ELF/Output``743 744``%{pathsep}``745 746 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows).747 748``%{fs-src-root}``749 Expands to the root component of file system paths for the source directory,750 i.e. ``/`` on Unix systems or ``C:\`` (or another drive) on Windows.751 752``%{fs-tmp-root}``753 Expands to the root component of file system paths for the test's temporary754 directory, i.e. ``/`` on Unix systems or ``C:\`` (or another drive) on755 Windows.756 757``%{fs-sep}``758 Expands to the file system separator, i.e. ``/`` or ``\`` on Windows.759 760``%/s, %/S, %/t, %/T``761 762 Act like the corresponding substitution above but replace any ``\``763 character with a ``/``. This is useful to normalize path separators.764 765 Example: ``%s: C:\Desktop Files/foo_test.s.tmp``766 767 Example: ``%/s: C:/Desktop Files/foo_test.s.tmp``768 769``%{s:real}, %{S:real}, %{t:real}, %{T:real}``770``%{/s:real}, %{/S:real}, %{/t:real}, %{/T:real}``771 772 Act like the corresponding substitution, including with ``/``, but use773 the real path by expanding all symbolic links and substitute drives.774 775 Example: ``%s: S:\foo_test.s.tmp``776 777 Example: ``%{/s:real}: C:/SDrive/foo_test.s.tmp``778 779``%:s, %:S, %:t, %:T``780 781 Act like the corresponding substitution above but remove colons at782 the beginning of Windows paths. This is useful to allow concatenation783 of absolute paths on Windows to produce a legal path.784 785 Example: ``%s: C:\Desktop Files\foo_test.s.tmp``786 787 Example: ``%:s: C\Desktop Files\foo_test.s.tmp``788 789``%errc_<ERRCODE>``790 791 Some error messages may be substituted to allow different spellings792 based on the host platform.793 794 The following error codes are currently supported:795 ENOENT, EISDIR, EINVAL, EACCES.796 797 Example: ``Linux %errc_ENOENT: No such file or directory``798 799 Example: ``Windows %errc_ENOENT: no such file or directory``800 801``%if feature %{<if branch>%} %else %{<else branch>%}``802 803 Conditional substitution: if ``feature`` is available it expands to804 ``<if branch>``, otherwise it expands to ``<else branch>``.805 ``%else %{<else branch>%}`` is optional and treated like ``%else %{%}``806 if not present.807 808``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``809 810 The number of the line where this substitution is used, with an811 optional integer offset. These expand only if they appear812 immediately in ``RUN:``, ``DEFINE:``, and ``REDEFINE:`` directives.813 Occurrences in substitutions defined elsewhere are never expanded.814 For example, this can be used in tests with multiple ``RUN`` lines,815 which reference the test file's line numbers.816 817**LLVM-specific substitutions:**818 819``%shlibext``820 The suffix for the host platforms shared library files. This includes the821 period as the first character.822 823 Example: ``.so`` (Linux), ``.dylib`` (macOS), ``.dll`` (Windows)824 825``%exeext``826 The suffix for the host platforms executable files. This includes the827 period as the first character.828 829 Example: ``.exe`` (Windows), empty on Linux.830 831**Clang-specific substitutions:**832 833``%clang``834 Invokes the Clang driver.835 836``%clang_cpp``837 Invokes the Clang driver as the preprocessor.838 839``%clang_cl``840 Invokes the CL-compatible Clang driver.841 842``%clangxx``843 Invokes the G++-compatible Clang driver.844 845``%clang_cc1``846 Invokes the Clang frontend.847 848``%itanium_abi_triple``, ``%ms_abi_triple``849 These substitutions can be used to get the current target triple adjusted to850 the desired ABI. For example, if the test suite is running with the851 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to852 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without853 constraining it to a specific triple.854 855**FileCheck-specific substitutions:**856 857``%ProtectFileCheckOutput``858 This should precede a ``FileCheck`` call if and only if the call's textual859 output affects test results. It's usually easy to tell: just look for860 redirection or piping of the ``FileCheck`` call's stdout or stderr.861 862.. _Test-specific substitutions:863 864**Test-specific substitutions:**865 866Additional substitutions can be defined as follows:867 868- Lit configuration files (e.g., ``lit.cfg`` or ``lit.local.cfg``) can define869 substitutions for all tests in a test directory. They do so by extending the870 substitution list, ``config.substitutions``. Each item in the list is a tuple871 consisting of a pattern and its replacement, which lit applies as plain text872 (even if it contains sequences that Python's ``re.sub`` considers to be873 escape sequences).874- To define substitutions within a single test file, lit supports the875 ``DEFINE:`` and ``REDEFINE:`` directives, described in detail below. So that876 they have no effect on other test files, these directives modify a copy of the877 substitution list that is produced by lit configuration files.878 879For example, the following directives can be inserted into a test file to define880``%{cflags}`` and ``%{fcflags}`` substitutions with empty initial values, which881serve as the parameters of another newly defined ``%{check}`` substitution:882 883.. code-block:: llvm884 885 ; DEFINE: %{cflags} =886 ; DEFINE: %{fcflags} =887 888 ; DEFINE: %{check} = \889 ; DEFINE: %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags} \890 ; DEFINE: -emit-llvm -o - %s | \891 ; DEFINE: FileCheck %{fcflags} %s892 893Alternatively, the above substitutions can be defined in a lit configuration894file to be shared with other test files. Either way, the test file can then895specify directives like the following to redefine the parameter substitutions as896desired before each use of ``%{check}`` in a ``RUN:`` line:897 898.. code-block:: llvm899 900 ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0 -fopenmp-simd901 ; REDEFINE: %{fcflags} = -check-prefix=SIMD902 ; RUN: %{check}903 904 ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu -fopenmp-simd905 ; REDEFINE: %{fcflags} = -check-prefix=SIMD906 ; RUN: %{check}907 908 ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0909 ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD910 ; RUN: %{check}911 912 ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu913 ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD914 ; RUN: %{check}915 916Besides providing initial values, the initial ``DEFINE:`` directives for the917parameter substitutions in the above example serve a second purpose: they918establish the substitution order so that both ``%{check}`` and its parameters919expand as desired. There's a simple way to remember the required definition920order in a test file: define a substitution before any substitution that might921refer to it.922 923In general, substitution expansion behaves as follows:924 925- Upon arriving at each ``RUN:`` line, lit expands all substitutions in that926 ``RUN:`` line using their current values from the substitution list. No927 substitution expansion is performed immediately at ``DEFINE:`` and928 ``REDEFINE:`` directives except ``%(line)``, ``%(line+<number>)``, and929 ``%(line-<number>)``.930- When expanding substitutions in a ``RUN:`` line, lit makes only one pass931 through the substitution list by default. In this case, a substitution must932 have been inserted earlier in the substitution list than any substitution933 appearing in its value in order for the latter to expand. (For greater934 flexibility, you can enable multiple passes through the substitution list by935 setting `recursiveExpansionLimit`_ in a lit configuration file.)936- While lit configuration files can insert anywhere in the substitution list,937 the insertion behavior of the ``DEFINE:`` and ``REDEFINE:`` directives is938 specified below and is designed specifically for the use case presented in the939 example above.940- Defining a substitution in terms of itself, whether directly or via other941 substitutions, should be avoided. It usually produces an infinitely recursive942 definition that cannot be fully expanded. It does *not* define the943 substitution in terms of its previous value, even when using ``REDEFINE:``.944 945The relationship between the ``DEFINE:`` and ``REDEFINE:`` directive is946analogous to the relationship between a variable declaration and variable947assignment in many programming languages:948 949- ``DEFINE: %{name} = value``950 951 This directive assigns the specified value to a new substitution whose952 pattern is ``%{name}``, or it reports an error if there is already a953 substitution whose pattern contains ``%{name}`` because that could produce954 confusing expansions (e.g., a lit configuration file might define a955 substitution with the pattern ``%{name}\[0\]``). The new substitution is956 inserted at the start of the substitution list so that it will expand first.957 Thus, its value can contain any substitution previously defined, whether in958 the same test file or in a lit configuration file, and both will expand.959 960- ``REDEFINE: %{name} = value``961 962 This directive assigns the specified value to an existing substitution whose963 pattern is ``%{name}``, or it reports an error if there are no substitutions964 with that pattern or if there are multiple substitutions whose patterns965 contain ``%{name}``. The substitution's current position in the substitution966 list does not change so that expansion order relative to other existing967 substitutions is preserved.968 969The following properties apply to both the ``DEFINE:`` and ``REDEFINE:``970directives:971 972- **Substitution name**: In the directive, whitespace immediately before or973 after ``%{name}`` is optional and discarded. ``%{name}`` must start with974 ``%{``, it must end with ``}``, and the rest must start with a letter or975 underscore and contain only alphanumeric characters, hyphens, underscores, and976 colons. This syntax has a few advantages:977 978 - It is impossible for ``%{name}`` to contain sequences that are special in979 Python's ``re.sub`` patterns. Otherwise, attempting to specify980 ``%{name}`` as a substitution pattern in a lit configuration file could981 produce confusing expansions.982 - The braces help avoid the possibility that another substitution's pattern983 will match part of ``%{name}`` or vice-versa, producing confusing984 expansions. However, the patterns of substitutions defined by lit985 configuration files and by lit itself are not restricted to this form, so986 overlaps are still theoretically possible.987 988- **Substitution value**: The value includes all text from the first989 non-whitespace character after ``=`` to the last non-whitespace character. If990 there is no non-whitespace character after ``=``, the value is the empty991 string. Escape sequences that can appear in Python ``re.sub`` replacement992 strings are treated as plain text in the value.993- **Line continuations**: If the last non-whitespace character on the line after994 ``:`` is ``\``, then the next directive must use the same directive keyword995 (e.g., ``DEFINE:``) , and it is an error if there is no additional directive.996 That directive serves as a continuation. That is, before following the rules997 above to parse the text after ``:`` in either directive, lit joins that text998 together to form a single directive, replaces the ``\`` with a single space,999 and removes any other whitespace that is now adjacent to that space. A1000 continuation can be continued in the same manner. A continuation containing1001 only whitespace after its ``:`` is an error.1002 1003.. _recursiveExpansionLimit:1004 1005**recursiveExpansionLimit:**1006 1007As described in the previous section, when expanding substitutions in a ``RUN:``1008line, lit makes only one pass through the substitution list by default. Thus,1009if substitutions are not defined in the proper order, some will remain in the1010``RUN:`` line unexpanded. For example, the following directives refer to1011``%{inner}`` within ``%{outer}`` but do not define ``%{inner}`` until after1012``%{outer}``:1013 1014.. code-block:: llvm1015 1016 ; By default, this definition order does not enable full expansion.1017 1018 ; DEFINE: %{outer} = %{inner}1019 ; DEFINE: %{inner} = expanded1020 1021 ; RUN: echo '%{outer}'1022 1023``DEFINE:`` inserts substitutions at the start of the substitution list, so1024``%{inner}`` expands first but has no effect because the original ``RUN:`` line1025does not contain ``%{inner}``. Next, ``%{outer}`` expands, and the output of1026the ``echo`` command becomes:1027 1028.. code-block:: shell1029 1030 %{inner}1031 1032Of course, one way to fix this simple case is to reverse the definitions of1033``%{outer}`` and ``%{inner}``. However, if a test has a complex set of1034substitutions that can all reference each other, there might not exist a1035sufficient substitution order.1036 1037To address such use cases, lit configuration files support1038``config.recursiveExpansionLimit``, which can be set to a non-negative integer1039to specify the maximum number of passes through the substitution list. Thus, in1040the above example, setting the limit to 2 would cause lit to make a second pass1041that expands ``%{inner}`` in the ``RUN:`` line, and the output from the ``echo``1042command would then be:1043 1044.. code-block:: shell1045 1046 expanded1047 1048To improve performance, lit will stop making passes when it notices the ``RUN:``1049line has stopped changing. In the above example, setting the limit higher than10502 is thus harmless.1051 1052To facilitate debugging, after reaching the limit, lit will make one extra pass1053and report an error if the ``RUN:`` line changes again. In the above example,1054setting the limit to 1 will thus cause lit to report an error instead of1055producing incorrect output.1056 1057Options1058-------1059 1060The llvm lit configuration allows some things to be customized with user options:1061 1062``llc``, ``opt``, ...1063 Substitute the respective llvm tool name with a custom command line. This1064 allows to specify custom paths and default arguments for these tools.1065 Example:1066 1067 % llvm-lit "-Dllc=llc -verify-machineinstrs"1068 1069``run_long_tests``1070 Enable the execution of long running tests.1071 1072``llvm_site_config``1073 Load the specified lit configuration instead of the default one.1074 1075 1076Other Features1077--------------1078 1079To make ``RUN`` line writing easier, several helper programs are available. These1080helpers are in the ``PATH`` when running tests, so you can just call them using1081their name. For example:1082 1083``not``1084 This program runs its arguments and then inverts the result code from it.1085 Zero result codes become 1. Non-zero result codes become 0.1086 1087To make the output more useful, :program:`lit` will scan1088the lines of the test case for ones that contain a pattern that matches1089``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number1090that is related to the test case. The number after "PR" specifies the1091LLVM Bugzilla number. When a PR number is specified, it will be used in1092the pass/fail reporting. This is useful to quickly get some context when1093a test fails.1094 1095Finally, any line that contains "END." will cause the special1096interpretation of lines to terminate. This is generally done right after1097the last ``RUN:`` line. This has two side effects:1098 1099(a) it prevents special interpretation of lines that are part of the test1100 program, not the instructions to the test case, and1101 1102(b) it speeds things up for really big test cases by avoiding1103 interpretation of the remainder of the file.1104