588 lines · plain
1.. _testing:2 3==============4Testing libc++5==============6 7.. contents::8 :local:9 10Getting Started11===============12 13libc++ uses LIT to configure and run its tests.14 15The primary way to run the libc++ tests is by using ``make check-cxx``.16 17However since libc++ can be used in any number of possible18configurations it is important to customize the way LIT builds and runs19the tests. This guide provides information on how to use LIT directly to20test libc++.21 22Please see the `Lit Command Guide`_ for more information about LIT.23 24.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html25 26Usage27-----28 29After :ref:`building libc++ <VendorDocumentation>`, you can run parts of the libc++ test suite by simply30running ``llvm-lit`` on a specified test or directory. If you're unsure31whether the required libraries have been built, you can use the32``cxx-test-depends`` target. For example:33 34.. code-block:: bash35 36 $ cd <monorepo-root>37 $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt38 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests39 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test40 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic41 42If you used **ninja** as your build system, running ``ninja -C <build> check-cxx`` will run43all the tests in the libc++ testsuite.44 45.. note::46 If you used the Bootstrapping build instead of the default runtimes build, the47 ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and48 you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the49 paths of all tests. For example, to run all the libcxx tests you can do50 ``<build>/bin/llvm-lit -sv <build>/runtimes/runtimes-bins/libcxx/test``.51 52In the default configuration, the tests are built against headers that form a53fake installation root of libc++. This installation root has to be updated when54changes are made to the headers, so you should re-run the ``cxx-test-depends``55target before running the tests manually with ``lit`` when you make any sort of56change, including to the headers. We recommend using the provided ``libcxx/utils/libcxx-lit``57script to automate this so you don't have to think about building test dependencies58every time:59 60.. code-block:: bash61 62 $ cd <monorepo-root>63 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/re # Build testing dependencies and run all of the std::regex tests64 65Sometimes you'll want to change the way LIT is running the tests. Custom options66can be specified using the ``--param <name>=<val>`` flag. The most common option67you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the68test suite will select the newest C++ dialect supported by the compiler and use69that. However, you can manually specify the option like so if you want:70 71.. code-block:: bash72 73 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers # Run the tests with the newest -std74 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++0375 76Other parameters are supported by the test suite. Those are defined in ``libcxx/utils/libcxx/test/params.py``.77If you want to customize how to run the libc++ test suite beyond what is available78in ``params.py``, you most likely want to use a custom site configuration instead.79 80The libc++ test suite works by loading a site configuration that defines various81"base" parameters (via Lit substitutions). These base parameters represent things82like the compiler to use for running the tests, which default compiler and linker83flags to use, and how to run an executable. This system is meant to be easily84extended for custom needs, in particular when porting the libc++ test suite to85new platforms.86 87.. note::88 If you run the test suite on Apple platforms, we recommend adding the terminal application89 used to run the test suite to the list of "Developer Tools". This prevents the system from90 trying to scan each individual test binary for malware and dramatically speeds up the test91 suite.92 93Using a Custom Site Configuration94---------------------------------95 96By default, the libc++ test suite will use a site configuration that matches97the current CMake configuration. It does so by generating a ``lit.site.cfg``98file in the build directory from one of the configuration file templates in99``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around100``llvm/utils/lit/lit.py``) to that file. So when you're running101``<build>/bin/llvm-lit`` either directly or indirectly, the generated ``lit.site.cfg``102file is always loaded instead of ``libcxx/test/lit.cfg.py``. If you want to use a103custom site configuration, simply point the CMake build to it using104``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration105will be used instead. That file can use CMake variables inside it to make106configuration easier.107 108 .. code-block:: bash109 110 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>111 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test # will use your custom config file112 113Additional tools114----------------115 116The libc++ test suite uses a few optional tools to improve the code quality.117 118These tools are:119 120- clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks)121 122Reproducing CI issues locally123-----------------------------124 125Libc++ has extensive CI that tests various configurations of the library. The testing for126all these configurations is located in ``libcxx/utils/ci/run-buildbot``. Most of our127CI jobs are being run on a Docker image for reproducibility. The definition of this Docker128image is located in ``libcxx/utils/ci/Dockerfile``. If you are looking to reproduce the129failure of a specific CI job locally, you should first drop into a Docker container that130matches our CI images by running ``libcxx/utils/ci/run-buildbot-container``, and then run131the specific CI job that you're interested in (from within the container) using the ``run-buildbot``132script above. If you want to control which compiler is used, you can set the ``CC`` and the133``CXX`` environment variables before calling ``run-buildbot`` to select the right compiler.134Take note that some CI jobs are testing the library on specific platforms and are *not* run135in our Docker image. In the general case, it is not possible to reproduce these failures136locally, unless they aren't specific to the platform.137 138Also note that the Docker container shares the same filesystem as your local machine, so139modifying files on your local machine will also modify what the Docker container sees.140This is useful for editing source files as you're testing your code in the Docker container.141 142Writing Tests143=============144 145When writing tests for the libc++ test suite, you should follow a few guidelines.146This will ensure that your tests can run on a wide variety of hardware and under147a wide variety of configurations. We have several unusual configurations such as148building the tests on one host but running them on a different host, which add a149few requirements to the test suite. Here's some stuff you should know:150 151- All tests are run in a temporary directory that is unique to that test and152 cleaned up after the test is done.153- When a test needs data files as inputs, these data files can be saved in the154 repository (when reasonable) and referenced by the test as155 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or156 directories will be made available to the test in the temporary directory157 where it is run.158- You should never hardcode a path from the build-host in a test, because that159 path will not necessarily be available on the host where the tests are run.160- You should try to reduce the runtime dependencies of each test to the minimum.161 For example, requiring Python to run a test is bad, since Python is not162 necessarily available on all devices we may want to run the tests on (even163 though supporting Python is probably trivial for the build-host).164 165Structure of the testing related directories166--------------------------------------------167 168The tests of libc++ are stored in libc++'s testing related subdirectories:169 170- ``libcxx/test/support`` This directory contains several helper headers with171 generic parts for the tests. The most important header is ``test_macros.h``.172 This file contains configuration information regarding the platform used.173 This is similar to the ``__config`` file in libc++'s ``include`` directory.174 Since libc++'s tests are used by other Standard libraries, tests should use175 the ``TEST_FOO`` macros instead of the ``_LIBCPP_FOO`` macros, which are176 specific to libc++.177- ``libcxx/test/std`` This directory contains the tests that validate the library under178 test conforms to the C++ Standard. The paths and the names of the test match179 the section names in the C++ Standard. Note that the C++ Standard sometimes180 reorganises its structure, therefore some tests are at a location based on181 where they appeared historically in the standard. We try to strike a balance182 between keeping things at up-to-date locations and unnecessary churn.183- ``libcxx/test/libcxx`` This directory contains the tests that validate libc++184 specific behavior and implementation details. For example, libc++ has185 "wrapped iterators" that perform bounds checks. Since those are specific to186 libc++ and not mandated by the Standard, tests for those are located under187 ``libcxx/test/libcxx``. The structure of this directories follows the188 structure of ``libcxx/test/std``.189 190Structure of a test191-------------------192 193Some platforms where libc++ is tested have requirement on the signature of194``main`` and require ``main`` to explicitly return a value. Therefore the195typical ``main`` function should look like:196 197.. code-block:: cpp198 199 int main(int, char**) {200 ...201 return 0;202 }203 204 205The C++ Standard has ``constexpr`` requirements. The typical way to test that,206is to create a helper ``test`` function that returns a ``bool`` and use the207following ``main`` function:208 209.. code-block:: cpp210 211 constexpr bool test() {212 ...213 return true;214 }215 216 int main(int, char**) {217 test()218 static_assert(test());219 220 return 0;221 }222 223Tests in libc++ mainly use ``assert`` and ``static_assert`` for testing. There224are a few helper macros and function that can be used to make it easier to225write common tests.226 227libcxx/test/support/assert_macros.h228~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~229 230The header contains several macros with user specified log messages. This is231useful when a normal assertion failure lacks the information to easily232understand why the test has failed. This usually happens when the test is in a233helper function. For example the ``std::format`` tests use a helper function234for its validation. When the test fails it will give the line in the helper235function with the condition ``out == expected`` failed. Without knowing what236the value of ``format string``, ``out`` and ``expected`` are it is not easy to237understand why the test has failed. By logging these three values the point of238failure can be found without resorting to a debugger.239 240Several of these macros are documented to take an ``ARG``. This ``ARG``:241 242 - if it is a ``const char*`` or ``std::string`` its contents are written to243 the ``stderr``,244 - otherwise it must be a callable that is invoked without any additional245 arguments and is expected to produce useful output to e.g. ``stderr``.246 247This makes it possible to write additional information when a test fails,248either by supplying a hard-coded string or generate it at runtime.249 250TEST_FAIL(ARG)251^^^^^^^^^^^^^^252 253This macro is an unconditional failure with a log message ``ARG``. The main254use-case is to fail when code is reached that should be unreachable.255 256 257TEST_REQUIRE(CONDITION, ARG)258^^^^^^^^^^^^^^^^^^^^^^^^^^^^259 260This macro requires its ``CONDITION`` to evaluate to ``true``. If that fails it261will fail the test with a log message ``ARG``.262 263 264TEST_LIBCPP_REQUIRE((CONDITION, ARG)265^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^266 267If the library under test is libc++ it behaves like ``TEST_REQUIRE``, else it268is a no-op. This makes it possible to test libc++ specific behaviour. For269example testing whether the ``what()`` of an exception thrown matches libc++'s270expectations. (Usually the Standard requires certain exceptions to be thrown,271but not the contents of its ``what()`` message.)272 273 274TEST_DOES_NOT_THROW(EXPR)275^^^^^^^^^^^^^^^^^^^^^^^^^276 277Validates execution of ``EXPR`` does not throw an exception.278 279TEST_THROWS_TYPE(TYPE, EXPR)280^^^^^^^^^^^^^^^^^^^^^^^^^^^^281 282Validates the execution of ``EXPR`` throws an exception of the type ``TYPE``.283 284 285TEST_VALIDATE_EXCEPTION(TYPE, PRED, EXPR)286^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^287 288Validates the execution of ``EXPR`` throws an exception of the type ``TYPE``289which passes validation of ``PRED``. Using this macro makes it easier to write290tests using exceptions. The code to write a test manually would be:291 292 293.. code-block:: cpp294 295 void test_exception([[maybe_unused]] int arg) {296 #ifndef TEST_HAS_NO_EXCEPTIONS // do nothing when tests are disabled297 try {298 foo(arg);299 assert(false); // validates foo really throws300 } catch ([[maybe_unused]] const bar& e) {301 LIBCPP_ASSERT(e.what() == what);302 return;303 }304 assert(false); // validates bar was thrown305 #endif306 }307 308The same test using a macro:309 310.. code-block:: cpp311 312 void test_exception([[maybe_unused]] int arg) {313 TEST_VALIDATE_EXCEPTION(bar,314 [](const bar& e) {315 LIBCPP_ASSERT(e.what() == what);316 },317 foo(arg));318 }319 320 321libcxx/test/support/concat_macros.h322~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~323 324This file contains a helper macro ``TEST_WRITE_CONCATENATED`` to lazily325concatenate its arguments to a ``std::string`` and write it to ``stderr``. When326the output can't be concatenated a default message will be written to327``stderr``. This is useful for tests where the arguments use different328character types like ``char`` and ``wchar_t``, the latter can't simply be329written to ``stderr``.330 331This macro is in a different header as ``assert_macros.h`` since it pulls in332additional headers.333 334 .. note: This macro can only be used in test using C++20 or newer. The macro335 was added at a time where most of libc++'s C++17 support was complete.336 Since it is not expected to add this to existing tests no effort was337 taken to make it work in earlier language versions.338 339 340Test names341----------342 343The names of test files have meaning for the libc++-specific configuration of344Lit. Based on the pattern that matches the name of a test file, Lit will test345the code contained therein in different ways. Refer to the `Lit Meaning of libc++346Test Filenames`_ when determining the names for new test files.347 348.. _Lit Meaning of libc++ Test Filenames:349.. list-table:: Lit Meaning of libc++ Test Filenames350 :widths: 25 75351 :header-rows: 1352 353 * - Name Pattern354 - Meaning355 * - ``FOO.pass.cpp``356 - Checks whether the C++ code in the file compiles, links and runs successfully.357 * - ``FOO.pass.mm``358 - Same as ``FOO.pass.cpp``, but for Objective-C++.359 360 * - ``FOO.compile.pass.cpp``361 - Checks whether the C++ code in the file compiles successfully. In general, prefer ``compile`` tests over ``verify`` tests,362 subject to the specific recommendations, below, for when to write ``verify`` tests.363 * - ``FOO.compile.pass.mm``364 - Same as ``FOO.compile.pass.cpp``, but for Objective-C++.365 * - ``FOO.compile.fail.cpp``366 - Checks that the code in the file does *not* compile successfully.367 368 * - ``FOO.verify.cpp``369 - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.370 For additional information about how to write ``verify`` tests, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.371 Prefer `verify` tests over ``compile`` tests to test that compilation fails for a particular reason. For example, use a ``verify`` test372 to ensure that373 374 * an expected ``static_assert`` is triggered;375 * the use of deprecated functions generates the proper warning;376 * removed functions are no longer usable; or377 * return values from functions marked ``[[nodiscard]]`` are stored.378 379 * - ``FOO.link.pass.cpp``380 - Checks that the C++ code in the file compiles and links successfully -- no run attempted.381 * - ``FOO.link.pass.mm``382 - Same as ``FOO.link.pass.cpp``, but for Objective-C++.383 * - ``FOO.link.fail.cpp``384 - Checks whether the C++ code in the file fails to link after successful compilation.385 * - ``FOO.link.fail.mm``386 - Same as ``FOO.link.fail.cpp``, but for Objective-C++.387 388 * - ``FOO.sh.<anything>``389 - A *builtin Lit Shell* test.390 * - ``FOO.gen.<anything>``391 - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected392 by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type393 of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties394 in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.395 396 * - ``FOO.bench.cpp``397 - A benchmark test. These tests are linked against the GoogleBenchmark library and generally consist of micro-benchmarks of individual398 components of the library.399 400 401libc++-Specific Lit Features402----------------------------403 404Custom Directives405~~~~~~~~~~~~~~~~~406 407Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes408writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES``, ``ADDITIONAL_COMPILE_FLAGS``, and ``MODULE_DEPENDENCIES`` libc++-specific directives.409 410.. _libc++-specific Lit Directives:411.. list-table:: libc++-specific Lit Directives412 :widths: 20 35 45413 :header-rows: 1414 415 * - Directive416 - Parameters417 - Usage418 * - ``FILE_DEPENDENCIES``419 - ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``420 - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test421 input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy422 them to the directory represented by the ``%{temp}`` substitution before the test executes. The copy is performed from the directory represented by the ``%S`` substitution423 (i.e. the source directory of the test being executed) which makes it possible to use relative paths to specify the location of dependency files. After Lit copies424 all the dependent files to the directory specified by the ``%{temp}`` substitution, that directory should contain *all* the necessary inputs to run. In other words,425 it should be possible to copy the contents of the directory specified by the ``%{temp}`` substitution to a remote host where the execution of the test will actually occur.426 * - ``ADDITIONAL_COMPILE_FLAGS``427 - ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...``428 - The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}``429 substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see430 `Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill.431 * - ``MODULE_DEPENDENCIES``432 - ``// MODULE_DEPENDENCIES: std std.compat``433 - This directive will build the required C++23 standard library434 modules and add the additional compiler flags in435 %{compile_flags}. (Libc++ offers these modules in C++20 as an436 extension.)437 438 439C++ Standard version tests440~~~~~~~~~~~~~~~~~~~~~~~~~~441 442Historically libc++ tests used to filter the tests for C++ Standard versions443with lit directives like:444 445.. code-block:: cpp446 447 // UNSUPPORTED: c++03, c++11, c++14, c++17, c++20, c++23448 449With C++ Standards released every 3 years, this solution is not scalable.450Instead use:451 452.. code-block:: cpp453 454 // REQUIRES: std-at-least-c++26455 456There is no corresponding ``std-at-most-c++23``. This could be useful when457tests are only valid for a small set of standard versions. For example, a458deprecation test is only valid when the feature is deprecated until it is459removed from the Standard. These tests should be written like:460 461.. code-block:: cpp462 463 // REQUIRES: c++17 || c++20 || c++23464 465.. note::466 467 There are a lot of tests with the first style, these can remain as they are.468 The new style is only intended to be used for new tests.469 470 471Benchmarks472==========473 474Libc++'s test suite also contains benchmarks. Many benchmarks are written using the `Google Benchmark`_475library, a copy of which is stored in the LLVM monorepo. For more information about using the Google476Benchmark library, see the `official documentation <https://github.com/google/benchmark>`_.477 478The benchmarks are located under ``libcxx/test/benchmarks``. Running a benchmark479works in the same way as running a test. Both the benchmarks and the tests share480the same configuration, so make sure to enable the relevant optimization level481when running the benchmarks. For example,482 483.. code-block:: bash484 485 $ libcxx/utils/libcxx-lit <build> libcxx/test/benchmarks/containers/string.bench.cpp --show-all --param optimization=speed486 487Note that benchmarks are only dry-run when run via the ``check-cxx`` target since488we only want to make sure they don't rot. Do not rely on the results of benchmarks489run through ``check-cxx`` for anything, instead run the benchmarks manually using490the instructions for running individual tests.491 492If you want to compare the results of different benchmark runs, we recommend using the493``compare-benchmarks`` helper tool. Note that the script has some dependencies, which can494be installed with:495 496.. code-block:: bash497 498 $ python -m venv .venv && source .venv/bin/activate # Optional but recommended499 $ pip install -r libcxx/utils/requirements.txt500 501Once that's done, start by configuring CMake in a build directory and running one or502more benchmarks, as usual:503 504.. code-block:: bash505 506 $ cmake -S runtimes -B <build> [...]507 $ libcxx/utils/libcxx-lit <build> libcxx/test/benchmarks/containers/string.bench.cpp --param optimization=speed508 509Then, get the consolidated benchmark output for that run using ``consolidate-benchmarks``:510 511.. code-block:: bash512 513 $ libcxx/utils/consolidate-benchmarks <build> > baseline.lnt514 515The ``baseline.lnt`` file will contain a consolidation of all the benchmark results present in the build516directory. You can then make the desired modifications to the code, run the benchmark(s) again, and then run:517 518.. code-block:: bash519 520 $ libcxx/utils/consolidate-benchmarks <build> > candidate.lnt521 522Finally, use ``compare-benchmarks`` to compare both:523 524.. code-block:: bash525 526 $ libcxx/utils/compare-benchmarks baseline.lnt candidate.lnt527 528 # Useful one-liner when iterating locally:529 $ libcxx/utils/compare-benchmarks baseline.lnt <(libcxx/utils/consolidate-benchmarks <build>)530 531The ``compare-benchmarks`` script provides some useful options like creating a chart to easily visualize532differences in a browser window. Use ``compare-benchmarks --help`` for details.533 534Additionally, adding a comment of the following form to a libc++ PR will cause the specified benchmarks to be run535on our pre-commit CI infrastructure and the results to be reported in the PR by our CI system:536 537.. code-block::538 539 /libcxx-bot benchmark <path/to/benchmark1.bench.cpp> <path/to/benchmark2.bench.cpp> ...540 541Note that this is currently experimental and the results should not be relied upon too strongly, since542we do not have dedicated hardware to run the benchmarks on.543 544.. _`Google Benchmark`: https://github.com/google/benchmark545 546.. _testing-hardening-assertions:547 548Testing hardening assertions549============================550 551Each hardening assertion should be tested using death tests (via the552``TEST_LIBCPP_ASSERT_FAILURE`` macro). Use the ``libcpp-hardening-mode`` Lit553feature to make sure the assertion is enabled in (and only in) the intended554modes. The convention is to use `assert.` in the name of the test file to make555it easier to identify as a hardening test, e.g. ``assert.my_func.pass.cpp``.556A toy example:557 558.. code-block:: cpp559 560 // Note: the following three annotations are currently needed to use the561 // `TEST_LIBCPP_ASSERT_FAILURE`.562 // REQUIRES: has-unix-headers563 // UNSUPPORTED: c++03564 // XFAIL: libcpp-hardening-mode=debug && availability-verbose_abort-missing565 566 // Example: only run this test in `fast`/`extensive`/`debug` modes.567 // UNSUPPORTED: libcpp-hardening-mode=none568 // Example: only run this test in the `debug` mode.569 // REQUIRES: libcpp-hardening-mode=debug570 // Example: only run this test in `extensive`/`debug` modes.571 // REQUIRES: libcpp-hardening-mode={{extensive|debug}}572 573 #include <header_being_tested>574 575 #include "check_assertion.h" // Contains the `TEST_LIBCPP_ASSERT_FAILURE` macro576 577 int main(int, char**) {578 std::type_being_tested foo;579 int bad_input = -1;580 TEST_LIBCPP_ASSERT_FAILURE(foo.some_function_that_asserts(bad_input),581 "The expected assertion message");582 583 return 0;584 }585 586Note that error messages are only tested (matched) if the ``debug``587hardening mode is used.588