199 lines · plain
1======================================2test-suite Makefile Guide (deprecated)3======================================4 5.. contents::6 :local:7 8Overview9========10 11First, all tests are executed within the LLVM object directory tree.12They *are not* executed inside of the LLVM source tree. This is because13the test suite creates temporary files during execution.14 15To run the test suite, you need to use the following steps:16 17#. Check out the ``test-suite`` module with:18 19 .. code-block:: bash20 21 % git clone https://github.com/llvm/llvm-test-suite.git test-suite22 23#. FIXME: these directions are outdated and won't work. Figure out24 what the correct thing to do is, and write it down here.25 26#. Configure and build ``llvm``.27 28#. Configure and build ``llvm-gcc``.29 30#. Install ``llvm-gcc`` somewhere.31 32#. *Re-configure* ``llvm`` from the top level of each build tree (LLVM33 object directory tree) in which you want to run the test suite, just34 as you do before building LLVM.35 36 During the *re-configuration*, you must either: (1) have ``llvm-gcc``37 you just built in your path, or (2) specify the directory where your38 just-built ``llvm-gcc`` is installed using39 ``--with-llvmgccdir=$LLVM_GCC_DIR``.40 41 You must also tell the configure machinery that the test suite is42 available so it can be configured for your build tree:43 44 .. code-block:: bash45 46 % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]47 48 [Remember that ``$LLVM_GCC_DIR`` is the directory where you49 *installed* llvm-gcc, not its src or obj directory.]50 51#. You can now run the test suite from your build tree as follows:52 53 .. code-block:: bash54 55 % cd $LLVM_OBJ_ROOT/projects/test-suite56 % make57 58Note that the second and third steps only need to be done once. After59you have the suite checked out and configured, you don't need to do it60again (unless the test code or configure script changes).61 62Configuring External Tests63==========================64 65In order to run the External tests in the ``test-suite`` module, you66must specify *--with-externals*. This must be done during the67*re-configuration* step (see above), and the ``llvm`` re-configuration68must recognize the previously-built ``llvm-gcc``. If any of these is69missing or neglected, the External tests won't work.70 71* *--with-externals*72 73* *--with-externals=<directory>*74 75This tells LLVM where to find any external tests. They are expected to76be in specifically named subdirectories of <``directory``>. If77``directory`` is left unspecified, ``configure`` uses the default value78``/home/vadve/shared/benchmarks/speccpu2000/benchspec``. Subdirectory79names known to LLVM include:80 81* spec9582 83* speccpu200084 85* speccpu200686 87* povray3188 89Others are added from time to time, and can be determined from90``configure``.91 92Running Different Tests93=======================94 95In addition to the regular "whole program" tests, the ``test-suite``96module also provides a mechanism for compiling the programs in different97ways. If the variable TEST is defined on the ``gmake`` command line, the98test system will include a Makefile named99``TEST.<value of TEST variable>.Makefile``. This Makefile can modify100build rules to yield different results.101 102For example, the LLVM nightly tester uses ``TEST.nightly.Makefile`` to103create the nightly test reports. To run the nightly tests, run104``gmake TEST=nightly``.105 106There are several TEST Makefiles available in the tree. Some of them are107designed for internal LLVM research and will not work outside of the108LLVM research group. They may still be valuable, however, as a guide to109writing your own TEST Makefile for any optimization or analysis passes110that you develop with LLVM.111 112Generating Test Output113======================114 115There are a number of ways to run the tests and generate output. The116most simple one is simply running ``gmake`` with no arguments. This will117compile and run all programs in the tree using a number of different118methods and compare results. Any failures are reported in the output,119but are likely drowned in the other output. Passes are not reported120explicitly.121 122Somewhat better is running ``gmake TEST=sometest test``, which runs the123specified test and usually adds per-program summaries to the output124(depending on which sometest you use). For example, the ``nightly`` test125explicitly outputs TEST-PASS or TEST-FAIL for every test after each126program. Though these lines are still drowned in the output, it's easy127to grep the output logs in the Output directories.128 129Even better are the ``report`` and ``report.format`` targets (where130``format`` is one of ``html``, ``csv``, ``text`` or ``graphs``). The131exact contents of the report are dependent on which ``TEST`` you are132running, but the text results are always shown at the end of the run and133the results are always stored in the ``report.<type>.format`` file (when134running with ``TEST=<type>``). The ``report`` also generate a file135called ``report.<type>.raw.out`` containing the output of the entire136test run.137 138Writing Custom Tests for the test-suite139=======================================140 141Assuming you can run the test suite, (e.g.142"``gmake TEST=nightly report``" should work), it is really easy to run143optimizations or code generator components against every program in the144tree, collecting statistics or running custom checks for correctness. At145base, this is how the nightly tester works, it's just one example of a146general framework.147 148Lets say that you have an LLVM optimization pass, and you want to see149how many times it triggers. First thing you should do is add an LLVM150`statistic <ProgrammersManual.html#Statistic>`_ to your pass, which will151tally counts of things you care about.152 153Following this, you can set up a test and a report that collects these154and formats them for easy viewing. This consists of two files, a155"``test-suite/TEST.XXX.Makefile``" fragment (where XXX is the name of156your test) and a "``test-suite/TEST.XXX.report``" file that indicates157how to format the output into a table. There are many example reports of158various levels of sophistication included with the test suite, and the159framework is very general.160 161If you are interested in testing an optimization pass, check out the162"libcalls" test as an example. It can be run like this:163 164.. code-block:: bash165 166 % cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level167 % make TEST=libcalls report168 169This will do a bunch of stuff, then eventually print a table like this:170 171::172 173 Name | total | #exit |174 ...175 FreeBench/analyzer/analyzer | 51 | 6 |176 FreeBench/fourinarow/fourinarow | 1 | 1 |177 FreeBench/neural/neural | 19 | 9 |178 FreeBench/pifft/pifft | 5 | 3 |179 MallocBench/cfrac/cfrac | 1 | * |180 MallocBench/espresso/espresso | 52 | 12 |181 MallocBench/gs/gs | 4 | * |182 Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |183 Prolangs-C/agrep/agrep | 33 | 12 |184 Prolangs-C/allroots/allroots | * | * |185 Prolangs-C/assembler/assembler | 47 | * |186 Prolangs-C/bison/mybison | 74 | * |187 ...188 189This basically is grepping the -stats output and displaying it in a190table. You can also use the "TEST=libcalls report.html" target to get191the table in HTML form, similarly for report.csv and report.tex.192 193The source for this is in ``test-suite/TEST.libcalls.*``. The format is194pretty simple: the Makefile indicates how to run the test (in this case,195"``opt -simplify-libcalls -stats``"), and the report contains one line196for each column of the output. The first value is the header for the197column and the second is the regex to grep the output of the command198for. There are lots of example reports that can do fancy stuff.199