439 lines · plain
1======================2Linux Kernel Selftests3======================4 5The kernel contains a set of "self tests" under the tools/testing/selftests/6directory. These are intended to be small tests to exercise individual code7paths in the kernel. Tests are intended to be run after building, installing8and booting a kernel.9 10Kselftest from mainline can be run on older stable kernels. Running tests11from mainline offers the best coverage. Several test rings run mainline12kselftest suite on stable releases. The reason is that when a new test13gets added to test existing code to regression test a bug, we should be14able to run that test on an older kernel. Hence, it is important to keep15code that can still test an older kernel and make sure it skips the test16gracefully on newer releases.17 18You can find additional information on Kselftest framework, how to19write new tests using the framework on Kselftest wiki:20 21https://kselftest.wiki.kernel.org/22 23On some systems, hot-plug tests could hang forever waiting for cpu and24memory to be ready to be offlined. A special hot-plug target is created25to run the full range of hot-plug tests. In default mode, hot-plug tests run26in safe mode with a limited scope. In limited mode, cpu-hotplug test is27run on a single cpu as opposed to all hotplug capable cpus, and memory28hotplug test is run on 2% of hotplug capable memory instead of 10%.29 30kselftest runs as a userspace process. Tests that can be written/run in31userspace may wish to use the `Test Harness`_. Tests that need to be32run in kernel space may wish to use a `Test Module`_.33 34Running the selftests (hotplug tests are run in limited mode)35=============================================================36 37To build the tests::38 39 $ make headers40 $ make -C tools/testing/selftests41 42To run the tests::43 44 $ make -C tools/testing/selftests run_tests45 46To build and run the tests with a single command, use::47 48 $ make kselftest49 50Note that some tests will require root privileges.51 52Kselftest supports saving output files in a separate directory and then53running tests. To locate output files in a separate directory two syntaxes54are supported. In both cases the working directory must be the root of the55kernel src. This is applicable to "Running a subset of selftests" section56below.57 58To build, save output files in a separate directory with O= ::59 60 $ make O=/tmp/kselftest kselftest61 62To build, save output files in a separate directory with KBUILD_OUTPUT ::63 64 $ export KBUILD_OUTPUT=/tmp/kselftest; make kselftest65 66The O= assignment takes precedence over the KBUILD_OUTPUT environment67variable.68 69The above commands by default run the tests and print full pass/fail report.70Kselftest supports "summary" option to make it easier to understand the test71results. Please find the detailed individual test results for each test in72/tmp/testname file(s) when summary option is specified. This is applicable73to "Running a subset of selftests" section below.74 75To run kselftest with summary option enabled ::76 77 $ make summary=1 kselftest78 79Running a subset of selftests80=============================81 82You can use the "TARGETS" variable on the make command line to specify83single test to run, or a list of tests to run.84 85To run only tests targeted for a single subsystem::86 87 $ make -C tools/testing/selftests TARGETS=ptrace run_tests88 89You can specify multiple tests to build and run::90 91 $ make TARGETS="size timers" kselftest92 93To build, save output files in a separate directory with O= ::94 95 $ make O=/tmp/kselftest TARGETS="size timers" kselftest96 97To build, save output files in a separate directory with KBUILD_OUTPUT ::98 99 $ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest100 101Additionally you can use the "SKIP_TARGETS" variable on the make command102line to specify one or more targets to exclude from the TARGETS list.103 104To run all tests but a single subsystem::105 106 $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests107 108You can specify multiple tests to skip::109 110 $ make SKIP_TARGETS="size timers" kselftest111 112You can also specify a restricted list of tests to run together with a113dedicated skiplist::114 115 $ make TARGETS="breakpoints size timers" SKIP_TARGETS=size kselftest116 117See the top-level tools/testing/selftests/Makefile for the list of all118possible targets.119 120Running the full range hotplug selftests121========================================122 123To build the hotplug tests::124 125 $ make -C tools/testing/selftests hotplug126 127To run the hotplug tests::128 129 $ make -C tools/testing/selftests run_hotplug130 131Note that some tests will require root privileges.132 133 134Install selftests135=================136 137You can use the "install" target of "make" (which calls the `kselftest_install.sh`138tool) to install selftests in the default location (`tools/testing/selftests/kselftest_install`),139or in a user specified location via the `INSTALL_PATH` "make" variable.140 141To install selftests in default location::142 143 $ make -C tools/testing/selftests install144 145To install selftests in a user specified location::146 147 $ make -C tools/testing/selftests install INSTALL_PATH=/some/other/path148 149Running installed selftests150===========================151 152Found in the install directory, as well as in the Kselftest tarball,153is a script named `run_kselftest.sh` to run the tests.154 155You can simply do the following to run the installed Kselftests. Please156note some tests will require root privileges::157 158 $ cd kselftest_install159 $ ./run_kselftest.sh160 161To see the list of available tests, the `-l` option can be used::162 163 $ ./run_kselftest.sh -l164 165The `-c` option can be used to run all the tests from a test collection, or166the `-t` option for specific single tests. Either can be used multiple times::167 168 $ ./run_kselftest.sh -c size -c seccomp -t timers:posix_timers -t timer:nanosleep169 170For other features see the script usage output, seen with the `-h` option.171 172Timeout for selftests173=====================174 175Selftests are designed to be quick and so a default timeout is used of 45176seconds for each test. Tests can override the default timeout by adding177a settings file in their directory and set a timeout variable there to the178configured a desired upper timeout for the test. Only a few tests override179the timeout with a value higher than 45 seconds, selftests strives to keep180it that way. Timeouts in selftests are not considered fatal because the181system under which a test runs may change and this can also modify the182expected time it takes to run a test. If you have control over the systems183which will run the tests you can configure a test runner on those systems to184use a greater or lower timeout on the command line as with the `-o` or185the `--override-timeout` argument. For example to use 165 seconds instead186one would use::187 188 $ ./run_kselftest.sh --override-timeout 165189 190You can look at the TAP output to see if you ran into the timeout. Test191runners which know a test must run under a specific time can then optionally192treat these timeouts then as fatal.193 194Packaging selftests195===================196 197In some cases packaging is desired, such as when tests need to run on a198different system. To package selftests, run::199 200 $ make -C tools/testing/selftests gen_tar201 202This generates a tarball in the `INSTALL_PATH/kselftest-packages` directory. By203default, `.gz` format is used. The tar compression format can be overridden by204specifying a `FORMAT` make variable. Any value recognized by `tar's auto-compress`_205option is supported, such as::206 207 $ make -C tools/testing/selftests gen_tar FORMAT=.xz208 209`make gen_tar` invokes `make install` so you can use it to package a subset of210tests by using variables specified in `Running a subset of selftests`_211section::212 213 $ make -C tools/testing/selftests gen_tar TARGETS="size" FORMAT=.xz214 215.. _tar's auto-compress: https://www.gnu.org/software/tar/manual/html_node/gzip.html#auto_002dcompress216 217Contributing new tests218======================219 220In general, the rules for selftests are221 222 * Do as much as you can if you're not root;223 224 * Don't take too long;225 226 * Don't break the build on any architecture, and227 228 * Don't cause the top-level "make run_tests" to fail if your feature is229 unconfigured.230 231 * The output of tests must conform to the TAP standard to ensure high232 testing quality and to capture failures/errors with specific details.233 The kselftest.h and kselftest_harness.h headers provide wrappers for234 outputting test results. These wrappers should be used for pass,235 fail, exit, and skip messages. CI systems can easily parse TAP output236 messages to detect test results.237 238Contributing new tests (details)239================================240 241 * In your Makefile, use facilities from lib.mk by including it instead of242 reinventing the wheel. Specify flags and binaries generation flags on243 need basis before including lib.mk. ::244 245 CFLAGS = $(KHDR_INCLUDES)246 TEST_GEN_PROGS := close_range_test247 include ../lib.mk248 249 * Use TEST_GEN_XXX if such binaries or files are generated during250 compiling.251 252 TEST_PROGS, TEST_GEN_PROGS mean it is the executable tested by253 default.254 255 TEST_GEN_MODS_DIR should be used by tests that require modules to be built256 before the test starts. The variable will contain the name of the directory257 containing the modules.258 259 TEST_CUSTOM_PROGS should be used by tests that require custom build260 rules and prevent common build rule use.261 262 TEST_PROGS are for test shell scripts. Please ensure shell script has263 its exec bit set. Otherwise, lib.mk run_tests will generate a warning.264 265 TEST_CUSTOM_PROGS and TEST_PROGS will be run by common run_tests.266 267 TEST_PROGS_EXTENDED, TEST_GEN_PROGS_EXTENDED mean it is the268 executable which is not tested by default.269 270 TEST_FILES, TEST_GEN_FILES mean it is the file which is used by271 test.272 273 TEST_INCLUDES is similar to TEST_FILES, it lists files which should be274 included when exporting or installing the tests, with the following275 differences:276 277 * symlinks to files in other directories are preserved278 * the part of paths below tools/testing/selftests/ is preserved when279 copying the files to the output directory280 281 TEST_INCLUDES is meant to list dependencies located in other directories of282 the selftests hierarchy.283 284 * First use the headers inside the kernel source and/or git repo, and then the285 system headers. Headers for the kernel release as opposed to headers286 installed by the distro on the system should be the primary focus to be able287 to find regressions. Use KHDR_INCLUDES in Makefile to include headers from288 the kernel source.289 290 * If a test needs specific kernel config options enabled, add a config file in291 the test directory to enable them.292 293 e.g: tools/testing/selftests/android/config294 295 * Create a .gitignore file inside test directory and add all generated objects296 in it.297 298 * Add new test name in TARGETS in selftests/Makefile::299 300 TARGETS += android301 302 * All changes should pass::303 304 kselftest-{all,install,clean,gen_tar}305 kselftest-{all,install,clean,gen_tar} O=abo_path306 kselftest-{all,install,clean,gen_tar} O=rel_path307 make -C tools/testing/selftests {all,install,clean,gen_tar}308 make -C tools/testing/selftests {all,install,clean,gen_tar} O=abs_path309 make -C tools/testing/selftests {all,install,clean,gen_tar} O=rel_path310 311Test Module312===========313 314Kselftest tests the kernel from userspace. Sometimes things need315testing from within the kernel, one method of doing this is to create a316test module. We can tie the module into the kselftest framework by317using a shell script test runner. ``kselftest/module.sh`` is designed318to facilitate this process. There is also a header file provided to319assist writing kernel modules that are for use with kselftest:320 321- ``tools/testing/selftests/kselftest_module.h``322- ``tools/testing/selftests/kselftest/module.sh``323 324Note that test modules should taint the kernel with TAINT_TEST. This will325happen automatically for modules which are in the ``tools/testing/``326directory, or for modules which use the ``kselftest_module.h`` header above.327Otherwise, you'll need to add ``MODULE_INFO(test, "Y")`` to your module328source. selftests which do not load modules typically should not taint the329kernel, but in cases where a non-test module is loaded, TEST_TAINT can be330applied from userspace by writing to ``/proc/sys/kernel/tainted``.331 332How to use333----------334 335Here we show the typical steps to create a test module and tie it into336kselftest. We use kselftests for lib/ as an example.337 3381. Create the test module339 3402. Create the test script that will run (load/unload) the module341 e.g. ``tools/testing/selftests/lib/printf.sh``342 3433. Add line to config file e.g. ``tools/testing/selftests/lib/config``344 3454. Add test script to makefile e.g. ``tools/testing/selftests/lib/Makefile``346 3475. Verify it works:348 349.. code-block:: sh350 351 # Assumes you have booted a fresh build of this kernel tree352 cd /path/to/linux/tree353 make kselftest-merge354 make modules355 sudo make modules_install356 make TARGETS=lib kselftest357 358Example Module359--------------360 361A bare bones test module might look like this:362 363.. code-block:: c364 365 // SPDX-License-Identifier: GPL-2.0+366 367 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt368 369 #include "../tools/testing/selftests/kselftest_module.h"370 371 KSTM_MODULE_GLOBALS();372 373 /*374 * Kernel module for testing the foobinator375 */376 377 static int __init test_function()378 {379 ...380 }381 382 static void __init selftest(void)383 {384 KSTM_CHECK_ZERO(do_test_case("", 0));385 }386 387 KSTM_MODULE_LOADERS(test_foo);388 MODULE_AUTHOR("John Developer <jd@fooman.org>");389 MODULE_LICENSE("GPL");390 MODULE_INFO(test, "Y");391 392Example test script393-------------------394 395.. code-block:: sh396 397 #!/bin/bash398 # SPDX-License-Identifier: GPL-2.0+399 $(dirname $0)/../kselftest/module.sh "foo" test_foo400 401 402Test Harness403============404 405The kselftest_harness.h file contains useful helpers to build tests. The406test harness is for userspace testing, for kernel space testing see `Test407Module`_ above.408 409The tests from tools/testing/selftests/seccomp/seccomp_bpf.c can be used as410example.411 412Example413-------414 415.. kernel-doc:: tools/testing/selftests/kselftest_harness.h416 :doc: example417 418 419Helpers420-------421 422.. kernel-doc:: tools/testing/selftests/kselftest_harness.h423 :functions: TH_LOG TEST TEST_SIGNAL FIXTURE FIXTURE_DATA FIXTURE_SETUP424 FIXTURE_TEARDOWN TEST_F TEST_HARNESS_MAIN FIXTURE_VARIANT425 FIXTURE_VARIANT_ADD426 427Operators428---------429 430.. kernel-doc:: tools/testing/selftests/kselftest_harness.h431 :doc: operators432 433.. kernel-doc:: tools/testing/selftests/kselftest_harness.h434 :functions: ASSERT_EQ ASSERT_NE ASSERT_LT ASSERT_LE ASSERT_GT ASSERT_GE435 ASSERT_NULL ASSERT_TRUE ASSERT_NULL ASSERT_TRUE ASSERT_FALSE436 ASSERT_STREQ ASSERT_STRNE EXPECT_EQ EXPECT_NE EXPECT_LT437 EXPECT_LE EXPECT_GT EXPECT_GE EXPECT_NULL EXPECT_TRUE438 EXPECT_FALSE EXPECT_STREQ EXPECT_STRNE439