312 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3===================================================4The Kernel Test Anything Protocol (KTAP), version 15===================================================6 7TAP, or the Test Anything Protocol is a format for specifying test results used8by a number of projects. It's website and specification are found at this `link9<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test10results. However, Kernel testing frameworks have special needs for test results11which don't align with the original TAP specification. Thus, a "Kernel TAP"12(KTAP) format is specified to extend and alter TAP to support these use-cases.13This specification describes the generally accepted format of KTAP as it is14currently used in the kernel.15 16KTAP test results describe a series of tests (which may be nested: i.e., test17can have subtests), each of which can contain both diagnostic data -- e.g., log18lines -- and a final result. The test structure and results are19machine-readable, whereas the diagnostic data is unstructured and is there to20aid human debugging.21 22KTAP output is built from four different types of lines:23- Version lines24- Plan lines25- Test case result lines26- Diagnostic lines27 28In general, valid KTAP output should also form valid TAP output, but some29information, in particular nested test results, may be lost. Also note that30there is a stagnant draft specification for TAP14, KTAP diverges from this in31a couple of places (notably the "Subtest" header), which are described where32relevant later in this document.33 34Version lines35-------------36 37All KTAP-formatted results begin with a "version line" which specifies which38version of the (K)TAP standard the result is compliant with.39 40For example:41- "KTAP version 1"42- "TAP version 13"43- "TAP version 14"44 45Note that, in KTAP, subtests also begin with a version line, which denotes the46start of the nested test results. This differs from TAP14, which uses a47separate "Subtest" line.48 49While, going forward, "KTAP version 1" should be used by compliant tests, it50is expected that most parsers and other tooling will accept the other versions51listed here for compatibility with existing tests and frameworks.52 53Plan lines54----------55 56A test plan provides the number of tests (or subtests) in the KTAP output.57 58Plan lines must follow the format of "1..N" where N is the number of tests or subtests.59Plan lines follow version lines to indicate the number of nested tests.60 61While there are cases where the number of tests is not known in advance -- in62which case the test plan may be omitted -- it is strongly recommended one is63present where possible.64 65Test case result lines66----------------------67 68Test case result lines indicate the final status of a test.69They are required and must have the format:70 71.. code-block:: none72 73 <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]74 75The result can be either "ok", which indicates the test case passed,76or "not ok", which indicates that the test case failed.77 78<number> represents the number of the test being performed. The first test must79have the number 1 and the number then must increase by 1 for each additional80subtest within the same test at the same nesting level.81 82The description is a description of the test, generally the name of83the test, and can be any string of characters other than # or a84newline. The description is optional, but recommended.85 86The directive and any diagnostic data is optional. If either are present, they87must follow a hash sign, "#".88 89A directive is a keyword that indicates a different outcome for a test other90than passed and failed. The directive is optional, and consists of a single91keyword preceding the diagnostic data. In the event that a parser encounters92a directive it doesn't support, it should fall back to the "ok" / "not ok"93result.94 95Currently accepted directives are:96 97- "SKIP", which indicates a test was skipped (note the result of the test case98 result line can be either "ok" or "not ok" if the SKIP directive is used)99- "TODO", which indicates that a test is not expected to pass at the moment,100 e.g. because the feature it is testing is known to be broken. While this101 directive is inherited from TAP, its use in the kernel is discouraged.102- "XFAIL", which indicates that a test is expected to fail. This is similar103 to "TODO", above, and is used by some kselftest tests.104- “TIMEOUT”, which indicates a test has timed out (note the result of the test105 case result line should be “not ok” if the TIMEOUT directive is used)106- “ERROR”, which indicates that the execution of a test has failed due to a107 specific error that is included in the diagnostic data. (note the result of108 the test case result line should be “not ok” if the ERROR directive is used)109 110The diagnostic data is a plain-text field which contains any additional details111about why this result was produced. This is typically an error message for ERROR112or failed tests, or a description of missing dependencies for a SKIP result.113 114The diagnostic data field is optional, and results which have neither a115directive nor any diagnostic data do not need to include the "#" field116separator.117 118Example result lines include::119 120 ok 1 test_case_name121 122The test "test_case_name" passed.123 124::125 126 not ok 1 test_case_name127 128The test "test_case_name" failed.129 130::131 132 ok 1 test # SKIP necessary dependency unavailable133 134The test "test" was SKIPPED with the diagnostic message "necessary dependency135unavailable".136 137::138 139 not ok 1 test # TIMEOUT 30 seconds140 141The test "test" timed out, with diagnostic data "30 seconds".142 143::144 145 ok 5 check return code # rcode=0146 147The test "check return code" passed, with additional diagnostic data “rcode=0”148 149 150Diagnostic lines151----------------152 153If tests wish to output any further information, they should do so using154"diagnostic lines". Diagnostic lines are optional, freeform text, and are155often used to describe what is being tested and any intermediate results in156more detail than the final result and diagnostic data line provides.157 158Diagnostic lines are formatted as "# <diagnostic_description>", where the159description can be any string. Diagnostic lines can be anywhere in the test160output. As a rule, diagnostic lines regarding a test are directly before the161test result line for that test.162 163Note that most tools will treat unknown lines (see below) as diagnostic lines,164even if they do not start with a "#": this is to capture any other useful165kernel output which may help debug the test. It is nevertheless recommended166that tests always prefix any diagnostic output they have with a "#" character.167 168Unknown lines169-------------170 171There may be lines within KTAP output that do not follow the format of one of172the four formats for lines described above. This is allowed, however, they will173not influence the status of the tests.174 175This is an important difference from TAP. Kernel tests may print messages176to the system console or a log file. Both of these destinations may contain177messages either from unrelated kernel or userspace activity, or kernel178messages from non-test code that is invoked by the test. The kernel code179invoked by the test likely is not aware that a test is in progress and180thus can not print the message as a diagnostic message.181 182Nested tests183------------184 185In KTAP, tests can be nested. This is done by having a test include within its186output an entire set of KTAP-formatted results. This can be used to categorize187and group related tests, or to split out different results from the same test.188 189The "parent" test's result should consist of all of its subtests' results,190starting with another KTAP version line and test plan, and end with the overall191result. If one of the subtests fail, for example, the parent test should also192fail.193 194Additionally, all lines in a subtest should be indented. One level of195indentation is two spaces: " ". The indentation should begin at the version196line and should end before the parent test's result line.197 198"Unknown lines" are not considered to be lines in a subtest and thus are199allowed to be either indented or not indented.200 201An example of a test with two nested subtests:202 203::204 205 KTAP version 1206 1..1207 KTAP version 1208 1..2209 ok 1 test_1210 not ok 2 test_2211 # example failed212 not ok 1 example213 214An example format with multiple levels of nested testing:215 216::217 218 KTAP version 1219 1..2220 KTAP version 1221 1..2222 KTAP version 1223 1..2224 not ok 1 test_1225 ok 2 test_2226 not ok 1 test_3227 ok 2 test_4 # SKIP228 not ok 1 example_test_1229 ok 2 example_test_2230 231 232Major differences between TAP and KTAP233--------------------------------------234 235================================================== ========= ===============236Feature TAP KTAP237================================================== ========= ===============238yaml and json in diagnosic message ok not recommended239TODO directive ok not recognized240allows an arbitrary number of tests to be nested no yes241"Unknown lines" are in category of "Anything else" yes no242"Unknown lines" are incorrect allowed243================================================== ========= ===============244 245The TAP14 specification does permit nested tests, but instead of using another246nested version line, uses a line of the form247"Subtest: <name>" where <name> is the name of the parent test.248 249Example KTAP output250--------------------251::252 253 KTAP version 1254 1..1255 KTAP version 1256 1..3257 KTAP version 1258 1..1259 # test_1: initializing test_1260 ok 1 test_1261 ok 1 example_test_1262 KTAP version 1263 1..2264 ok 1 test_1 # SKIP test_1 skipped265 ok 2 test_2266 ok 2 example_test_2267 KTAP version 1268 1..3269 ok 1 test_1270 # test_2: FAIL271 not ok 2 test_2272 ok 3 test_3 # SKIP test_3 skipped273 not ok 3 example_test_3274 not ok 1 main_test275 276This output defines the following hierarchy:277 278A single test called "main_test", which fails, and has three subtests:279- "example_test_1", which passes, and has one subtest:280 281 - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1"282 283- "example_test_2", which passes, and has two subtests:284 285 - "test_1", which is skipped, with the explanation "test_1 skipped"286 - "test_2", which passes287 288- "example_test_3", which fails, and has three subtests289 290 - "test_1", which passes291 - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails.292 - "test_3", which is skipped with the explanation "test_3 skipped"293 294Note that the individual subtests with the same names do not conflict, as they295are found in different parent tests. This output also exhibits some sensible296rules for "bubbling up" test results: a test fails if any of its subtests fail.297Skipped tests do not affect the result of the parent test (though it often298makes sense for a test to be marked skipped if _all_ of its subtests have been299skipped).300 301See also:302---------303 304- The TAP specification:305 https://testanything.org/tap-version-13-specification.html306- The (stagnant) TAP version 14 specification:307 https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md308- The kselftest documentation:309 Documentation/dev-tools/kselftest.rst310- The KUnit documentation:311 Documentation/dev-tools/kunit/index.rst312