# cukinia 0.6.2 -- Savoir-faire Linux's embedded-Linux validation framework. # # cukinia is a POSIX-sh test RUNNER, so this suite is a test framework testing a # test framework, and the distinction that keeps it honest is: it asserts what # cukinia REPORTS, never what the firmware under it scores. The bundled example # (cukinia.conf.sample) is written to fail about half its cases on purpose -- # they name a www-data user, an nginx process and a reachable www.google.com -- # so "how many of the sample's cases pass" is a property of the image, changes # with every port that lands, and is worth nothing as an assertion. What is # invariant, and what the ticket actually asks for, is that every case in the # file is EXECUTED and REPORTED, identically, in each output mode. # # Hence the shape below: the sample's 15 cases are counted in all three modes # and the three reports are required to AGREE -- same case count, same failure # count. A mode that silently dropped a case, or a formatter that lost the # verdict, cannot pass that even though each report would still look plausible # read on its own. # # FIFTEEN, FROM SIXTEEN DIRECTIVES, AND THE MISSING ONE IS NOT A BUG. The # sample's `cukinia_run_dir /etc/cukinia/tests.d` names a directory this image # does not have, and cukinia_run_dir skips a missing dir with `[ -d "$dir" ] || # continue` -- no case, no warning. The count is pinned at 15 rather than merely # cross-checked because SRCREV is pinned too: if it ever reads something else, # either upstream changed the sample or a directive stopped emitting, and both # are things a reviewer should be told rather than have averaged away. # # WGETRC IS SET, AND IT IS NOT A WORKAROUND FOR ANYTHING OF OURS. Two of the # sample's cases are cukinia_http_request, which is `wget -q -O /dev/null ` # with no bound of its own. This guest has no route to the internet, so both are # expected to fail -- but "fail" and "hang for wget's default 900 s read timeout, # three times over" are the same verdict and very different suites. GNU wget # reads $WGETRC, so the bound goes in there. It cannot change any verdict: with # no network the request fails at 5 s instead of eventually, and on a host that # does have one 5 s is not a close call. . "$BST_LIB" require cukinia cukinia/present SAMPLE=/usr/share/cukinia/cukinia.conf.sample t=${TMPDIR:-/tmp}/bst-cukinia.$$ mkdir -p "$t/pass.d" "$t/fail.d" printf 'timeout = 5\ntries = 1\n' > "$t/wgetrc" WGETRC="$t/wgetrc"; export WGETRC # PLAIN-MODE OUTPUT IS COLOURED ON THIS CONSOLE, AND EVERY COUNT BELOW HAS TO # ALLOW FOR IT. cukinia builds its verdict token with `_colorize green PASS`, # which emits a bare "PASS" only when the caller is writing somewhere that cannot # render it -- an output FORMAT is set (csv/junitxml), or -o redirected the # report to a file, or $TERM is one it does not recognise. Otherwise the token is # wrapped in SGR and the line reads `[[1;32mPASS[0m] ...`. # # The guest console is a recognised terminal, so a `grep '\[PASS\]'` here matches # NOTHING while csv and junitxml both report a full set of cases -- which is # exactly how it first presented: eleven reds that were all one wrong grep, on a # port that was working. Shell redirection does not suppress it, because cukinia # tests $__log_file (still the string "/dev/stdout") rather than isatty. # # So counts are taken from a de-coloured copy, and the colouring itself is # asserted further down rather than merely tolerated. esc=$(printf '\033') decolour() { sed "s/${esc}\[[0-9;]*m//g" "$1"; } # --- the interpreter itself --------------------------------------------------- check_out cukinia/version "Cukinia version 0.6.2" cukinia --version check_out cukinia/help "firmware validation framework" cukinia --help check cukinia/sample-installed test -f "$SAMPLE" # --- the bundled example, in each of the three output modes ------------------- # Plain is the default: one "[PASS]"/"[FAIL]" line per case, carrying whatever # `logging prefix` the config set (the sample sets "cukinia: "). # # $TERM is forced here rather than inherited, and the capture is reused as the # coloured-output evidence further down instead of being taken again. Both are # for the same reason: EVERY RUN OF THE SAMPLE COSTS ~45 FORKS ON A PLATFORM # WHERE FORKS ARE NOT FREE. cukinia forks per case (cukinia_runner, mktemp, and # a command substitution inside _colorize), forks leak coroutine park slots # (brintos/hwjs-cc#69, brintos/hardwarejs#17), and the guest's host process grows # with them until the runner's rate guard ends it -- which is exactly what # happened here at 2050 MiB/min against a 2048 bound, on a suite that had passed # twice. So the sample is run FIVE times, not seven, and every one that remains # earns its place: coloured plain, uncoloured plain, csv, junitxml, and the # junitxml stdout refusal (which exits before running a single case). # # BRINTOS-WORKAROUND: hardwarejs#20 revisit -- the suite is shaped around the # guest's memory growth under fork load rather than written the obvious way, # which would take the sample seven times and read more clearly. When #20 closes, # the merge below can be undone and each assertion given its own invocation. TERM=xterm cukinia "$SAMPLE" > "$t/plain.raw" 2>&1; plain_rc=$? decolour "$t/plain.raw" > "$t/plain.out" plain_n=$(grep -c '^cukinia: \[PASS\]\|^cukinia: \[FAIL\]' "$t/plain.out") plain_f=$(grep -c '^cukinia: \[FAIL\]' "$t/plain.out") check_eq cukinia/sample-plain-cases 15 "$plain_n" check_eq cukinia/sample-plain-rc 1 "$plain_rc" # The prefix is asserted separately from the count because the count above is # already anchored on it: if `logging prefix` stopped applying, every result line # would lose the prefix at once and sample-plain-cases would read 0 rather than # wrong. This case says which of the two happened -- so it is phrased as "no # result line is missing the prefix" rather than as a second count. (A count here # would have to be 16, not 15: the sample's closing cukinia_log is free text and # takes the prefix too, which is correct and is asserted just below.) check_eq cukinia/sample-plain-prefix 0 \ "$(grep -c '^\[PASS\]\|^\[FAIL\]' "$t/plain.out")" # The sample's closing `cukinia_log "result: $cukinia_failures failure(s)"` is # free text, not a case -- it is the counter, and it must agree with the lines. check_eq cukinia/sample-counter "result: $plain_f failure(s)" \ "$(sed -n 's/^cukinia: \(result: .*\)$/\1/p' "$t/plain.out")" # CSV: a header line, then one quoted row per case with four fields. cukinia -f csv "$SAMPLE" > "$t/out.csv" 2>&1; csv_rc=$? csv_n=$(grep -c "^'" "$t/out.csv") # ",FAIL," and not ",FAIL,$": a row is `'',,,` and the # sample sets neither class nor suite, so every row ends in two empty fields. csv_f=$(grep -c ",FAIL," "$t/out.csv") check_out cukinia/sample-csv-header "TEST MESSAGE, RESULT, TEST CLASS, TEST SUITE" \ head -n 1 "$t/out.csv" check_eq cukinia/sample-csv-cases 15 "$csv_n" check_eq cukinia/sample-csv-rc 1 "$csv_rc" # Every row must carry a verdict cukinia recognises. A formatter that emitted an # empty RESULT field would still produce 15 rows. check_eq cukinia/sample-csv-verdicts 15 \ "$(grep -c ",PASS,\|,FAIL," "$t/out.csv")" # --no-header drops the header and nothing else. Asserted against a two-case # config rather than the sample: what is under test is header suppression, and # re-running fifteen cases to count rows buys nothing but forks (see above). printf 'cukinia_cmd true\nnot cukinia_cmd false\n' > "$t/hdr.conf" cukinia -f csv "$t/hdr.conf" > "$t/hdr.csv" 2>&1 cukinia -f csv --no-header "$t/hdr.conf" > "$t/nohdr.csv" 2>&1 check_eq cukinia/sample-csv-noheader 2 "$(grep -c "^'" "$t/nohdr.csv")" check_eq cukinia/csv-header-is-the-only-difference 1 \ "$(( $(wc -l < "$t/hdr.csv") - $(wc -l < "$t/nohdr.csv") ))" # JUnit XML. It refuses stdout by design -- assert the refusal, since a mode that # quietly wrote XML to the console would pass every other case here. check_rc cukinia/junit-needs-outfile 1 cukinia -f junitxml "$SAMPLE" cukinia -f junitxml -o "$t/out.xml" "$SAMPLE" >/dev/null 2>&1; xml_rc=$? xml_cases=$(grep -c '' "$xml_decl" check_eq cukinia/sample-junit-cases 15 "$xml_cases" check_out cukinia/sample-junit-attrs 'tests="15"' grep -o 'tests="[0-9]*"' "$t/out.xml" check_out cukinia/sample-junit-root '' tail -n 1 "$t/out.xml" # Well-formedness, from a parser rather than from grep. xmllint ships in # libxml2-utils on this image; if a future image drops it the case skips rather # than silently downgrading to the structural checks above. if command -v xmllint >/dev/null 2>&1; then check cukinia/sample-junit-wellformed xmllint --noout "$t/out.xml" else bst_skip cukinia/sample-junit-wellformed "xmllint not installed on this image" fi # --- the three reports must describe the SAME run ----------------------------- # This is the assertion the ticket is really asking for. Each report above is # self-consistent; only together do they establish that no mode drops or # reclassifies a case. Note it is deliberately blind to WHICH cases failed -- # that is the image's business, not cukinia's. check_eq cukinia/modes-agree-cases "$plain_n" "$csv_n" check_eq cukinia/modes-agree-cases-xml "$plain_n" "$xml_cases" check_eq cukinia/modes-agree-failures "$plain_f" "$csv_f" check_eq cukinia/modes-agree-failures-xml "$plain_f" "$xml_fails" check_out cukinia/junit-failure-attr "failures=\"$plain_f\"" \ grep -o 'failures="[0-9]*"' "$t/out.xml" # --- and the colouring is asserted, not merely tolerated ---------------------- # Both directions, because either one alone also passes against a cukinia that # has stopped consulting $TERM at all. $TERM is forced rather than read from the # console: what is under test is cukinia's decision, and a case that instead # depended on whatever the guest's console advertises would change meaning the # day that changed, in a suite about a different subject. # # The TERM=xterm side is the plain.raw capture from the top of the file -- the # same invocation, not a second one. TERM=dumb cukinia "$SAMPLE" > "$t/dumb.out" 2>&1 check cukinia/plain-colours-known-term grep -q "\[${esc}\[1;32mPASS" "$t/plain.raw" check cukinia/plain-bare-unknown-term grep -q '\[PASS\]' "$t/dumb.out" check_eq cukinia/plain-no-escapes-dumb 0 "$(grep -c "$esc" "$t/dumb.out")" # Whatever the spelling, both must describe the same run as the capture above. check_eq cukinia/plain-term-agnostic-count "$plain_n" \ "$(grep -c '^cukinia: \[PASS\]\|^cukinia: \[FAIL\]' "$t/dumb.out")" # --- the primitives actually evaluate, under THIS image's shell --------------- # Everything above would pass against a cukinia that printed FAIL for every case. # These two configs pin the verdict itself: one whose every case must pass on any # working image, one whose every case must fail, and both exercise the `not` # inverter, so a broken inverter cannot make both green. # # The mount case is the three-argument form deliberately. Two arguments would be # the shorter and more natural spelling and it is what upstream's own README # shows -- but it can abort the entire config on a stricter /bin/sh, so it gets a # config to itself further down rather than the chance to truncate this one. Four # would drag in the mount OPTIONS, making a case about cukinia's matcher depend # on how this image happens to have mounted /proc. ln -sf /etc/passwd "$t/link" cat > "$t/pass.conf" < "$t/pass.d/01-pass"; chmod +x "$t/pass.d/01-pass" printf '#!/bin/sh\nexit 1\n' > "$t/fail.d/01-fail"; chmod +x "$t/fail.d/01-fail" cukinia "$t/pass.conf" > "$t/pass.raw" 2>&1; pass_rc=$? decolour "$t/pass.raw" > "$t/pass.out" check_eq cukinia/allpass-rc 0 "$pass_rc" check_eq cukinia/allpass-cases 10 "$(grep -c '\[PASS\]\|\[FAIL\]' "$t/pass.out")" check_eq cukinia/allpass-nofail 0 "$(grep -c '\[FAIL\]' "$t/pass.out")" # `as` renames the case in the report; without it the line would read # "Running "true" is successful". check_out cukinia/allpass-as-decorator "renamed by the as() decorator" \ cat "$t/pass.out" cat > "$t/fail.conf" < "$t/fail.raw" 2>&1; fail_rc=$? decolour "$t/fail.raw" > "$t/fail.out" check_eq cukinia/allfail-rc 1 "$fail_rc" check_eq cukinia/allfail-cases 5 "$(grep -c '\[PASS\]\|\[FAIL\]' "$t/fail.out")" check_eq cukinia/allfail-nopass 0 "$(grep -c '\[PASS\]' "$t/fail.out")" # The two-argument cukinia_mount, in a config of its own -- see the comment on # `cukinia_mount proc /proc proc` above for why it cannot share one. # # _cukinia_mount() reads `shift 2; local fstype="$1"; shift`, so the documented # two-argument form (README: "cukinia_mount [fstype] # [options]", and upstream's own example is `cukinia_mount sysfs /sys`) shifts # once past the end of "$@". POSIX leaves that unspecified and the shells split: # busybox ash and bash treat it as a no-op returning non-zero, dash treats it as # a fatal error and ABORTS THE WHOLE RUN -- and because the abort lands ahead of # _cukinia_prepare, not one case is emitted, cukinia exits 2 rather than 0 or 1, # and the report is EMPTY rather than short. # # WHICH SHELL IS DECIDED BY cukinia's OWN `#!/bin/sh`, NOT BY WHAT RAN IT. That # is worth stating because it is the whole of the host/target difference here and # it is easy to test wrongly: running this file under `busybox ash` on the build # host still gets a dash cukinia, since that host's /bin/sh is dash. Measured on # the same config, same file: /bin/sh 0 cases, dash 0, busybox ash 2, bash 2. # # So this image is on the tolerant side (/bin/sh -> busybox.nosuid) and both # cases below are expected to PASS on target while failing in a build-host dry # run -- an expected artifact of the host's /bin/sh, not a port defect. They are # pinned anyway, and in isolation, for two reasons: the whole point of this suite # is that a config runs to completion, and this is the one construct known to be # able to truncate one; and if /bin/sh on a future image were switched to dash, # this case is the one that would say so, instead of the truncation surfacing as # a dozen unrelated suites quietly losing their tails. printf 'cukinia_mount proc /proc\ncukinia_cmd true\n' > "$t/mount2.conf" cukinia "$t/mount2.conf" > "$t/mount2.raw" 2>&1; mount2_rc=$? decolour "$t/mount2.raw" > "$t/mount2.out" check_eq cukinia/mount-2arg-completes 2 \ "$(grep -c '\[PASS\]\|\[FAIL\]' "$t/mount2.out")" check_eq cukinia/mount-2arg-rc 0 "$mount2_rc" # cukinia_conf_include sources another config in place; the included cases must # land in the same report. printf 'cukinia_cmd true\ncukinia_cmd true\n' > "$t/inc.conf" printf 'cukinia_cmd true\ncukinia_conf_include %s/inc.conf\n' "$t" > "$t/main.conf" cukinia "$t/main.conf" > "$t/inc.raw" 2>&1; inc_rc=$? decolour "$t/inc.raw" > "$t/inc.out" check_eq cukinia/conf-include-rc 0 "$inc_rc" check_eq cukinia/conf-include-cases 3 "$(grep -c '\[PASS\]' "$t/inc.out")" # `logging class`/`logging suite` are junitxml-only directives, so they are the # one part of the config language plain mode cannot show. printf 'logging suite "brintos"\nlogging class "selftest"\ncukinia_cmd true\n' > "$t/cls.conf" cukinia -f junitxml -o "$t/cls.xml" "$t/cls.conf" >/dev/null 2>&1 check_out cukinia/junit-suite-name 'name="brintos"' grep -o 'name="brintos"' "$t/cls.xml" check_out cukinia/junit-class-name 'classname="selftest"' grep -o 'classname="selftest"' "$t/cls.xml" rm -rf "$t"