# BST-TIMEOUT: 2400 # 16 fresh guest processes carrying 270 cases between them, and # # almost every case forks a server child and drives it over a # # loopback TCP connection -- so the suite pays a process # # creation and a socket round trip per case, not a function # # call. Natively the same 16 binaries take 8 seconds; the # # bound is set for the arm rather than for the native cost. A # # trip costs the whole run's ==BST-END== sentinel, so it is # # generous on purpose and still under the runner's 3600 cap. # # THE MARKER IS ON LINE 1 BECAUSE THE RUNNER ONLY READS THE FIRST 20 LINES for it # (brintos-selftest, _suite_bound). Below that window it is not "ignored" in a way # anyone would see: the suite simply runs under the 60s default while its own file # says 2400, which is the failure a bound is supposed to prevent. # neon 0.33.0 (an HTTP and WebDAV client library with a C interface) -- BRINTOS-518. # # Drives UPSTREAM's own ptest suite under ptest-runner and scores it against a native # x86_64 control built from the identical tarball with the identical patches and the # identical configure line (data/neon-native.txt, regenerated by src/neon_mknative.sh). # The control fixes the binary names, the case names and the expected counts BEFORE # the image is booted, so a target red is a statement about this arm rather than about # neon or about the suite. # # TWO LAYERS, BOTH READ FROM ONE STREAM. Unlike libyang, neon's run-ptest does not # redirect each binary's stdout to a file -- it runs `./$t` straight into its own # stdout -- so ptest-runner's capture carries both the per-binary verdict and every # per-case line, interleaved in run order: # PASS: run-ptest's own line, i.e. the binary's exit status # CASE: / the harness's `NN. ...... pass` line # The second is attributed to the first by position: run-ptest echoes `Running ...` # before each binary and that marker is what the join keys on. # # It then asserts what the ptest structurally CANNOT speak for. Every test binary # links libneon.a at build time, so the whole suite would pass unchanged on an image # where the shipped archive, the headers and the pkg-config file were all missing -- # the ptest tests the code, not the port. neon installs no user-facing tool at all # (neon-config is Yocto's disabled stub), so those packaging assertions are the only # thing standing between "the suite is green" and "the port shipped anything". # . "$BST_LIB" PT=/usr/lib/neon/ptest PC=/usr/lib/pkgconfig/neon.pc CTL=$(bst_local "$BST_DATA/neon-native.txt") # tmpfs copy: read as an argument here, # but kept uniform with the suites that # must redirect from it (brintos/linux#73) TD=${TMPDIR:-/tmp} # The 16 binaries meta-oe's do_install_ptest packages, in the order its run-ptest # runs them. Named here rather than globbed: the staging below has to reproduce # run-ptest's fixed list exactly, because a binary the script names and the tree does # not hold scores as a FAIL rather than as a shorter run. NEON_TESTS="auth basic redirect request session socket string-tests stubs uri-tests \ util-tests acl3744 lock oldacl props xml xmlreq" # --------------------------------------------------------------------------- # 1. What the port ships. The ptest cannot fail on any of these being absent. # --------------------------------------------------------------------------- # # The archive, not a shared object: this distro appends --disable-shared to every # class-target configure, so libneon.a IS the library this port delivers. check neon/archive test -s /usr/lib/libneon.a check neon/header test -s /usr/include/neon/ne_session.h check neon/pkgconfig test -s "$PC" # ...and they have to carry the real thing, not merely exist: a packaging pass that # shipped a truncated archive or a stub header passes the size checks above and # nothing else here would notice. ne_session_create is the entry point every neon # consumer calls first, so it is asserted in both places -- declared in the header # and actually present in the archive. check neon/header-api grep -q ne_session_create /usr/include/neon/ne_session.h check neon/archive-api grep -q ne_session_create /usr/lib/libneon.a # --enable-webdav is a PACKAGECONFIG this port keeps, and six of the sixteen ptest # binaries are the DAV suites, so the header that arm installs is asserted rather # than assumed: without it the library is HTTP-only and half the gate is meaningless. check neon/header-webdav grep -q ne_lock_discover /usr/include/neon/ne_locks.h check_out neon/pc-version "0.33.0" sh -c "sed -n 's/^Version: //p' $PC" # THE TWO Libs.private CHECKS ARE THE REGRESSION GUARD FOR THIS PORT'S ONE RECIPE # CHANGE, and the negative is the load-bearing half. # # meta-oe's pkgconfig.patch replaces neon.pc.in's `Libs.private: @NEON_LIBS@` with a # hardcoded `-lz -lgcrypt -lgpg-error -lexpat -lgnutls`, which is true only of its # default gnutls arm. On a static-only image Libs.private IS the link line # `pkg-config --static --libs neon` hands a consumer, so with that patch applied this # port shipped a .pc naming three archives the image does not contain. The bbappend # drops the patch and the substitution comes back right by construction. # # A positive check alone would not catch a regression: `-lexpat` is in the hardcoded # string too. Only the absence of gnutls distinguishes the two. check_out neon/pc-private-libs "-lexpat" sh -c "sed -n 's/^Libs.private://p' $PC" check_rc neon/pc-no-gnutls 1 grep -q gnutls "$PC" # neon installs no user-facing tool. neon-config is what upstream's autotools build # would install and what binconfig-disabled.bbclass replaces with a refusal, so this # says two things at once: the -dev package landed, and what landed is the stub # rather than a live script that would hand a consumer this build host's paths. check_rc neon/config-refuses 1 neon-config --libs # The suite payload itself, so a truncated ptest package is not read as a short run. check neon/ptest-payload test -x "$PT/run-ptest" n=0 for t in $NEON_TESTS; do [ -x "$PT/test/$t" ] && n=$((n+1)); done check_eq neon/ptest-binaries 16 "$n" # --------------------------------------------------------------------------- # 2. Upstream's suite, under ptest-runner, joined against the native control. # --------------------------------------------------------------------------- if [ ! -s "$CTL" ]; then bst_notok neon/control-present "native control $CTL missing or empty" return 0 2>/dev/null || exit 0 fi bst_ok neon/control-present if ! command -v ptest-runner >/dev/null 2>&1; then bst_skip neon/ptest-runner "ptest-runner not installed on this image" return 0 2>/dev/null || exit 0 fi bst_ok neon/ptest-runner # UPSTREAM'S run-ptest CANNOT RUN IN PLACE, FOR TWO REASONS AT ONCE, AND ONLY ONE OF # THEM IS THE ONE THE OTHER PTEST SUITES ON THIS IMAGE HIT. # # (a) run-ptest's line 10 is `cd test`, and chdir(2) on the virtiofs root returns the # raw kernel restart sentinel 513 -- so run in place the script dies under its # own `set -e` before a single binary runs, and all 16 score as ``. # MEASURED HERE, not inherited: `ptest-runner neon` in place produced # `RESTART-SENTINEL-LEAK val=513 nr=80 site=syscall_ret` (80 is chdir) and then # `run-ptest: cd: line 10: can't cd to test: Unknown error 513`. It is # INTERMITTENT -- twenty chdir(2) calls into the same directory from this # suite's own shell, in the same boot, all succeeded -- which is why the staging # is unconditional rather than a fallback. libtdb, libtevent and libyang meet the # same refusal on the same line of their own generated scripts, so it is shared # infrastructure and not this package. # # (b) neon's harness needs a WRITABLE cwd whatever chdir does, which the others do # not. test/common/tests.c fopen()s debug.log and child.log before it runs a # single case and `return -1`s if either fails, and run-ptest itself writes # foobar.txt -- a fixture three of the request cases read. So even on a kernel # where (a) is fixed, the ptest directory being read-only would leave this suite # with sixteen binaries that exit non-zero having run nothing. (It is writable on # this image -- measured -- so (b) is latent here and (a) is what bites.) # # A THIRD REASON WAS FIXED IN THE RECIPE RATHER THAN HERE. run-ptest's two `ulimit` # calls return ENOSYS on this kernel (no rlimit syscall is wired: brintos/linux#20) # and `set -e` killed the script at line 6, before even reaching the chdir. That one # is corrected in neon_%.bbappend's do_install_ptest, so the SHIPPED script carries # `|| true` on both lines -- which means a developer running `ptest-runner neon` by # hand gets the same script this suite stages. Nothing about it is done here. # # THE BINARIES ARE SYMLINKED RATHER THAN COPIED, which is where this differs from # libtdb's staging. The tree is 66 MiB of statically linked wasm and the guest's /tmp # is a RAM-backed tmpfs in a 1 GiB machine, so copying it is affordable but not free; # nothing in the suite reads the binary through its own path (argv[0] is used only # for the display name), and reads and execs across virtiofs work -- it is chdir(2) # that does not. So only the 429-byte run-ptest is copied, and what lands on tmpfs is # the writable cwd the harness needs. A symlink that does not resolve executable is # replaced by a copy, so a guest that refuses them still runs the suite. # # BRINTOS-WORKAROUND: brintos/linux#73 remove -- when chdir(2) stops returning the # restart sentinel on the virtiofs root, (a) is gone; (b) still requires a writable # cwd, so the staging stays until the ptest directory is writable too. PTRUN=$TD/bst-neon-ptest.$$ OUTF=$TD/bst-neon-out.$$ mkdir -p "$PTRUN/neon/ptest/test" 2>/dev/null cp "$PT/run-ptest" "$PTRUN/neon/ptest/" 2>/dev/null chmod 755 "$PTRUN/neon/ptest/run-ptest" 2>/dev/null linked=0; copied=0 for t in $NEON_TESTS; do ln -s "$PT/test/$t" "$PTRUN/neon/ptest/test/$t" 2>/dev/null if [ -x "$PTRUN/neon/ptest/test/$t" ]; then linked=$((linked+1)) else rm -f "$PTRUN/neon/ptest/test/$t" 2>/dev/null cp "$PT/test/$t" "$PTRUN/neon/ptest/test/$t" 2>/dev/null \ && chmod 755 "$PTRUN/neon/ptest/test/$t" 2>/dev/null \ && copied=$((copied+1)) fi done check neon/ptest-staged test -x "$PTRUN/neon/ptest/run-ptest" check_eq neon/ptest-staged-tests 16 "$((linked+copied))" # A pipe rather than a redirect, in both directions. ptest-runner's stdout is never # held in a shell variable -- 270 case lines on a memory-tight guest -- and the read # side never performs a stdin redirect, which is the other half of brintos/linux#73. # tr is what makes the stream parseable at all: the harness writes each case as # `\rNN. name...... ` and then again with its verdict, so without splitting on CR # every binary's cases are one line. ptest-runner -d "$PTRUN" neon 2>&1 | tr '\r' '\n' > "$OUTF" # A RUN THAT REPORTED NOTHING CARRIES ITS OWN FIRST DIAGNOSTIC LINE INTO THE RESULT. # Without this the evidence goes into $OUTF and $OUTF is deleted at the end, so the # one thing that would name the cause is thrown away -- which cost this suite two # extra boots on its first target run, once for the ulimit ENOSYS and once for the # chdir 513, both of which were sitting in that stream. `grep -v` on the case lines # first, because the first line of a healthy run is ptest-runner's own banner and the # first line of a broken one is the shell's complaint. if ! grep -qE '^(PASS|FAIL):' "$OUTF"; then bst_notok neon/ptest-verdicts \ "no PASS/FAIL from ptest-runner: $(_bst_1line "$(grep -vE '^ *[0-9]+\. ' "$OUTF" | grep -vE '^(START|BEGIN|DURATION|END|STOP|[0-9]{4}-)' | head -3 | tr '\n' ' ')")" else bst_ok neon/ptest-verdicts fi # THE TOTALS ARE EMITTED BEFORE THE PER-ITEM DETAIL, ON PURPOSE. A suite whose shell # dies part-way through its own reporting loop must lose detail, never the verdict -- # so the three counts that SAY the run was short are asserted first. nctlbin=$(grep -c '^PASS: ' "$CTL") # run-ptest emits `PASS:$t` with no space where the control writes `PASS: $t`, so the # target stream is normalised into the control's shape once, here, rather than at # each of the three places that read it. sed -n 's/^\(PASS\|FAIL\):[ ]*\([^ ]*\)$/\1: \2/p' "$OUTF" > "$OUTF.bin" ntgtbin=$(grep -cE '^(PASS|FAIL): ' "$OUTF.bin") check_eq neon/bin-complete "$nctlbin" "$ntgtbin" # ptest-runner reached the end of the suite rather than dying inside it. Missing END # with verdicts present is the signature of linux-in-the-browser#105, where the # runner itself traps part-way and takes every binary after it. nend=$(grep -c "END: .*neon/ptest" "$OUTF") check_eq neon/ptest-completed 1 "$nend" # The per-case join. `Running ...` is run-ptest's own marker and is what attributes # a case line to a binary; the awk keys on it rather than on anything in the case line # itself, because the case name alone is not unique across the sixteen (lookup_localhost # appears in four of them). awk ' /^Running / { cur=$2; sub(/\.\.\.$/,"",cur); next } cur != "" && /^ *[0-9]+\. .* pass/ { line=$0 sub(/^ *[0-9]+\. /,"",line) sub(/\.* +pass.*$/,"",line) if (line != "") print "CASE: " cur "/" line } ' "$OUTF" | sort -u > "$OUTF.case" nctl=$(grep -c '^CASE: ' "$CTL") ntgt=$(grep -c '^CASE: ' "$OUTF.case") check_eq neon/join-complete "$nctl" "$ntgt" # Layer 1 -- run-ptest's own verdict per binary, driven from the CONTROL so a binary # the target never reported is a red rather than a silently shorter run. # # A `for` over a command substitution rather than `while read < "$CTL"`: a read loop # is what left 96-libyang five cases short of its own reporting on its first target # run, and a loop whose iteration count cannot be cut by an interrupted read is one # fewer way for that to happen quietly. for name in $(sed -n 's/^PASS: //p' "$CTL"); do if grep -qx "PASS: $name" "$OUTF.bin"; then bst_ok "neon/bin-$name" else got=$(grep -xE "(FAIL|SKIP): $name" "$OUTF.bin" | head -1) bst_notok "neon/bin-$name" "native PASS, target: ${got:-}" fi done # Layer 2 -- every case the control proved green natively must be green here. # # ONLY THE MISSING ONES ARE EMITTED INDIVIDUALLY. 270 green case lines would be more # console traffic than any suite on this image produces, on a guest whose console is # the scarce resource, and they would say nothing `join-complete` above does not # already say. A case that is MISSING is named in full, which is the half that carries # information. The set difference is computed once, with sort+comm, so this whole # layer costs four processes rather than 270. ctlc=$TD/bst-neon-ctl.$$ sed -n 's|^CASE: ||p' "$CTL" | sort > "$ctlc" sed -n 's|^CASE: ||p' "$OUTF.case" | sort > "$OUTF.tgt" missing=$(comm -23 "$ctlc" "$OUTF.tgt") if [ -z "$missing" ]; then bst_ok neon/cases-all-green else for c in $missing; do bst_notok "neon/case-$c" "native pass, target: no pass line" done fi rm -rf "$PTRUN" "$OUTF" "$OUTF.bin" "$OUTF.case" "$OUTF.tgt" "$ctlc"