# BST-TIMEOUT: 2400 # 53 fresh guest processes carrying 428 cmocka cases between # # them -- schema compilation, XML/JSON/LYB parsing and # # validation, each a full libyang context built and torn down -- # # plus nine ptest-runner invocations and the tmpfs staging each # # one needs (see the chunking note below). A trip costs the whole # # run's ==BST-END== sentinel, so this asks for the cap. # # THE MARKER IS ON LINE 1 BECAUSE THE RUNNER ONLY READS THE FIRST 20 LINES for it # (brintos-selftest, _suite_bound). Below that window it is not "ignored" in a way # anyone would see: the suite simply runs under the 60s default while its own file # says 2400, which is the failure a bound is supposed to prevent. # libyang 2.1.148 (the YANG data-modelling language parser and toolkit) -- BRINTOS-517. # # Drives UPSTREAM's own ptest suite under ptest-runner and scores it against a native # x86_64 control built from the identical sources with the identical patches # (data/libyang-native.txt, regenerated by src/libyang_mknative.sh). The control fixes # the binary names, the case names and the expected counts BEFORE the image is booted, # so a target red is a statement about this arm rather than about the suite. # # TWO LAYERS, AND THEY COME FROM DIFFERENT PLACES. upstream's run-ptest reports one # verdict per BINARY -- 53 lines -- and redirects each binary's own stdout into # tests/.out, so cmocka's per-case lines never reach ptest-runner. The control # therefore carries both, and this file reads each from where it actually appears: # PASS: ptest-runner's stdout # CASE: / tests/.out, cmocka's own "[ OK ] " # Unlike libtdb, EVERY binary here prints a line per case, so layer 2 covers the whole # suite rather than one programme of it: 428 cases against 53 exit statuses. # # It then asserts what the ptest structurally CANNOT speak for. Every ptest binary is # statically linked against the objects, so the whole suite would pass unchanged on an # image where the shipped archive, headers, pkg-config file and YANG models were all # missing -- the ptest tests the code, not the port. And the ptest never runs yanglint # or yangre, so the two tools this port puts in ${bindir} would be equally unexercised. # . "$BST_LIB" PT=/usr/lib/libyang/ptest CTL=$(bst_local "$BST_DATA/libyang-native.txt") # tmpfs copy: a `done < "$CTL"` runs in this shell (brintos/linux#73) TD=${TMPDIR:-/tmp} require yanglint libyang/present # --------------------------------------------------------------------------- # 1. What the port ships. The ptest cannot fail on any of these being absent. # --------------------------------------------------------------------------- # # The archive, not a shared object: on a static-only arm ENABLE_STATIC=ON is the only # buildable setting, so libyang.a IS the library this port delivers. check libyang/archive test -s /usr/lib/libyang.a check libyang/header test -s /usr/include/libyang/libyang.h check libyang/pkgconfig test -s /usr/lib/pkgconfig/libyang.pc # ...and they have to carry the real thing, not merely exist: a packaging pass that # shipped a truncated archive or a stub header passes the size checks above and # nothing else here would notice. # # THE API CHECK IS ON context.h, NOT ON libyang.h, and that is the header layout # rather than a detail: libyang.h is an umbrella of fourteen #includes and declares no # function of its own, so `grep ly_ctx_new libyang.h` is red against a perfectly good # install. It was, on the first target run. Both halves are asserted -- the umbrella # pulls context.h in, and context.h declares the entry point -- because a -dev package # that shipped one without the other is broken in a way neither check alone sees. check libyang/header-umbrella grep -q "include \"context.h\"" /usr/include/libyang/libyang.h check libyang/header-api grep -q ly_ctx_new /usr/include/libyang/context.h check_out libyang/pc-version "2.1.148" sh -c "sed -n 's/^Version: //p' /usr/lib/pkgconfig/libyang.pc" # The eight YANG models the library compiles in are a package of their own here -- # FILES:${PN} on this arm is the ${datadir}/yang tree and nothing else, because the # .so glob it would otherwise match cannot exist. A context that cannot find # ietf-yang-types cannot compile most real models, so this is load-bearing, not # inventory. check libyang/models test -s /usr/share/yang/modules/libyang/ietf-yang-types@2013-07-15.yang # The suite payload itself, so a truncated ptest package is not read as a short run. check libyang/ptest-payload test -x "$PT/run-ptest" check libyang/ptest-modules test -d "$PT/tests/modules" # --------------------------------------------------------------------------- # 2. The two tools, driven as a user would drive them. # --------------------------------------------------------------------------- # # This is the only thing here that exercises the ported library through its own # command-line front ends, on files on the target's filesystem. # # /tmp, not the rootfs: the root filesystem is virtio-fs and several ports have found # it refusing operations tmpfs accepts, which would make a libyang failure # indistinguishable from a filesystem one. T=$TD/bst-libyang.$$ rm -rf "$T"; mkdir -p "$T" cat > "$T/bst-libyang.yang" <<'YEOF' module bst-libyang { namespace "urn:brintos:bst:libyang"; prefix b; leaf greeting { type string; } leaf count { type uint8 { range "1..10"; } } } YEOF cat > "$T/good.xml" <<'XEOF' hello 7 XEOF cat > "$T/bad.xml" <<'XEOF' 99 XEOF check_out libyang/tool-version "yanglint 2.1.148" yanglint --version # Schema parse + compile, reported in upstream's own tree rendering. check_out libyang/tool-tree "+--rw greeting? string" yanglint -f tree "$T/bst-libyang.yang" # The real round trip, and the one that proves the parser and the printer are both # live: read a schema, read an XML instance against it, validate it, and print it back # in a DIFFERENT encoding. A library that had lost either half still passes every # file-existence check above. check_out libyang/tool-xml2json '"bst-libyang:count": 7' yanglint -f json "$T/bst-libyang.yang" "$T/good.xml" # ...and validation has to REFUSE what the model forbids. Without this the case above # is satisfied by a build that parses and prints and never checks anything: `count` # is restricted to 1..10 and 99 must not survive it. yanglint exits 1 on a data file # it could not parse, so the status is the verdict and the message is corroboration. check_rc libyang/tool-range-rejected 1 yanglint "$T/bst-libyang.yang" "$T/bad.xml" # yangre is the port's only user-facing exercise of the libpcre2 dependency the # ticket names: YANG patterns are compiled through PCRE2, so a match and a refusal # together say the static libpcre2-8.a is linked in and working. check_rc libyang/tool-re-match 0 yangre -p "[a-z]+" brintos check_rc libyang/tool-re-nomatch 2 yangre -p "[a-z]+" BRINTOS rm -rf "$T" # --------------------------------------------------------------------------- # 3. Upstream's suite, under ptest-runner, joined against the native control. # --------------------------------------------------------------------------- if [ ! -s "$CTL" ]; then bst_notok libyang/control-present "native control $CTL missing or empty" return 0 2>/dev/null || exit 0 fi bst_ok libyang/control-present if ! command -v ptest-runner >/dev/null 2>&1; then bst_skip libyang/ptest-runner "ptest-runner not installed on this image" return 0 2>/dev/null || exit 0 fi bst_ok libyang/ptest-runner # UPSTREAM'S run-ptest CANNOT RUN IN PLACE, AND THE TREE IS TOO BIG TO STAGE WHOLE. # Two constraints meet here, and the suite has to satisfy both at once. # # (a) run-ptest's line 4 is `cd "$ptestdir"/tests`, and chdir(2) on the virtiofs root # returns the raw kernel restart sentinel 513 -- so run in place the script dies # before a single binary runs, ptest-runner prints BEGIN/END with no verdicts, and # all 53 binaries score as ``. Measured exactly that way on the first # target run of this suite. $PT is writable; writability is not the cause. libtdb # and libtevent hit the same refusal on the same line of the same generated script, # so it is shared infrastructure and not this package. # BRINTOS-WORKAROUND: brintos/linux#73 remove -- when chdir(2) stops returning the # restart sentinel on the virtiofs root, run `ptest-runner libyang` in place again # and delete the staging below. # # (b) libtdb's answer -- copy the whole ptest tree to tmpfs -- DOES NOT SCALE HERE. # 53 statically-linked wasm binaries are 694 MiB, and the guest's /tmp is a # RAM-backed tmpfs in a 1 GiB machine, so the copy fails outright: measured # `could not stage /usr/lib/libyang/ptest under /tmp` on that same run, which is # what sent it down the in-place path and into (a). # # So the tree is staged in CHUNKS: run-ptest (340 bytes) plus a few binaries at a # time, one ptest-runner invocation per chunk, freed before the next. The binaries # carry their data paths as ABSOLUTE strings baked in at build time # (/usr/lib/libyang/ptest/tests/modules/yang, from the recipe's tests_config.h sed), # so they read their YANG modules from the real tree wherever they are executed from # -- reads across virtiofs work, it is only chdir(2) that does not. Chunking also # bounds the blast radius of linux-in-the-browser#105, where ptest-runner itself traps # part-way through a long suite and takes every binary after it: here that costs one # chunk rather than the tail of the run. PTRUN=$TD/bst-libyang-ptest.$$ ACC=$TD/bst-libyang-acc.$$ # accumulated "/" lines, all chunks OUTF=$TD/bst-libyang-out.$$ # accumulated ptest-runner stdout, all chunks : > "$ACC"; : > "$OUTF" # The staging tree is created ONCE and refilled, rather than removed and remade per # chunk. On the run that introduced chunking, chunk 1 -- and only chunk 1 -- came back # with BEGIN/END and no verdicts, which is the signature of run-ptest's `cd tests` # failing; the README records chdir(2) returning 513 "on a tmpfs directory the suite # had just created". Nine chunks each remaking the directory is nine chances to lose # that race, so it is made once and reused, and each chunk only replaces its contents. mkdir -p "$PTRUN/libyang/ptest/tests" 2>/dev/null cp "$PT/run-ptest" "$PTRUN/libyang/ptest/" 2>/dev/null chmod 755 "$PTRUN/libyang/ptest/run-ptest" 2>/dev/null check libyang/ptest-staged test -x "$PTRUN/libyang/ptest/run-ptest" total=0 for f in "$PT"/tests/utest_*; do [ -f "$f" ] && total=$((total+1)); done CHUNK=6 n=0; i=0; chunks=0; staged=0; retried=0 for f in "$PT"/tests/utest_*; do [ -f "$f" ] || continue [ "$n" -eq 0 ] && rm -f "$PTRUN"/libyang/ptest/tests/* 2>/dev/null cp "$f" "$PTRUN/libyang/ptest/tests/" 2>/dev/null && { n=$((n+1)); staged=$((staged+1)); } i=$((i+1)) if [ "$n" -ge "$CHUNK" ] || [ "$i" -ge "$total" ]; then chmod 755 "$PTRUN"/libyang/ptest/tests/utest_* 2>/dev/null chunks=$((chunks+1)) co=$(ptest-runner -d "$PTRUN" libyang 2>&1) # A CHUNK THAT REPORTED NOTHING IS RETRIED ONCE, AND ITS OUTPUT IS PRINTED. # Both halves are corrections to the first chunked run. It reported nothing # for chunk 1 and the reason was unrecoverable afterwards, because the chunk's # own stdout had gone into $OUTF and $OUTF is deleted at the end -- the suite # had thrown away the one thing that would have named the cause. So an empty # verdict set now carries the chunk's own first line into the result detail. # The retry is the documented remedy for an INTERMITTENT chdir(2) 513, not a # way to make a deterministic failure quiet: if the second attempt is empty # too, that is reported as its own red. if ! printf '%s\n' "$co" | grep -qE '^(PASS|FAIL): '; then first=$co retried=$((retried+1)) co=$(ptest-runner -d "$PTRUN" libyang 2>&1) if ! printf '%s\n' "$co" | grep -qE '^(PASS|FAIL): '; then bst_notok "libyang/chunk$chunks-verdicts" \ "no PASS/FAIL from either attempt: $(_bst_1line "$first")" fi fi printf '%s\n' "$co" >> "$OUTF" for o in "$PTRUN"/libyang/ptest/tests/*.out; do [ -f "$o" ] || continue b=${o##*/}; b=${b%.out} sed -n "s|^\[ *OK *\] \(.*\)$|$b/\1|p" "$o" >> "$ACC" done n=0 fi done # Retries are reported rather than hidden: a run that needed one is a run that met the # 513 race, and a suite that stops mentioning it is how a platform defect gets counted # as fixed. Zero is the expected value and the only green one. check_eq libyang/chunk-retries 0 "$retried" # THE TOTALS ARE EMITTED BEFORE THE PER-ITEM DETAIL, ON PURPOSE. On the first target # run this suite's shell died five binaries short of the end of its own reporting loop # -- 65 cases where 73 were owed -- and because the completeness checks sat after that # loop, the one set of results that would have SAID the run was short went missing # with it. A truncated tail must cost detail, never the verdict. nctlbin=$(grep -c '^PASS: ' "$CTL") check_eq libyang/staged-complete "$nctlbin" "$staged" ntgtbin=$(grep -cE '^(PASS|FAIL): ' "$OUTF") check_eq libyang/bin-complete "$nctlbin" "$ntgtbin" nctl=$(grep -c '^CASE: ' "$CTL") ntgt=$(wc -l < "$ACC" | tr -d ' ') check_eq libyang/join-complete "$nctl" "$ntgt" # Every chunk's ptest-runner reached the end of its own run rather than dying inside # it. Exactly one END marker per chunk -- only the LAST attempt's output is appended, # so a retry does not inflate this -- and fewer means a trap took one # (linux-in-the-browser#105). A chunk that printed BEGIN/END and no verdicts is green # HERE and red at bin-complete; the two say different things and both are wanted. nend=$(grep -c "END: .*libyang/ptest" "$OUTF") check_eq libyang/ptest-completed "$chunks" "$nend" # Layer 1 -- run-ptest's own verdict per binary, driven from the control so a binary # the target never reported is a red rather than a silently shorter run. # # A `for` over a command substitution rather than `while read < "$CTL"`: the read loop # is what stopped five short on the first run, and a loop whose iteration count cannot # be cut by an interrupted read is one fewer way for this to happen quietly. for name in $(sed -n 's/^PASS: //p' "$CTL"); do if grep -qx "PASS: $name" "$OUTF"; then bst_ok "libyang/bin-$name" else got=$(grep -xE "(FAIL|SKIP): $name" "$OUTF" | head -1) bst_notok "libyang/bin-$name" "native PASS, target: ${got:-}" fi done # Layer 2 -- every cmocka case the control proved green natively must be green here. # # ONLY THE MISSING ONES ARE EMITTED INDIVIDUALLY. 428 green case lines would be more # console traffic than any suite on this image produces, on a guest whose console is # the scarce resource -- and they would say nothing that `join-complete` above does not # already say. A case that is MISSING is named in full, which is the half that carries # information. The set difference is computed once, with sort+comm, so this whole layer # costs four processes rather than 428. ctlc=$TD/bst-libyang-ctl.$$ tgtc=$TD/bst-libyang-tgt.$$ sed -n 's|^CASE: ||p' "$CTL" | sort > "$ctlc" sort "$ACC" > "$tgtc" missing=$(comm -23 "$ctlc" "$tgtc") if [ -z "$missing" ]; then bst_ok libyang/cases-all-green else for c in $missing; do bst_notok "libyang/case-$c" "native OK, target: no cmocka OK line" done fi rm -rf "$PTRUN" "$ACC" "$OUTF" "$ctlc" "$tgtc"