# BST-TIMEOUT: 600 # one instantiation of a ~2.9 MB probe + one pass over a 432 kB archive . "$BST_LIB" # snappy (meta-oe recipes-extended/snappy, 1.1.10 at SRCREV dc05e026) -- Google's # fast block compressor: no entropy coder, tuned for throughput rather than # ratio, and the codec LevelDB and many storage layers default to. Seventh port # of the C++ round (BRINTOS-1094), after abseil-cpp, re2, gflags, glog, fmt and # double-conversion. # # THE VERSION IS 1.1.10, not the 1.2.2 the ticket names -- that is upstream's # current release, and this layer's meta-openembedded carries 1.1.10. Everything # below is measured against 1.1.10. # # --------------------------------------------------------------------------- # TWO CONSUMERS, BECAUSE THEY PROVE DIFFERENT THINGS # # The PAYLOAD blocks assert what a downstream recipe reaches -- the four # headers, the archive, the four CMake package files -- plus the C++ arm # witnesses. This port has a witness double-conversion could not offer: it is # the first of the round whose archive carries VTABLES. snappy-sinksource.h # declares Source and Sink with nine virtual members, so _ZTVN6snappy6SourceE # and its siblings have to be in the symbol index, and a C compile of the same # sources could not put them there. # # data/snappy-probe is a CONSUMER of that payload: a program compiled against # these headers and LINKED AGAINST THIS BUILD'S libsnappy.a, built by the # recipe so it cannot drift from what shipped. # # THE PROBE IS WHY THIS PORT CAN CLAIM ANYTHING AT ALL. snappy is static-only # here and installs no program, so without a consumer the recipe ships four # headers and an archive -- and an undefined symbol in a static archive is not # an error until something links it. That is how glog (BRINTOS-1101) shipped # green and unlinkable, and what fmt's __udivmodti4 turned out to be # (BRINTOS-1102, linux-in-the-browser#155). Here the risk sits in the virtual # dispatch above: the sink/* cases DRIVE Compress(Source*, Sink*) rather than # merely linking it. # # WHAT IS COMPARED BYTE-FOR-BYTE. The probe's run is compared line for line # against data/snappy-native.txt -- the identical source built and run on x86-64 # from the snappy-native recipe the bbappend adds -- with NO tolerance band and # NO NORMALISATION AT ALL. Every corpus is generated inside the probe from a # literal or a hand-rolled xorshift64 (never rand(), whose sequence is a # property of the libc); nothing reads the clock, the environment, the # filesystem, an address or a locale-sensitive default; and no size_t, pointer # or long is printed at its native width, so the ILP32/LP64 split between the # arms cannot show up as a formatting difference. One differing line is a defect. # # THAT COMPARISON IS A REAL CLAIM, NOT A FORMALITY, AND IT WAS CHECKED BEFORE IT # WAS MADE. snappy's FindMatchLength has two implementations, and the arms do not # take the same one: snappy-internal.h:174 selects the specialised version on # __x86_64__/ARCH_ARM/ARCH_PPC and the generic one otherwise, so x86-64 takes the # first and wasm32 the second. Compiling both on ONE host, with the arch guard as # the only variable, produced identical output across all nine corpora -- so the # split does not move the emitted stream and a cross-arm byte-compare is a # statement about this port rather than about snappy's dispatch. # # THE ANCHORED CASES EXIST BECAUSE AN ORACLE CAN BE WRONG. A control captured # from an already-broken build would make every compared line agree with it. So # the things the snappy format and this library's contract fix independently -- # MaxCompressedLength's arithmetic, the round-trip verdicts, the rejection of # every truncated frame, the two alternative APIs agreeing with the buffer one -- # are ALSO asserted here against literals. Those hold whatever the control says. # --------------------------------------------------------------------------- # A root prefix, empty on the target and set only when this suite is validated # natively before it ships. The runner never sets it. R="${BST_SNAPPY_ROOT:-}" ARCHIVE=$R/usr/lib/libsnappy.a INCDIR=$R/usr/include CMDIR=$R/usr/lib/cmake/Snappy PROBE=$BST_DATA/snappy-probe NATIVE=$BST_DATA/snappy-native.txt # THE CANARY, which is not an assertion about snappy at all. # # brintos_local_verify.py loses a suite's FIRST BST result line before the host # parser ever sees it -- BRINTOS-1149, a defect in our own host-side tooling and # not in the runner, the guest or any package. The harness reports the deficit # (`declared N case(s), N-1 reached the console`) but computes its pass count as # total-minus-failures, so THE DROPPED LINE IS COUNTED AS A PASS. A suite whose # first case is a genuine `not ok` therefore loses the one line that mattered, # silently. # # So the first line out of this file asserts something already true. It hides # nothing -- the harness still reports the deficit and every real case below is # still counted. Same idiom, and the same reason, as 82-double-conversion # (BRINTOS-1103) and 83-flatbuffers (BRINTOS-1114). # # Remove this case when 1149 is fixed. bst_ok snappy/suite-canary if [ ! -f "$ARCHIVE" ]; then bst_skip snappy/present "snappy not installed on this image" exit 0 fi bst_ok snappy/present nl=' ' tmp=${TMPDIR:-/tmp} # has # A shell `case` rather than a grep: this suite makes ~80 content assertions and # an exec apiece is what killed the first draft of the abseil suite on target, # where every process is a fresh instantiation of a multi-megabyte module. has() { case "$2" in *"$nl$3$nl"*) bst_ok "$1" ;; *) bst_notok "$1" "missing: $3" ;; esac } # sub sub() { case "$2" in *"$3"*) bst_ok "$1" ;; *) bst_notok "$1" "missing: $3" ;; esac } # hasnt -- substring, for "must NOT be there" hasnt() { case "$2" in *"$3"*) bst_notok "$1" "present but must not be: $3" ;; *) bst_ok "$1" ;; esac } # --- the package is whole ---------------------------------------------------- # # Four headers, not three: snappy-stubs-public.h does NOT exist in the source # tree. cmake generates it from snappy-stubs-public.h.in at configure time, so # its presence is evidence the configure step ran and substituted, and its # contents are checked below for what it substituted. check sn/hdr-snappy test -f "$INCDIR/snappy.h" check sn/hdr-snappy-c test -f "$INCDIR/snappy-c.h" check sn/hdr-snappy-sinksource test -f "$INCDIR/snappy-sinksource.h" check sn/hdr-snappy-stubs-public test -f "$INCDIR/snappy-stubs-public.h" check sn/archive-present test -f "$ARCHIVE" check sn/cmake-config test -f "$CMDIR/SnappyConfig.cmake" check sn/cmake-version test -f "$CMDIR/SnappyConfigVersion.cmake" check sn/cmake-targets test -f "$CMDIR/SnappyTargets.cmake" check sn/cmake-targets-noconfig test -f "$CMDIR/SnappyTargets-noconfig.cmake" # Shape, not absence: this arm converts to shared libraries per recipe # (BRINTOS-1740), so a .so here is a conversion, not a policy breach. bst_lib_shape sn/no-shared-lib "$R/usr/lib/libsnappy.so*" # --- the generated header states the version, and the arms agree on iovec ---- STUBS=$nl$(cat "$INCDIR/snappy-stubs-public.h" 2>/dev/null)$nl has sn/ver-major "$STUBS" "#define SNAPPY_MAJOR 1" has sn/ver-minor "$STUBS" "#define SNAPPY_MINOR 1" has sn/ver-patch "$STUBS" "#define SNAPPY_PATCHLEVEL 10" sub sn/ver-composite "$STUBS" "#define SNAPPY_VERSION" # HAVE_SYS_UIO_H is a configure PROBE against the wasm32 sysroot, and its answer # decides which struct iovec the public API uses: the system one, or snappy's own # fallback in namespace snappy. It resolved to 1 here, so CompressFromIOVec takes # the real type and the fallback is compiled out. Asserted in both # directions because a silent flip to the fallback would give downstream code a # DIFFERENT type of the same name -- which compiles and then does not link. has sn/uio-detected "$STUBS" "#if 1 // HAVE_SYS_UIO_H" has sn/uio-fallback-off "$STUBS" "#if !1 // !HAVE_SYS_UIO_H" # --- the declared API -------------------------------------------------------- SNH=$nl$(cat "$INCDIR/snappy.h" 2>/dev/null)$nl sub sn/api-compress "$SNH" "size_t Compress(const char* input, size_t input_length," sub sn/api-uncompress "$SNH" "bool Uncompress(const char* compressed, size_t compressed_length," sub sn/api-maxlen "$SNH" "size_t MaxCompressedLength(size_t source_bytes);" sub sn/api-isvalid "$SNH" "bool IsValidCompressedBuffer(const char* compressed," sub sn/api-getlen "$SNH" "bool GetUncompressedLength(const char* compressed, size_t compressed_length," sub sn/api-rawcompress "$SNH" "void RawCompress(const char* input," sub sn/api-rawuncompress "$SNH" "bool RawUncompress(const char* compressed, size_t compressed_length," sub sn/api-source-sink "$SNH" "size_t Compress(Source* source, Sink* sink);" SNC=$nl$(cat "$INCDIR/snappy-c.h" 2>/dev/null)$nl sub sn/capi-compress "$SNC" "snappy_status snappy_compress(const char* input," sub sn/capi-uncompress "$SNC" "snappy_status snappy_uncompress(const char* compressed," sub sn/capi-maxlen "$SNC" "size_t snappy_max_compressed_length(size_t source_length);" sub sn/capi-validate "$SNC" "snappy_status snappy_validate_compressed_buffer(const char* compressed," SNS=$nl$(cat "$INCDIR/snappy-sinksource.h" 2>/dev/null)$nl sub sn/sink-class "$SNS" "class Sink {" sub sn/source-class "$SNS" "class Source {" sub sn/bytearraysource "$SNS" "class ByteArraySource : public Source {" # --- the archive is this arm's, and it carries the C++ arm's own symbols ----- # # Read through the ar SYMBOL INDEX -- member "/", the map of exported symbol to # member offset -- rather than over the whole 432 kB body. A 16 kB window covers # it (the index is ~4.7 kB here) and is the more precise place to look: it lists # what the archive EXPORTS. Member NAMES are deliberately NOT asserted: only the # three longer than 15 characters live in the long-name table inside this window, # while snappy.cc.o is short enough to sit inline in its own member header far # deeper in the file -- measured, and the trap that would have cost four false # reds. check_out sn/ar-magic "!" head -c 8 "$ARCHIVE" idx=$tmp/bst-snappy-idx.$$ head -c 16384 "$ARCHIVE" | tr -c '\40-\176\n' ' ' > "$idx" 2>/dev/null IDX=$nl$(cat "$idx" 2>/dev/null)$nl rm -f "$idx" # THE VTABLES. This is the assertion double-conversion's suite could not make: # that library has no virtual function anywhere, so its C++ evidence had to come # from mangled names and __cxa_guard_*. snappy's Source and Sink are abstract, so # the vtable symbols must be exported, and a C compile of these sources could not # emit them. sub sn/sym-vtable-source "$IDX" "_ZTVN6snappy6SourceE" sub sn/sym-vtable-sink "$IDX" "_ZTVN6snappy4SinkE" sub sn/sym-vtable-bytearray "$IDX" "_ZTVN6snappy15ByteArraySourceE" # ...and NO typeinfo objects, because the recipe builds -fno-rtti. Asserted # deliberately rather than left unstated: _ZTV without _ZTI is exactly what # -fno-rtti produces, and a reader finding no _ZTI should be able to see that it # was expected rather than wonder whether the scan simply missed them. hasnt sn/sym-no-typeinfo "$IDX" "_ZTIN6snappy" sub sn/sym-compress "$IDX" "_ZN6snappy8CompressEPKcm" sub sn/sym-uncompress "$IDX" "_ZN6snappy10UncompressEPKcm" sub sn/sym-c-compress "$IDX" "snappy_compress" sub sn/sym-c-validate "$IDX" "snappy_validate_compressed_buffer" # libc++, not some other standard library: the bbappend's whole C++ opt-in is # pointing CXX at wasmc++-glibc so brintos-libcxx resolves. _ZNSt3__1 is libc++'s # inline namespace and nothing else emits it. sub sn/sym-libcxx "$IDX" "_ZNSt3__1" # The members are LLVM IR BITCODE, not native objects: this toolchain defers # codegen to link time, so every member opens with BC C0 DE (42 43 c0 de) -- not # the ELF magic a native x86-64 snappy would carry, and not the \0asm of a # finished wasm module either. This is the ONE assertion in the file that a # native build fails, which makes it the arm witness the headers cannot provide: # the probe's output is deliberately identical on both arms, so the probe cannot # be the thing that says which arm ran. # WALKED, NOT ASSUMED, BECAUSE THIS ARCHIVE HAS A LONG-NAME MEMBER. After the # symbol index "/" comes an OPTIONAL "//" member holding the names too long for # the 16-byte inline field -- three of these four are -- and only then the first # object. Skipping it unconditionally would land 112 bytes early, in the middle # of the name table, and read `snap` as the member magic: that is exactly what # the first dry run of this file did. 57-lzo.test's archive has no such member, # so its arithmetic stops one member sooner and cannot simply be copied. ARHDR=$(head -c 68 "$ARCHIVE" 2>/dev/null | tail -c 60) IDXSZ=$(printf '%s' "$ARHDR" | cut -c49-58 | tr -d ' ') if [ -n "$IDXSZ" ] && [ "$IDXSZ" -gt 0 ] 2>/dev/null; then bst_ok sn/ar-index-sized OFF=$((68 + IDXSZ)) [ $((OFF % 2)) -eq 1 ] && OFF=$((OFF + 1)) NAME2=$(od -An -c -j "$OFF" -N 2 "$ARCHIVE" 2>/dev/null | tr -d ' \n') if [ "$NAME2" = "//" ]; then bst_ok sn/ar-longname-member LNSZ=$(dd if="$ARCHIVE" bs=1 skip=$((OFF + 48)) count=10 2>/dev/null | tr -d ' ') OFF=$((OFF + 60 + LNSZ)) [ $((OFF % 2)) -eq 1 ] && OFF=$((OFF + 1)) else bst_notok sn/ar-longname-member "expected a // long-name member, read '$NAME2'" fi # Assert the name field before the magic, so a wrong magic below cannot be # blamed on arithmetic that landed somewhere other than a member header. MNAME=$(od -An -c -j "$OFF" -N 13 "$ARCHIVE" 2>/dev/null | tr -d ' \n') check_eq sn/ar-first-member "snappy-c.cc.o" "$MNAME" MAGIC=$(od -An -tx1 -j $((OFF + 60)) -N 4 "$ARCHIVE" 2>/dev/null | tr -d ' \n') check_eq sn/ar-member-bitcode 4243c0de "$MAGIC" if [ "$MAGIC" = "7f454c46" ]; then bst_notok sn/ar-not-elf "members are ELF -- this is a NATIVE build, not wasm32" else bst_ok sn/ar-not-elf; fi if [ "$MAGIC" = "0061736d" ]; then bst_notok sn/ar-not-wasm-module "members are finished wasm modules, not bitcode" else bst_ok sn/ar-not-wasm-module; fi else bst_notok sn/ar-index-sized "could not read the ar symbol-index size" fi # --- the probe: does the shipped archive COMPRESS, and compress correctly? --- if [ ! -f "$PROBE" ]; then bst_notok sn/probe-present "$PROBE missing -- the library cannot be proven to run" elif [ ! -f "$NATIVE" ]; then bst_notok sn/native-present "$NATIVE missing -- no x86-64 control for the probe run" else bst_ok sn/probe-present bst_ok sn/native-present # do_install stages data/* at 0644 and the deployed rootfs is read-only at # browser runtime, so the payload cannot be executed where it lies. Stage an # executable copy under TMPDIR, as every probe-driven suite here does. pd=$tmp/bst-snappy.$$ p=$pd/snappy-probe mkdir -p "$pd" 2>/dev/null if cp "$PROBE" "$p" 2>/dev/null && chmod 0755 "$p" 2>/dev/null && [ -x "$p" ]; then bst_ok sn/probe-exec out=$("$p" 2>"$pd/run.err"); rc=$? errout=$(cat "$pd/run.err" 2>/dev/null) rm -rf "$pd" if [ "$rc" -eq 0 ]; then bst_ok sn/probe-rc else bst_notok sn/probe-rc "probe exited $rc"; fi # The library writes nothing to stderr on a successful run. Its failure # channel under -fno-exceptions is assert() -> __assert_fail, which prints # there and abort()s -- so a port whose internal assertion fired on some # corpus would show up here even if the surviving output still matched line # for line. if [ -z "$errout" ]; then bst_ok sn/probe-silent-stderr else bst_notok sn/probe-silent-stderr "$(_bst_1line "$errout")"; fi O=$nl$out$nl # The whole-stream gate. Kept separate from the anchored cases below, because # a probe that died partway would leave those it never reached with nothing # to disagree with and could look like a shorter clean run. case "$O" in *"${nl}snappy-probe-end${nl}") bst_ok sn/probe-complete ;; *) bst_notok sn/probe-complete "probe did not reach its final line" ;; esac # The version the probe was COMPILED against (the recipe's PV) beside the # one the GENERATED HEADER carries. The recipe filename, the source's CMake # project() version and snappy-stubs-public.h are three separate claims; # this line is where a header/archive mismatch becomes visible rather than # inferred. All three read 1.1.10 at SRCREV dc05e026. 65802 = (1<<16)|(1<<8)|10. has sn/probe-version "$O" "snappy-probe 1.1.10 hdr=1.1.10 ver=65802" # === 1. the byte-compare against x86-64 ================================== if [ "$out" = "$(cat "$NATIVE" 2>/dev/null)" ]; then bst_ok sn/probe-matches-x86-64 else first=$(printf '%s\n' "$out" | diff - "$NATIVE" 2>/dev/null | head -4) bst_notok sn/probe-matches-x86-64 "$(_bst_1line "${first:-output differs from the control}")" fi # === 2. the round-trip contract, stated without the control ============== # # The ticket's core requirement: compress a corpus, decompress it back, and # require the result to be the input BYTE FOR BYTE. The probe emits a verdict # token per corpus; these cases turn the whole table into pass/fail without # depending on the oracle. # # THE COUNTS ARE ASSERTED, not just the absence of a failure token. A probe # that emitted nothing would satisfy "no RT-MISMATCH anywhere" perfectly -- # that is the vacuous-green shape this port's siblings kept finding -- so the # number of successful verdicts is pinned too. Nine corpora: text, # repetitive, random, already-compressed, runs, mixed, empty, one, sixteen. hasnt sn/no-rt-mismatch "$O" "RT-MISMATCH" hasnt sn/no-retlen-bad "$O" "RETLEN-BAD" hasnt sn/no-len-bad "$O" "LEN-BAD" hasnt sn/no-invalid "$O" "|INVALID" check_eq sn/rt-ok-count 9 "$(printf '%s\n' "$out" | grep -c '^rt/.*RT-OK$')" check_eq sn/valid-count 9 "$(printf '%s\n' "$out" | grep -c '^v/.*|VALID$')" check_eq sn/lenok-count 9 "$(printf '%s\n' "$out" | grep -c '^gl/.*|LEN-OK$')" # === 3. the two alternative APIs agree with the buffer API =============== # # Compress(Source*, Sink*) is the only virtual dispatch in this library, so # this is the case that proves the vtables asserted above are not merely # present but correct. The C entry points are a separate symbol set in the # same archive; a port can link the C++ ones and still be missing these. check_eq sn/sink-agrees-count 9 "$(printf '%s\n' "$out" | grep -c 'SAME-AS-BUFFER-API$')" check_eq sn/capi-agrees-count 9 "$(printf '%s\n' "$out" | grep -c 'SAME-AS-CXX')" hasnt sn/no-sink-divergence "$O" "DIFFERS-FROM-BUFFER-API" hasnt sn/no-capi-divergence "$O" "DIFFERS-FROM-CXX" hasnt sn/no-capi-failure "$O" "COMPRESS-FAIL" # === 4. a truncated frame is REJECTED, not silently accepted ============= # # The ticket's second explicit requirement. Eighteen cuts -- six positions # across three corpora -- and each is tried through BOTH doors: the # validator, which walks the stream without allocating, and Uncompress, # which is what a caller actually uses. A decoder that accepted a short # frame would hand its caller a buffer it never filled, which is a wrong # answer rather than an error. check_eq sn/trunc-rejected-count 18 "$(printf '%s\n' "$out" | grep -c '^trunc/.*|REJECTED$')" hasnt sn/no-trunc-accepted "$O" "ACCEPTED-BOTH" hasnt sn/no-trunc-validator "$O" "ACCEPTED-BY-VALIDATOR" hasnt sn/no-trunc-uncompress "$O" "ACCEPTED-BY-UNCOMPRESS" # === 5. anchored values ================================================== # # MaxCompressedLength is pure arithmetic on the input size -- 32 + n + n/6 -- # so these hold whatever the codec does and whatever the control says. They # are the cheapest possible witness that the archive linked at all, and the # first thing to read when the byte-compare goes red: if these are also # wrong, the archive is not snappy's. has sn/max-0 "$O" "max/0|32" has sn/max-1 "$O" "max/1|33" has sn/max-100 "$O" "max/100|148" has sn/max-8192 "$O" "max/8192|9589" has sn/max-1000000 "$O" "max/1000000|1166698" # The compressed lengths themselves, for the three shapes the ticket names. # THE LENGTH IS THE POINT: a codec that still round-trips but no longer # produces the same stream passes every case above and fails these. They are # stated here as literals as well as being covered by the byte-compare, so # that a missing or stale oracle cannot take them down with it. # # text 8192 -> 530 English prose, the ordinary case # repetitive 8192 -> 391 "ABCD" x2048, almost all copies # random 8192 -> 8197 xorshift64 bytes: INCOMPRESSIBLE, so the # output is the input plus a 2-byte length # prefix and 3 bytes of literal tags -- snappy # never expands beyond MaxCompressedLength # already-compressed 530 -> 197 snappy's own output on `text`, fed back # in: literally already-compressed bytes has sn/len-text "$O" "c/text|raw=8192|max=9589|comp=530|rhash=7482204616c373b7|chash=b65dcf35ce514269" has sn/len-repetitive "$O" "c/repetitive|raw=8192|max=9589|comp=391|rhash=36e065d238c2c383|chash=253c96b2779985fd" has sn/len-random "$O" "c/random|raw=8192|max=9589|comp=8197|rhash=c9db26c1a1454acb|chash=0760a0c684f47ab3" has sn/len-already "$O" "c/already-compressed|raw=530|max=650|comp=197|rhash=b65dcf35ce514269|chash=bc4e3c9c9383c35e" # The empty and one-byte inputs, where a length-prefixed format is most # likely to go wrong: an empty input is the single varint byte 0x00, and one # byte is a varint 1 plus a one-byte literal tag plus the byte. has sn/len-empty "$O" "c/empty|raw=0|max=32|comp=1|rhash=14650fb0739d0383|chash=44bd2bd473ccf799" has sn/len-one "$O" "c/one|raw=1|max=33|comp=3|rhash=44bda3d473cdc381|chash=a09aa24f35aa0fde" # rhash of the empty corpus is 14650fb0739d0383, which is FNV-1a/64's offset # basis unchanged -- the digest of nothing. It is worth one line of comment # because it is the one value here a reader can verify without running # anything, and it pins that the probe hashed an empty buffer rather than # skipping the corpus. # The copy-heavy shapes, which exercise the paths a compressor gets wrong # differently from the prose one: `runs` is four 1024-byte single-byte runs # (offset-1 overlapping copies, where a decompressor bug shows first) and # `mixed` splices compressible, incompressible and repetitive blocks so no # single strategy covers the whole input. has sn/len-runs "$O" "c/runs|raw=4096|max=4810|comp=202|rhash=98d9192c71742b83|chash=30a342f120ba4f6f" has sn/len-mixed "$O" "c/mixed|raw=8192|max=9589|comp=2672|rhash=b37ac8e7df751927|chash=f485b277b481ea3c" else bst_notok sn/probe-exec "could not stage an executable copy of the probe under $tmp" fi fi