. "$BST_LIB" # BST-TIMEOUT: 600 # Six encodes and eight decodes of a 96x64x16 clip, each a # # fresh wasm module instantiation. Measured well inside 120s # # on the local runner; 600 is what the neighbouring codec # # suites use and gives a bare `brintos-selftest` headroom on # # a loaded host. # libvpx 1.14.1 (meta-oe recipes-multimedia/webm, SRCREV 12f3a2ac) -- the WebM # project's VP8/VP9 reference codec: libvpx.a, the headers under /usr/include/vpx, # vpx.pc, and the two command-line drivers upstream calls UTILS -- vpxenc and # vpxdec. The bbappend also installs upstream's tools/tiny_ssim, which `make # install` drops on every non-MSVS target, as /usr/bin/vpx-tiny-sim's qualified # name vpx-tiny-ssim; it is this suite's independent second oracle. # # THIS SUITE ASSERTS THE ENCODE, NOT ONLY THE DECODE, AND THAT IS THE UNUSUAL # PART. The two codec suites beside it can only pin a decode: 57-libtheora's # container draws a random Ogg serial per run, and 57-libvorbis/57-speexdsp # refuse a stored cross-machine digest outright because their decoders are free # to choose their own float arithmetic. libvpx is a different case on both # counts, and both differences were measured rather than assumed. # # THE CONTAINER. With --disable-webm-io (see the bbappend) vpxenc writes IVF: # 32 bytes of header -- "DKIF", version 0, header length 32, a fourcc, WxH, # rate, scale, frame count, four reserved -- and a 12-byte length+pts before # each frame. No serial number, no muxing-application string, no date. There is # nothing in it to canonicalise, so the encoded file can be `cmp`'d whole. # # THE ARITHMETIC. VP8 and VP9 are integer codecs: the inverse transform, # dequantisation, motion compensation and the loop filter are all specified in # integers, and libvpx's own SIMD-consistency discipline requires every # hand-written kernel to be bit-exact with its C reference. The target build is # configure's `generic-gnu` -- pure C, VPX_ARCH_X86/ARM all zero, no runtime CPU # detection -- which is that reference, not an approximation of it. # # HOW THE FIVE PAYLOADS WERE MADE AND WHAT WAS MEASURED BEFORE THEY WERE TRUSTED. # src/libvpx_mkfixture.py states the whole recipe. The short version: the same # SRCREV built natively on x86-64 with the SAME configure line the bbappend gives # the target, then encoded and decoded with the pinned $ENCOPTS below. Two # controls were run on x86-64 before the references were committed: # # * REPEAT. Re-encoding and re-decoding on the same build gives byte-identical # output, for both codecs. (Asserted again on target below, as # vpx/*-encode-deterministic -- what makes a stored reference an oracle at # all is asserted, not assumed.) # * OPTIMISATION LEVEL. The same source at gcc -O3 (the default) and at gcc -O0 # (`--disable-optimizations`) produces byte-identical encodes AND decodes, for # both codecs. That is the control that matters here, because the one thing # that could legitimately differ between two pure-C builds of an integer codec # is a float path the compiler is allowed to reassociate -- and there is none # that reaches the bitstream. # # The control NOT run, stated so nobody assumes it was: an x86-64 build with # the hand-written assembly ON. It needs yasm or nasm and the build host has # neither. It would test libvpx's SIMD-consistency discipline rather than this # port, and the port's claim does not rest on it -- generic-gnu is the C # reference the asm is tested against, not the other way round. # # READING A RED. The reds here separate cleanly and the separation is the point: # # * vpx/vp8-encode-exact or vpx/vp9-encode-exact red, with the matching # *-decode-exact GREEN -- the ENCODER decided differently. Every encoder # input is pinned below, so this is a real divergence in mode decision, rate # control or entropy coding, not a settings drift. # * a *-decode-exact red -- the DECODER produced different pixels. For a # normative integer codec there is no legitimate source of drift, so this is # a miscompile or a bad kernel, never something to re-reference away. The # suite then pays for a byte-level walk and prints max deviation and the # differing-byte count so the design doc can say WHAT went wrong. # * vpx/*-md5-agrees red with *-decode-exact green (or the reverse) -- the two # read the same decode through different code (vpxdec's internal digest # against the file `cmp`), so disagreement means one of the two paths is # broken, not the codec. # * every decode case green and vpx/ssim-* red -- vpx-tiny-ssim is the only # binary in this package that links libm; a red confined to it is a float or # libm fault, not a codec one. # * the presence cases red with everything else green -- the bbappend's # tiny_ssim install or the static-only packaging stopped taking. # * vpx/webm-refused red -- webm_io got re-enabled, which silently changes what # vpxenc writes by default and would invalidate every encode reference here. # # vpx/trunc-* and vpx/junk-* pin UPSTREAM behaviour, some of it arguably wrong # (a truncated stream exits 0). A red there is a behaviour CHANGE, not # necessarily a regression. require vpxdec vpx/vpxdec-present tmp=${TMPDIR:-/tmp}/bst-vpx.$$ mkdir -p "$tmp" 2>/dev/null size() { wc -c < "$1" 2>/dev/null | tr -d ' '; } md5of() { md5sum "$1" 2>/dev/null | cut -d' ' -f1; } have_md5=0; command -v md5sum >/dev/null 2>&1 && have_md5=1 SRC=$BST_DATA/libvpx-clip.src.y4m IVF8=$BST_DATA/libvpx-clip.vp8.ivf IVF9=$BST_DATA/libvpx-clip.vp9.ivf REF8=$BST_DATA/libvpx-clip.vp8.ref.y4m REF9=$BST_DATA/libvpx-clip.vp9.ref.y4m # THE PINNED ENCODER LINE. Every word is here because it removes a degree of # freedom, and a reference encode is only an oracle if none is left: # --ivf container, and the reason this suite can cmp the encode # --passes=1 a two-pass encode writes a stats file and re-runs; one pass # is one deterministic sweep # --threads=1 the codec partitions work per thread, so the THREAD COUNT # changes the bitstream even though the arithmetic does not. # The image is built --disable-multithread (see the bbappend), # so this is belt and braces -- and it is what keeps the # reference valid against a threaded build. # --good --cpu-used=0 # the deadline and the speed/quality step. --cpu-used is the # single largest lever on mode decision; 0 is the slowest and # most thorough, so the encoder actually visits the paths the # texture was built to reach. # --lag-in-frames=0 no lookahead buffer, so frame N depends only on 0..N # --end-usage=q --cq-level=32 --min-q=0 --max-q=63 # constant-quality rate control. The bitrate-targeting modes # adapt to a moving average and are the classic way an encode # stops being reproducible from its inputs alone. # --kf-min-dist=0 --kf-max-dist=9999 # exactly one keyframe, the first; no automatic re-keying # --auto-alt-ref=0 no synthesised alt-ref frame (VP9 would otherwise insert one) # --static-thresh=0 no "skip this block, it barely moved" shortcut # --quiet vpxenc's progress line is ~1 kB of carriage-return redraw per # frame, and this console is read back over a serial capture ENCOPTS="--ivf --passes=1 --threads=1 --good --cpu-used=0 --lag-in-frames=0 \ --end-usage=q --cq-level=32 --kf-min-dist=0 --kf-max-dist=9999 \ --auto-alt-ref=0 --static-thresh=0 --min-q=0 --max-q=63 --quiet" # --------------------------------------------------------------- what shipped --- check vpx/vpxenc-present test -x /usr/bin/vpxenc check vpx/tiny-ssim-present test -x /usr/bin/vpx-tiny-ssim check vpx/lib-static test -f /usr/lib/libvpx.a check vpx/pc-vpx test -f /usr/lib/pkgconfig/vpx.pc check vpx/header-codec test -f /usr/include/vpx/vpx_codec.h check vpx/header-decoder test -f /usr/include/vpx/vpx_decoder.h check vpx/header-encoder test -f /usr/include/vpx/vpx_encoder.h check vpx/header-image test -f /usr/include/vpx/vpx_image.h check vpx/header-vp8 test -f /usr/include/vpx/vp8.h check vpx/header-vp8cx test -f /usr/include/vpx/vp8cx.h check vpx/header-vp8dx test -f /usr/include/vpx/vp8dx.h # Shape, not absence: this arm converts to shared libraries per recipe # (BRINTOS-1740), so a .so here is a conversion, not a policy breach. bst_lib_shape vpx/static-only "/usr/lib/libvpx*.so*" # Both codecs, both directions, from the binaries' own registration tables. check_out vpx/dec-vp8-registered "WebM Project VP8 Decoder v1.14.1" sh -c 'vpxdec --help 2>&1' check_out vpx/dec-vp9-registered "WebM Project VP9 Decoder v1.14.1" sh -c 'vpxdec --help 2>&1' check_out vpx/enc-vp8-registered "WebM Project VP8 Encoder v1.14.1" sh -c 'vpxenc --help 2>&1' check_out vpx/enc-vp9-registered "WebM Project VP9 Encoder v1.14.1" sh -c 'vpxenc --help 2>&1' # webm_io is OFF, and this is the only place that is observable on target: the # --webm flag is registered unconditionally and refuses at runtime. It matters # beyond tidiness -- with webm_io on, vpxenc's DEFAULT container changes and # every encode reference in this file would compare against the wrong bytes. vpxenc --codec=vp9 --webm -o "$tmp/w.webm" "$SRC" >/dev/null 2>"$tmp/webm.err"; rc=$? check_eq vpx/webm-refused-rc 1 "$rc" case "$(cat "$tmp/webm.err" 2>/dev/null)" in *"--webm specified but webm is disabled"*) bst_ok vpx/webm-refused ;; *) bst_notok vpx/webm-refused "$(_bst_1line "$(cat "$tmp/webm.err" 2>/dev/null)")" ;; esac # ------------------------------------------------------------------ payloads --- # Pinned BEFORE anything is encoded or decoded. Without this a later mismatch has # two indistinguishable causes -- the codec changed, or the payload did -- and # the wrong one gets reported. pin() { # pin if [ "$have_md5" -eq 1 ]; then check_eq "$1" "$2" "$(md5of "$3")" else bst_skip "$1" "no md5sum on this image"; fi } pin vpx/fixture-src-digest 0a38b7b645e22beea07eb29090f5e287 "$SRC" pin vpx/fixture-vp8-digest 08bfe552bfa31bf667786544952961e0 "$IVF8" pin vpx/fixture-vp9-digest 27f608bb8ba9d260699b60a7c4b1d92d "$IVF9" pin vpx/reference-vp8-digest c3c7c249857d73a93728a53a98104c36 "$REF8" pin vpx/reference-vp9-digest 65494c77982ff1078d683d546d670f51 "$REF9" for f in "$SRC" "$IVF8" "$IVF9" "$REF8" "$REF9"; do if [ ! -f "$f" ]; then bst_notok vpx/payloads "missing $f -- remaining cases not run" rm -rf "$tmp"; exit 0 fi done check_eq vpx/fixture-src-size 147593 "$(size "$SRC")" check_eq vpx/fixture-vp8-size 20471 "$(size "$IVF8")" check_eq vpx/fixture-vp9-size 41165 "$(size "$IVF9")" # The IVF header, byte by byte, before anything is compared against it. 32 bytes: # "DKIF", version 0, header length 32, fourcc, W=96, H=64, rate=30, scale=1, # frame count=16, four reserved. Asserted because a whole-file cmp that failed # would not say WHICH of the container and the coded data went wrong. ivfhdr() { od -An -tx1 -N32 -v "$1" 2>/dev/null | tr -s ' \n' ' ' | sed 's/^ //;s/ $//'; } check_eq vpx/vp8-ivf-header \ "44 4b 49 46 00 00 20 00 56 50 38 30 60 00 40 00 1e 00 00 00 01 00 00 00 10 00 00 00 00 00 00 00" \ "$(ivfhdr "$IVF8")" check_eq vpx/vp9-ivf-header \ "44 4b 49 46 00 00 20 00 56 50 39 30 60 00 40 00 1e 00 00 00 01 00 00 00 10 00 00 00 00 00 00 00" \ "$(ivfhdr "$IVF9")" # ------------------------------------------------------------- the encode ----- # The ticket's criterion, half one. Anything that differs here differs in the # encoder's decisions, because every input to those decisions is pinned above. encode() { # encode # shellcheck disable=SC2086 vpxenc --codec="$1" $ENCOPTS -o "$2" "$SRC" 2>"$tmp/enc.$1.err" } agree() { # agree if cmp -s "$2" "$3"; then bst_ok "$1-exact"; bst_ok "$1-max-deviation"; bst_ok "$1-differing-bytes" echo "==LIBVPX-AGREE $4 byte-identical to reference==" return 0 fi bst_notok "$1-exact" "differs from $(basename "$3")" # Only now pay for the byte-level walk, and bound the damage so the design doc # can say WHAT went wrong rather than only that something did. od -An -v -tu1 -w1 "$2" > "$tmp/a.txt" 2>/dev/null od -An -v -tu1 -w1 "$3" > "$tmp/b.txt" 2>/dev/null set -- $(awk 'NR==FNR{a[FNR]=$1;next} {d=a[FNR]-$1; if(d<0)d=-d; if(d>m)m=d; if(d)n++} END{print m+0, n+0, FNR}' "$tmp/a.txt" "$tmp/b.txt") echo "==LIBVPX-AGREE $4 max=${1:-?} ndiff=${2:-?} of ${3:-?}==" return 1 } # The deviation bounds are set for a DECODED PIXEL PLANE and are meaningless on a # compressed bitstream, so the encode cases report the walk without a band: a # single differing bit in an arithmetic-coded stream desynchronises everything # after it, and "max deviation 255" is the expected reading, not extra # information. What the walk is for there is the OFFSET of the first difference, # which cmp prints. encdiff() { # encdiff cmp "$2" "$3" > "$tmp/cmpout" 2>&1 && return 0 bst_notok "$1-first-difference" "$(_bst_1line "$(cat "$tmp/cmpout")")" return 1 } for c in vp8 vp9; do case $c in vp8) REFIVF=$IVF8 ;; *) REFIVF=$IVF9 ;; esac encode "$c" "$tmp/$c.ivf"; rc=$? check_eq "vpx/$c-encode-rc" 0 "$rc" if [ "$rc" -eq 0 ]; then if ! agree "vpx/$c-encode" "$tmp/$c.ivf" "$REFIVF" "$c-encode"; then encdiff "vpx/$c-encode" "$tmp/$c.ivf" "$REFIVF" else bst_ok "vpx/$c-encode-first-difference" fi # What makes a stored reference an oracle at all: asserted, not assumed. encode "$c" "$tmp/$c.again.ivf" 2>/dev/null check "vpx/$c-encode-deterministic" cmp -s "$tmp/$c.ivf" "$tmp/$c.again.ivf" else bst_notok "vpx/$c-encode-exact" "encoder exited $rc -- $(_bst_1line "$(cat "$tmp/enc.$c.err" 2>/dev/null)")" bst_notok "vpx/$c-encode-deterministic" "encoder exited $rc" fi done # ------------------------------------------------------------- the decode ----- # The ticket's criterion, half two -- and the stronger of the two claims, because # VP8 and VP9 decode is NORMATIVE: a conforming decoder has no freedom here at # all, so a difference is a fault by definition rather than a judgement call. for c in vp8 vp9; do case $c in vp8) IVF=$IVF8; REF=$REF8; MD5=c3c7c249857d73a93728a53a98104c36 ;; *) IVF=$IVF9; REF=$REF9; MD5=65494c77982ff1078d683d546d670f51 ;; esac vpxdec --codec="$c" -o "$tmp/$c.y4m" "$IVF" >/dev/null 2>"$tmp/dec.$c.err"; rc=$? check_eq "vpx/$c-decode-rc" 0 "$rc" check_eq "vpx/$c-decode-size" 147588 "$(size "$tmp/$c.y4m")" # 147588 is not a magic number: 36 bytes of Y4M header, then 16 frames of # ("FRAME\n" = 6) + (96*64 luma + 2*48*32 chroma = 9216) = 9222. A decoder that # dropped or duplicated a frame lands on a different one, and that is the # cheapest possible detector for it. check_eq "vpx/$c-decode-header" "YUV4MPEG2 W96 H64 F15:1 Ip C420jpeg" \ "$(head -1 "$tmp/$c.y4m")" # F15:1 where the SOURCE clip says F30:1, and that asymmetry is upstream's, not # this port's: IVF stores a timebase and vpxdec reconstructs a frame rate from # it without the factor of two vpxenc applied. Pinned rather than corrected -- # an unpinned upstream wart is indistinguishable from a port defect the first # time someone meets it. # Unlike the encode above, the deviation bands DO mean something here: these # are pixel planes, so "max |delta| 1 over 40 bytes" and "max 255 over half the # frame" are different findings and agree() prints which. agree "vpx/$c-decode" "$tmp/$c.y4m" "$REF" "$c-decode" # The independent second path: vpxdec's own MD5 of the decoded frames, computed # inside the decoder rather than over the written file. A fault that defeated # both would have to corrupt the decode and the file write the same way. d=$(vpxdec --codec="$c" --md5 "$IVF" 2>/dev/null | head -1 | cut -d' ' -f1) check_eq "vpx/$c-md5-agrees" "$MD5" "$d" # Round trip: the stream this target just ENCODED, decoded back. Green here # with -encode-exact red would mean the encoder is self-consistent but has # diverged from x86-64; green with both -exact cases green is the whole port. if [ -f "$tmp/$c.ivf" ]; then vpxdec --codec="$c" -o "$tmp/$c.rt.y4m" "$tmp/$c.ivf" >/dev/null 2>&1 check "vpx/$c-round-trip" cmp -s "$tmp/$c.rt.y4m" "$REF" else bst_notok "vpx/$c-round-trip" "no encoded stream from this target to decode" fi # Raw I420 out: same pixels, no container. 16 * 9216 = 147456. vpxdec --codec="$c" --i420 -o "$tmp/$c.i420" "$IVF" >/dev/null 2>&1 check_eq "vpx/$c-i420-size" 147456 "$(size "$tmp/$c.i420")" done # ------------------------------------------------------------- vpx-tiny-ssim --- # The second oracle, and the only binary in this package that links libm. It # re-reads both files through its own y4m parser and its own arithmetic; on # identical input it prints the literal MAX_PSNR cap (tools/tiny_ssim.c:78, # `#define MAX_PSNR 100`), so the self-comparison needs no float tolerance. if command -v vpx-tiny-ssim >/dev/null 2>&1; then s=$(vpx-tiny-ssim "$REF9" "$REF9" 2>&1) check_eq vpx/ssim-self-ssim "SSIM: 1.000000" "$(printf '%s\n' "$s" | grep '^SSIM:')" check_eq vpx/ssim-self-psnr "AvgPSNR: 100.000000" "$(printf '%s\n' "$s" | grep '^AvgPSNR:')" # And the real measurement: the source clip against each decode. Float, so # asserted as a band -- wide enough to absorb a last-ulp libm disagreement and # far too narrow to admit a decode that went wrong. Measured on x86-64: # VP8 24.39 dB / SSIM 0.9446, VP9 36.80 dB / SSIM 0.9960. The 12 dB gap is not # an artefact: --cq-level is a quantizer INDEX and the two codecs number their # quantizers differently, so the same 32 is a far coarser step for VP8. It is # left alone deliberately -- one setting for both codecs is one fewer thing # that can drift, and a coarse VP8 encode exercises the dequantiser harder. for c in vp8 vp9; do case $c in vp8) REF=$REF8; LO=23; HI=26 ;; *) REF=$REF9; LO=35; HI=38 ;; esac p=$(vpx-tiny-ssim "$SRC" "$REF" 2>&1 | awk '/^AvgPSNR:/{print int($2)}') if [ -n "$p" ] && [ "$p" -ge "$LO" ] 2>/dev/null && [ "$p" -le "$HI" ] 2>/dev/null then bst_ok "vpx/ssim-$c-psnr" else bst_notok "vpx/ssim-$c-psnr" "AvgPSNR = ${p:-} dB (want $LO..$HI)"; fi done # The two behaviours 0001-tiny_ssim-require-two-files-... restores, pinned so a # recipe bump that drops the patch reddens a named case instead of silently # bringing the crash back. Both were measured on x86-64 BEFORE the patch: one # file argument segfaulted (rc 139), and an unreadable input exited 0 while # printing its diagnostic. check_rc vpx/ssim-missing-rc 1 vpx-tiny-ssim "$tmp/no-such.y4m" "$REF9" check_rc vpx/ssim-one-arg-rc 1 vpx-tiny-ssim "$REF9" else bst_notok vpx/tiny-ssim-runnable "vpx-tiny-ssim is not on PATH" fi # ------------------------------------------------------- malformed input ------ # Each of these pins CURRENT upstream behaviour. The truncation case is wrong in # upstream's own terms -- a stream that ends mid-frame exits 0 -- and is pinned # anyway, because an unpinned wrong is indistinguishable from a port defect the # first time someone meets it. vpxdec --codec=vp9 -o "$tmp/miss.y4m" "$tmp/no-such.ivf" >/dev/null 2>"$tmp/miss.err" check_eq vpx/missing-rc 1 "$?" case "$(cat "$tmp/miss.err" 2>/dev/null)" in *"Failed to open input file"*) bst_ok vpx/missing-diagnostic ;; *) bst_notok vpx/missing-diagnostic "$(_bst_1line "$(cat "$tmp/miss.err" 2>/dev/null)")" ;; esac # Not a container libvpx knows: rejected before any decoding is attempted. printf 'NOT-AN-IVF-FILE-AT-ALL-000000000000' > "$tmp/junk.ivf" vpxdec --codec=vp9 -o "$tmp/junk.y4m" "$tmp/junk.ivf" >/dev/null 2>"$tmp/junk.err" check_eq vpx/junk-rc 1 "$?" case "$(cat "$tmp/junk.err" 2>/dev/null)" in *"Unrecognized input file type"*) bst_ok vpx/junk-diagnostic ;; *) bst_notok vpx/junk-diagnostic "$(_bst_1line "$(cat "$tmp/junk.err" 2>/dev/null)")" ;; esac # A VALID prefix of a valid stream: the header parses, some frames decode, and # the last one is short. Upstream warns and exits 0, and the frames it did decode # are the reference's own first frames -- which is the assertion worth making, # because it says the decoder degraded rather than corrupted. head -c 13000 "$IVF9" > "$tmp/trunc.ivf" 2>/dev/null vpxdec --codec=vp9 -o "$tmp/trunc.y4m" "$tmp/trunc.ivf" >/dev/null 2>"$tmp/trunc.err" check_eq vpx/trunc-rc 0 "$?" case "$(cat "$tmp/trunc.err" 2>/dev/null)" in *"Failed to read full frame"*) bst_ok vpx/trunc-diagnostic ;; *) bst_notok vpx/trunc-diagnostic "$(_bst_1line "$(cat "$tmp/trunc.err" 2>/dev/null)")" ;; esac check_eq vpx/trunc-size 36924 "$(size "$tmp/trunc.y4m")" if [ -s "$tmp/trunc.y4m" ] && dd if="$REF9" bs=1 count="$(size "$tmp/trunc.y4m")" 2>/dev/null \ | cmp -s - "$tmp/trunc.y4m"; then bst_ok vpx/trunc-prefix-matches else bst_notok vpx/trunc-prefix-matches "the frames that did decode are not the reference's own"; fi # An encoder handed a file that is not there. vpxenc --codec=vp9 --ivf -o "$tmp/o.ivf" "$tmp/no-such.y4m" >/dev/null 2>"$tmp/encmiss.err" check_eq vpx/enc-missing-rc 1 "$?" rm -rf "$tmp"