. "$BST_LIB" # BST-TIMEOUT: 1200 # THREE python3 starts, and they are the whole cost. Every # # other assertion below reads the filesystem or dpkg with # # shell builtins, or greps the output of one of those three. # # # # The three are: a bare `import', a second import whose # # STDERR is the assertion (`import-quiet'), and the probe, # # which takes ONE start for a hundred measurements. # # # # There is no fourth. The distribution declares no # # `[project.scripts]' and the package installs no # # executable, so there is nothing to reach through a # # shebang; `no-bin-script' asserts that absence rather than # # leaving it unstated. /usr/bin/lz4 DOES exist on this # # image and belongs to the `lz4' package (BRINTOS-165), not # # to this one -- `no-bin-script' names the binding's own # # would-be entry point instead, so the two cannot be # # confused. # # # # The work inside the probe is ~700 KB of LZ4 through both # # surfaces at several levels -- milliseconds beside an # # interpreter start on this arm, which is tens of seconds # # before the first line of the program runs. The one item # # that is not negligible is the 300000-byte `big' payload, # # built in a Python loop: 300k iterations of xorshift # # arithmetic, which is seconds rather than milliseconds on # # a wasm interpreter and is the reason the probe's own # # bound below is 900 rather than 300. # # # # 1200 is 400s per start, matching 99-python3-xxhash's and # # 99-python3-pyparsing's budget for their own three and # # roughly six times the ~60s a python3 start has measured # # at on this arm. # # # # A suite-timeout red is a batch-wide cost, so if the three # # starts prove slower than that on target the RIGHT fix is # # this number, not fewer cases. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-lz4 4.3.3 (meta-openembedded/meta-python # recipes-devtools/python/python3-lz4_4.3.3.bb, `inherit pkgconfig pypi # python_setuptools_build_meta ptest', PyPI sdist) on the wasm32-hwjs glibc # image. # # ============================================================================ # WHAT THE PACKAGE DOES. # ============================================================================ # # LZ4 is the codec you reach for when DECOMPRESSION SPEED matters more than # ratio. This distribution is the Python binding for Yann Collet's reference # implementation, and it ships two independent surfaces: # # lz4.block -- a RAW LZ4 block. `compress' prefixes a 4-byte little-endian # uncompressed length by default (`store_size=True'); with # `store_size=False' it does not, and the caller must hand the # length back as `uncompressed_size' at decompression. There is # no header, no checksum and no framing: it is the codec and # nothing else. # # lz4.frame -- the INTEROPERABLE LZ4 frame format, which is what `lz4(1)' # writes and what another implementation can read. It carries a # magic, a block size, optional checksums and an optional content # length, and it comes one-shot (`compress'/`decompress') and # incremental (`LZ4FrameCompressor'/`LZ4FrameDecompressor'). # # A third, `lz4.stream', is experimental upstream and is NOT BUILT here -- # setup.py leaves it out and the recipe drops `tests/stream' with it. # `stream-absent' asserts that, so an image which later gained it is noticed # rather than acquiring an untested surface in silence. # # ============================================================================ # THE MODULE IS COMPILED CODE THROUGHOUT, AND THAT IS THE ONE THING TO READ # BEFORE ANYTHING ELSE HERE. # ============================================================================ # # There is no pure-Python half anywhere in this distribution. `lz4/__init__.py' # is six lines over `from ._version import ...', so even the TOP-LEVEL import # runs a `.so'; `lz4.block' and `lz4.frame' are each a thin `__init__.py' over # their own extension. THREE separate wasm side modules have to load before any # assertion below means anything: # # lz4/_version.cpython-312.so 343 KB # lz4/block/_block.cpython-312.so 676 KB # lz4/frame/_frame.cpython-312.so 913 KB # # THE PLATFORM QUESTION THAT MAKES THAT INTERESTING, which this port shares with # BRINTOS-2035 (python3-xxhash) and does not settle. This image's python3 is ONE # STATIC BINARY: `configure' picks `MODULE_BUILDTYPE=static' from # `host_cpu=wasm32', every stdlib extension is linked into the interpreter, and # `lib-dynload' is EMPTY BY CONSTRUCTION (BRINTOS-1476, and 99-python3.test # asserts that emptiness). Nothing about that construction settles what happens # to a THIRD-PARTY extension, and the two facts point opposite ways: the target # `pyconfig-32.h' defines HAVE_DLOPEN, HAVE_DLFCN_H and HAVE_DYNAMIC_LOADING, so # CPython's `dynload_shlib' path is compiled in -- while the arm is `static-only # ... until the HWJS-5 dlopen substrate is wired in' # (conf/machine/include/tune-wasm32.inc), its dynamic linking exercised so far # at EXEC time through NEEDED (BRINTOS-1638) rather than through a runtime # `dlopen(3)' of a side module at an arbitrary path. # # Each of the three objects IS a real wasm side module: measured at build time, # every one carries a `dylink.0' section, the same shape as the thirteen # converted libraries BRINTOS-1415 and BRINTOS-1442 produced. Whether the # runtime can dlopen one is what `import' below MEASURES. This suite therefore # front-loads the import: `import', `probe-run' and the `ext-*' cases come # before any round trip, so that a red is legible as "the loader could not take # it" rather than as forty broken codec cases. # # THE PREDICTION, LABELLED AS ONE. Nothing in this file was run on target when # it was written -- phase A of a batched Epic builds at recipe scope and never # boots an image -- so the paragraph above is a reading of the source and the # config, NOT A RESULT. If `import' reds with a loader error, that is a PLATFORM # GAP to be attributed, not a defect in this port: the recipe builds clean and # the package installs. The test plan carries the measured outcome; this comment # carries only what was expected and why. # # ============================================================================ # liblz4 IS THE PORTED ONE, AND HOW IT ARRIVES DEPENDS ON THE lz4 RECIPE'S ARM. # ============================================================================ # # The recipe DEPENDS on the target `lz4' recipe and finds it through pkg-config, # so the LZ4 code these three objects run is this image's own lz4 1.9.4 # (BRINTOS-165) and not a copy vendored in the sdist. `libver-string' is what # says so at runtime. # # Where `lz4' is on the converted arm it ships liblz4.so, the extension link's # -llz4 binds it, and each of the three records it in its own needed list # (`{"needed":["liblz4.so.1.9.4."]}'), so the dlopen that `import' # performs loads liblz4.so first. The package then Depends on `lz4'. # # Where `lz4' is kept static there is no liblz4.so and the link pulls in # `liblz4.a': every LZ4_* symbol each object imports is also one it exports, # the wasm convention for a symbol a module defines itself and routes through # the GOT. The package then Depends on `python3-core' alone. # # `libver-string' is the assertion that matters on both arms, because a # fallback to a bundled liblz4 would round-trip every case in this file and # would show up nowhere else. # # ============================================================================ # WHY NO COMPRESSED BYTE STRING IS EVER ASSERTED AGAINST A LITERAL. # ============================================================================ # # LZ4's compressor reads `sizeof(reg_t)' when it counts a match, so a 32-bit # build and a 64-bit build find different match lengths and emit DIFFERENT, # both-correct, compressed streams. Asserting this target's compressed output # against the x86-64 control's would therefore RED ON A CORRECT PORT, and the # same goes for the compressed LENGTHS of any payload where a match exists -- # which is why `run', `rand', `big' and `mix' are asserted through inequalities # (`*-grew', `*-shrank', `level-hi-not-worse') while only the three payloads # whose encoding is structurally forced -- empty, one byte, and 256 distinct # bytes -- carry a literal length. # # WHAT IS ARCH-INDEPENDENT IS THE OTHER DIRECTION: THE FORMAT. So the control's # own compressed blobs are embedded in the probe as hex and the target is asked # to DECOMPRESS them (`ctl-*'). That is what separates a correct implementation # from a consistently wrong one -- the thing a round trip alone cannot do, since # a build that compressed and decompressed with the same wrong constant would # round-trip every payload in this file. # # ============================================================================ # `-ptest' IS INHERITED AND IS NOT INSTALLED, WHICH IS WHY THE SUITE IS HERE # AND NOT UPSTREAM'S. # ============================================================================ # # The recipe inherits `ptest' and its `do_install_ptest' copies the upstream # `tests/' tree, minus `tests/stream'. The `-ptest' sub-package's own RDEPENDS # reach `python3-pytest', `python3-pytest-cov', `python3-coverage', # `python3-pytest-runner', `python3-psutil' and # `python3-unittest-automake-output', none of them ported or ticketed, so it # cannot be installed on this image and the round does not install it. This # build in fact emitted no `-ptest' package at all -- `ptest' is not in # DISTRO_FEATURES here -- and `no-ptest-dir' asserts the resulting absence on # the image, so a distro that later turned the feature on is noticed here. # # This suite drives the public API directly, using the upstream tests as the # reference for WHAT to assert rather than as the mechanism. # # ============================================================================ # MEASURED ON AN x86-64 CONTROL AT THE SAME DISTRIBUTION VERSION -- lz4 4.3.3 # over liblz4 1.9.4, the same liblz4 this image ships -- and every expected # value below is that control's OUTPUT rather than a reading of the # documentation. That distinction earned its keep three times before this file # was finished: `lz4.block.compress' takes `compression', not the # `compression_level' the frame API uses, and the first draft died on the # control with a TypeError; `BLOCKSIZE_MAX64KB' IS the default block size, so # the non-default-block-size case had to move to 256 KiB to be a case at all; # and the compression LEVEL changes nothing on either a long run or # incompressible bytes, so `level-outputs-differ' would have read `no' on a # perfectly correct build until the `mix' payload was built for it. # # The cases that could not be measured there are the packaging ones, whose paths # and control stanza differ on a Debian host by construction. # ============================================================================ P=python3-lz4 # `suite-canary' is emitted FIRST and unconditionally. A suite that dies before # its first result line is indistinguishable from one that was never scheduled, # so this one line is what separates "ran and found nothing" from "did not run". bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/lz4 DI=$SITE/lz4-4.3.3.dist-info PKGLIST=/var/lib/dpkg/info/python3-lz4.list PROBE=$BST_DATA/python3-lz4-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-lz4.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ---------------------------------------------------------------- packaging -- # WHERE THE PACKAGING SAYS THE MODULE LANDS. These run before any import, so a # red here says the .deb is wrong rather than that the module misbehaves. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" check $P/package-init test -s "$PKG/__init__.py" # `version.py' is written by setuptools_scm at build time and is the only place # the distribution version exists as Python; `version-attr' below reads what it # produced. check $P/module-version test -s "$PKG/version.py" # IT IS A PACKAGE OF PACKAGES, not a flat module and not one directory. The two # surfaces are separate sub-packages with separate extensions, and a release # that flattened either would still import and still pass every codec case. check $P/no-flat-module test ! -f "$SITE/lz4.py" check $P/block-dir test -d "$PKG/block" check $P/block-init test -s "$PKG/block/__init__.py" check $P/frame-dir test -d "$PKG/frame" check $P/frame-init test -s "$PKG/frame/__init__.py" # `lz4.stream' IS NOT BUILT -- experimental upstream, left out of setup.py, and # its tests dropped by the recipe. Asserted as an absent DIRECTORY here and as # an ImportError naming the module in `stream-absent' below; the two are # different claims and a port could satisfy one without the other. check $P/no-stream-dir test ! -d "$PKG/stream" # ----------------------------------------------------- the three extensions -- # THE COMPILED EXTENSIONS, AS FILES. These cases separate "the packaging shipped # the .so" from "the loader could take it" -- the second is `import' below, and # keeping them apart is what makes a loader red legible. check $P/ext-version-present test -s "$PKG/_version.cpython-312.so" check $P/ext-block-present test -s "$PKG/block/_block.cpython-312.so" check $P/ext-frame-present test -s "$PKG/frame/_frame.cpython-312.so" # The SOABI tag is the interpreter this image ships (CPython 3.12). A .so built # for another would install cleanly and never be found by the import machinery. # There are EXACTLY THREE, counted across the whole package tree, so a stale # object from another SOABI cannot sit beside them unnoticed. check_eq $P/ext-count "3" \ "$(find "$PKG" -name '*.so' 2>/dev/null | wc -l | tr -d ' ')" check_eq $P/ext-soabi-count "3" \ "$(find "$PKG" -name '*.cpython-312.so' 2>/dev/null | wc -l | tr -d ' ')" # NO EXECUTABLE FROM THIS PACKAGE. The distribution declares no # `[project.scripts]'. /usr/bin/lz4 is on this image and is the `lz4' package's # CLI (BRINTOS-165), so the assertion names the binding's own would-be entry # point rather than the codec's, which would red for the wrong reason. check $P/no-bin-script test ! -e /usr/bin/python3-lz4 # `-ptest' IS NOT INSTALLED -- see the header. check $P/no-ptest-dir test ! -d /usr/lib/python3-lz4/ptest # --------------------------------------------------------------- dist-info --- # The version is read from METADATA rather than from the module, so that a # mismatch between what the packaging claims and what imports is visible AS a # mismatch (`version-attr' below reads the other side). check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 4.3.3" cat "$DI/METADATA" check_out $P/dist-info-name "Name: lz4" cat "$DI/METADATA" check $P/dist-info-wheel test -f "$DI/WHEEL" # NOT PURE PYTHON, stated by the packaging itself. `Root-Is-Purelib: false' is # the wheel's own claim that something in it is architecture-specific -- which # for this distribution is three quarters of it by size. check_out $P/dist-info-purelib "Root-Is-Purelib: false" cat "$DI/WHEEL" # THE WHEEL TAG SAYS `linux_x86_64' AND THAT IS A CROSS-BUILD ARTEFACT, NOT A # WRONG BINARY. `bdist_wheel' names the tag from the BUILD host's platform; # nothing consumes the tag here, because bitbake unpacks the wheel itself and # installs the tree, and `ext-soabi-count' above is what actually pins the # objects to this interpreter. Asserted so the artefact is recorded rather than # rediscovered by the next reader of a `.dist-info' on this image. check_out $P/dist-info-wheel-tag "Tag: cp312-cp312-linux_x86_64" cat "$DI/WHEEL" # `top_level.txt' names the ONE top-level package. The extensions are inside it, # which is why -- unlike python3-xxhash, whose `_xxhash' is a sibling of the # package -- nothing but `lz4' appears here. check_out $P/dist-info-top-level "lz4" cat "$DI/top_level.txt" # ------------------------------------------------------------ the .deb file --- # The installed file list, read once into $W. The copy is what the loop below # redirects from: a stdin redirect the top-level shell performs on the virtiofs # root makes dup2(2) return 513 and ENDS the suite silently (brintos/linux#73), # and $W is on tmpfs. cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # EXACTLY THREE COMPILED OBJECTS SHIP, asserted off the shipped file list rather # than off the filesystem, so the two `ext-count' cases above and this one are # genuinely independent readings. check_eq $P/pkg-three-so "3" \ "$(grep -c '\.so$' "$W/pkgf" 2>/dev/null | tr -d ' ')" # Every payload path the package ships is under site-packages, and is a .py, a # .pyc, one of the three .so, or dist-info metadata. Anything else is a file # this port did not expect to be shipping and wants looking at. _bad="" while IFS= read -r _f; do case "$_f" in ''|/usr/lib/python3.12/site-packages) continue ;; */) continue ;; esac [ -d "$_f" ] && continue case "$_f" in *.py|*.pyc|*.pyi|*.so) ;; # dist-info metadata. Which of these the backend writes is a property of the # BUILD BACKEND rather than of this package, so the whole standard set is # allowed. `pkg-three-so' above carries the exact claim. */METADATA|*/WHEEL|*/RECORD|*/INSTALLER|*/LICENSE|*/entry_points.txt|*/top_level.txt) ;; *) _bad="$_bad $_f" ;; esac done < "$W/pkgf" check_eq $P/pkg-only-expected "" "$_bad" # ------------------------------------------------------------- dpkg status --- sed -n '/^Package: python3-lz4$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # The dependency set follows lz4's arm (see the header): `python3-core' always, # and `lz4' too where the extensions bind liblz4.so. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-python3-core "python3-core" cat "$W/dep" if bst_ps_converted /usr/lib/liblz4.so /usr/lib/liblz4.so.*; then check_out $P/pkg-depends-lz4 "lz4" sh -c "sed 's/python3-core//' '$W/dep'" else bst_skip $P/pkg-depends-lz4 "lz4 ships no liblz4.so here; the extensions carry liblz4.a" fi # THE PORTED lz4 IS ON THE IMAGE on either arm, and `libver-string' below # asserts the version this package's LZ4 code shares with it. If the codec package were absent, that assertion would be comparing # against a number nothing on the image could still produce. check $P/lz4-cli-present test -x /usr/bin/lz4 # ------------------------------------------------------------------ import --- # THE CASE THIS PORT TURNS ON. Everything below it is a statement about modules # that loaded; this is the one that says whether the runtime can dlopen a wasm # side module out of site-packages at all -- and `import lz4' ALONE is enough to # require it, because the top-level `__init__.py' reaches straight into # `_version'. See the header: a red here is a platform gap to attribute, not a # broken recipe. check $P/import python3 -c 'import lz4' # The two surfaces are separate objects and load separately, so a loader that # managed the first and not the others would pass the case above. Asserted in # one start, since the interpreter start is the expensive part and this is a # statement about all three together. check $P/import-surfaces python3 -c 'import lz4.block, lz4.frame' # A package that imports but WARNS on the way in is a different defect from one # that fails, and it is invisible to the cases above because the exit status is # 0 either way. The assertion is that stderr is EMPTY. check_eq $P/import-quiet "" \ "$(python3 -c 'import lz4, lz4.block, lz4.frame' 2>&1 >/dev/null)" # ------------------------------------------------------------------- probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "probe missing at $PROBE" exit 0 fi bst_ok $P/probe-present # THE THIRD START, and the one that carries the suite. Bounded at 900s inside # the 1200s suite budget so that a hung probe is reported as a probe overrun # with the last line it printed, rather than as a suite timeout that says # nothing about where it stopped. bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" # pv : the value the probe emitted for , or empty. `head -n 1' # because a key is emitted once and a duplicate would otherwise concatenate; # `cut -f2-' because a value may legitimately contain `='. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # The probe reached its end. Every case below reads a value the probe printed, # so a probe that stopped early would otherwise turn into a run of empty-string # comparisons that each look like their own finding. check_eq $P/probe-done "yes" "$(pv probe_done)" check_eq $P/probe-count "101" "$(pv emitted_count)" # ...and the five cases below cannot say that for themselves. Their expected # value is the empty string -- the NAMES of the payloads that disagreed -- so an # absent probe reads exactly like agreement and they go GREEN on an image where # no LZ4 byte was ever compressed (BRINTOS-2081; measured 2026-09-01, when the # extension could not be dlopen'd, 80 of the suite's 120 cases went red and # these five did not). Every other probe-derived case reds honestly with an # empty `got', so this is the one place the suite can withdraw invisibly. _probe_done=$(pv probe_done) # check_empty : is empty, WITNESSED. Skip -- not green -- # when the probe did not finish, the shape 99-python3-crc32c uses for its own # post-import cases. An assertion whose expected value is "nothing disagreed" # needs a separate witness that the comparison was made at all. check_empty() { if [ "$_probe_done" = "yes" ]; then check_eq "$1" "" "$(pv "$2")" else bst_skip "$1" "probe did not complete: no $2 measurement was taken" fi } # ---------------------------------------------------------------- identity --- check_eq $P/module-file "$PKG/__init__.py" "$(pv module_file)" check_eq $P/is-package "yes" "$(pv is_package)" check_eq $P/version-attr "4.3.3" "$(pv version_attr)" check_eq $P/version-dunder "4.3.3" "$(pv version_dunder)" # THE PORTED liblz4's VERSION. poky's lz4 recipe is 1.9.4 and the encoded number # is major*10000 + minor*100 + release. Both forms are asserted, because they # come from two different C entry points and a binding could wire one of them to # a constant. check_eq $P/libver-string "1.9.4" "$(pv library_version_string)" check_eq $P/libver-number "10904" "$(pv library_version_number)" # THE THREE EXTENSIONS, AS LOADED MODULES rather than as files. `*-is-so' is # what turns "it imported" into "compiled code is what answered": a pure-Python # fallback would report a `.py' here, and this distribution has none. check_eq $P/ext-version-loaded "yes" "$(pv ext_version_present)" check_eq $P/ext-version-is-so "yes" "$(pv ext_version_is_so)" check_eq $P/ext-block-loaded "yes" "$(pv ext_block_present)" check_eq $P/ext-block-is-so "yes" "$(pv ext_block_is_so)" check_eq $P/ext-frame-loaded "yes" "$(pv ext_frame_present)" check_eq $P/ext-frame-is-so "yes" "$(pv ext_frame_is_so)" # The experimental surface is absent, and the error names the module rather than # something further up -- so an ImportError arriving for an unrelated reason is # not read as this expected one. check_eq $P/stream-absent "yes" "$(pv stream_absent)" check_eq $P/stream-error "ModuleNotFoundError" "$(pv stream_error)" # ------------------------------------------------------------- the corpus --- # The probe builds all seven payloads arithmetically. Their lengths and SHA-256 # digests are asserted against the control's, so that a divergence in the # FIXTURE is reported as a fixture divergence rather than showing up later as a # codec red -- which is what a differing `randbytes' would have looked like. check_eq $P/corpus-empty-len "0" "$(pv corpus_empty_len)" check_eq $P/corpus-one-len "1" "$(pv corpus_one_len)" check_eq $P/corpus-all256-len "256" "$(pv corpus_all256_len)" check_eq $P/corpus-run-len "4096" "$(pv corpus_run_len)" check_eq $P/corpus-rand-len "4096" "$(pv corpus_rand_len)" check_eq $P/corpus-big-len "300000" "$(pv corpus_big_len)" check_eq $P/corpus-mix-len "97313" "$(pv corpus_mix_len)" check_eq $P/corpus-all256-sha \ "40aff2e9d2d8922e47afd4648e6967497158785fbd1da870e7110266bf944880" \ "$(pv corpus_all256_sha)" check_eq $P/corpus-run-sha \ "f600eca824e84a43f0691b267bd620e462c50da165c5b80e17aecb7a924f1fa8" \ "$(pv corpus_run_sha)" check_eq $P/corpus-rand-sha \ "b732710b70c1ffd070517040de42e9106aa241d41aae714a430c80ef61a3a685" \ "$(pv corpus_rand_sha)" check_eq $P/corpus-big-sha \ "377f8246f5109e286d1f6b28deb4654f811370d4bf30225af342d70606a7abff" \ "$(pv corpus_big_sha)" check_eq $P/corpus-mix-sha \ "88306cc9df038be94fe51cafbc41d8ffbede2cfcd10cbdce44c80fd6672dc80d" \ "$(pv corpus_mix_sha)" # -------------------------------------------------------- block round trip --- # EMPTY on agreement, and otherwise the NAMES of the payloads that disagreed -- # the fact a bare `no' would have withheld. Both storage modes are covered, # because `store_size' selects between two separate paths in `_block' and a port # could break either alone. check_empty $P/block-sized-roundtrip block_sized_bad check_empty $P/block-nosize-roundtrip block_nosize_bad # THE THREE COMPRESSED LENGTHS THAT ARE STRUCTURALLY FORCED, and only those -- # see the header on why `run', `rand', `big' and `mix' carry no literal. Empty # input is one token byte; one byte is a token plus the byte; 256 distinct bytes # contain no match on any word size, so they are a 2-byte literal header plus # the payload. check_eq $P/block-nosize-empty-clen "1" "$(pv block_nosize_empty_clen)" check_eq $P/block-nosize-one-clen "2" "$(pv block_nosize_one_clen)" check_eq $P/block-nosize-all256-clen "258" "$(pv block_nosize_all256_clen)" # ...and the SIZE PREFIX is four bytes, asserted as the difference between the # two forms rather than by reading the bytes, so the case survives a payload # whose compressed body differs between arms. check_eq $P/block-prefix-bytes "4" "$(pv block_prefix_bytes)" check_eq $P/block-sized-empty-clen "5" "$(pv block_sized_empty_clen)" # COMPRESSION GROWS INCOMPRESSIBLE INPUT and SHRINKS a long run. A codec that # silently stored a raw copy would round-trip every payload above perfectly and # would be caught only by the second of these. check_eq $P/block-rand-grew "yes" "$(pv block_rand_grew)" check_eq $P/block-run-shrank "yes" "$(pv block_run_shrank)" # -------------------------------------------------------- frame round trip --- check_empty $P/frame-roundtrip frame_bad # THE ONE COMPRESSED-SIDE LITERAL THAT IS ARCH-INDEPENDENT: the frame magic, # 04 22 4D 18 little-endian. It is the format's constant, not the compressor's # choice, and it is what another LZ4 implementation looks for first. check_eq $P/frame-magic "04224d18" "$(pv frame_magic)" # ----------------------------------------------------- frame, incremental --- # `LZ4FrameCompressor' fed in 7000-byte chunks must produce a frame that # decompresses to the same bytes as the one-shot call -- the property every # streaming consumer depends on. The chunk size divides no corpus length, so the # last chunk is short everywhere. check_empty $P/frame-incremental frame_incremental_bad # The chunk COUNT is pinned so that a compressor which silently consumed the # whole payload on the first call is not read as a pass. 300000 / 7000 = 42.8. check_eq $P/frame-incremental-chunks "43" "$(pv frame_incremental_big_chunks)" # ...and the DECOMPRESSOR fed 512 bytes at a time over a one-shot frame. The two # halves are separate objects in `_frame' and a port could break either alone. check_empty $P/frame-decompressor frame_decompressor_bad # A STREAMING WRITER DOES NOT KNOW THE LENGTH IN ADVANCE, so the incremental # frame's `content_size' is 0 while the one-shot frame's is the payload length # (`ctl-frame-content-size' below). That difference is the single most likely # thing for a consumer to misread as a truncated frame, so both sides are # asserted rather than either alone. check_eq $P/incremental-content-size "0" "$(pv incremental_info_content_size)" check_eq $P/incremental-block-size "65536" "$(pv incremental_info_block_size)" # ------------------------------------------------ decompressing the CONTROL --- # The x86-64 control's own output, embedded in the probe as hex and decompressed # here. These are the cases that say this build reads the FORMAT correctly # rather than merely agreeing with itself; see the header for why the reverse # assertion would red on a correct port. check_eq $P/vec-len "72" "$(pv vec_len)" check_eq $P/ctl-block-sized \ "3975b04ce243b39d46db72fedc2f7abad60463426276c3a1057e699ae58155a3" \ "$(pv ctl_block_sized_sha)" check_eq $P/ctl-block-nosize \ "3975b04ce243b39d46db72fedc2f7abad60463426276c3a1057e699ae58155a3" \ "$(pv ctl_block_nosize_sha)" check_eq $P/ctl-frame \ "3975b04ce243b39d46db72fedc2f7abad60463426276c3a1057e699ae58155a3" \ "$(pv ctl_frame_sha)" # The header fields of that same control frame, read on target. These are format # constants and must agree exactly -- a frame reader that got any of them wrong # would still return the right bytes for this small payload, which is why they # are asserted separately from `ctl-frame'. check_eq $P/ctl-frame-content-size "72" "$(pv ctl_frame_content_size)" check_eq $P/ctl-frame-block-size "65536" "$(pv ctl_frame_block_size)" check_eq $P/ctl-frame-block-size-id "4" "$(pv ctl_frame_block_size_id)" check_eq $P/ctl-frame-block-linked "no" "$(pv ctl_frame_block_linked)" check_eq $P/ctl-frame-block-checksum "no" "$(pv ctl_frame_block_checksum)" check_eq $P/ctl-frame-content-checksum "no" "$(pv ctl_frame_content_checksum)" check_eq $P/ctl-frame-skippable "no" "$(pv ctl_frame_skippable)" # ----------------------------------------------- the documented FAILURES --- # ASSERTED, NOT TOLERATED. Each of these is a way a caller's data can be wrong, # and a codec that returned silently instead of raising would hand the caller a # short buffer with no way to know. The EXCEPTION TYPE is asserted, not merely # that something was raised: the two surfaces raise different types, and a build # where either degraded to a bare Exception is a real change to what callers can # catch. check_eq $P/err-frame-truncated "RuntimeError" "$(pv err_frame_truncated)" check_eq $P/err-frame-bad-magic "RuntimeError" "$(pv err_frame_bad_magic)" check_eq $P/err-frame-not-compressed "RuntimeError" "$(pv err_frame_not_compressed)" check_eq $P/err-block-wrong-size "LZ4BlockError" "$(pv err_block_wrong_size)" check_eq $P/err-block-not-compressed "LZ4BlockError" "$(pv err_block_not_compressed)" # A sized block whose 4-byte prefix claims ~2 GB. The binding allocates from that # prefix before it decodes anything, so an unchecked value is an allocation bug # rather than a decode one -- and on a 32-bit arm with a 4 GB address space it is # the case most likely to behave differently from the control. check_eq $P/err-block-huge-prefix "LZ4BlockError" "$(pv err_block_huge_prefix)" # The exception class callers actually name in an `except' clause. check_eq $P/block-error-name "LZ4BlockError" "$(pv block_error_name)" check_eq $P/block-error-is-exception "yes" "$(pv block_error_is_exception)" # --------------------------------------------------- compression levels --- # Both ends of the documented high-compression range, asserted two ways: the # outputs DIFFER (a level argument accepted and dropped would make them equal) # and both decompress to the original (a level that changed the output into # something unreadable is the other failure). `level-hi-not-worse' is an # inequality rather than two lengths, for the arch reason in the header. check_eq $P/level-outputs-differ "yes" "$(pv level_outputs_differ)" check_eq $P/level-lo-roundtrip "yes" "$(pv level_lo_roundtrip)" check_eq $P/level-hi-roundtrip "yes" "$(pv level_hi_roundtrip)" check_eq $P/level-hi-not-worse "yes" "$(pv level_hi_not_worse)" # `mode="fast"' is the third mode and takes the OTHER knob, `acceleration'. A # binding that wired the two arguments to one another would pass every case # above and fail `fast-accel-costs-ratio', which is what says the argument # reached the library rather than being accepted and dropped. check_eq $P/fast-roundtrip "yes" "$(pv fast_roundtrip)" check_eq $P/fast-accel-costs-ratio "yes" "$(pv fast_accel_costs_ratio)" # The frame API's own level and block-size constants are part of what a caller # writes against, and they are SEPARATE exports from the block API's knobs. # `cl-minhc' is 3 while `cl-min' is 0, which is not the symmetry the names # suggest and is exactly the sort of value a suite should pin rather than infer. check_eq $P/cl-min "0" "$(pv cl_min)" check_eq $P/cl-minhc "3" "$(pv cl_minhc)" check_eq $P/cl-max "16" "$(pv cl_max)" # The BLOCKSIZE_* constants are frame-format IDs, not byte counts -- 0 means # "the library's default", and 4..7 are 64 KiB..4 MiB. Pinned because a caller # passing 65536 instead of the ID gets a different frame and no error. check_eq $P/bs-default "0" "$(pv bs_default)" check_eq $P/bs-max64kb "4" "$(pv bs_max64kb)" check_eq $P/bs-max256kb "5" "$(pv bs_max256kb)" check_eq $P/bs-max1mb "6" "$(pv bs_max1mb)" check_eq $P/bs-max4mb "7" "$(pv bs_max4mb)" # A FRAME AT A NON-DEFAULT BLOCK SIZE records that choice in its own header, so # the round trip and the header are asserted together -- a binding that accepted # the argument and wrote the default would pass the round trip alone. 256 KiB, # because 64 KiB IS the default here and a case built on it would have read the # same value either way. check_eq $P/frame-bs256-header "262144" "$(pv frame_bs256_block_size)" check_eq $P/frame-bs256-roundtrip "yes" "$(pv frame_bs256_roundtrip)" # CONTENT CHECKSUM is off by default and on request; it is the frame's own # end-to-end integrity check and the flag has to reach the header, not merely be # accepted by the call. check_eq $P/frame-content-checksum-flag "yes" "$(pv frame_content_checksum_flag)" check_eq $P/frame-content-checksum-roundtrip "yes" "$(pv frame_content_checksum_roundtrip)" rm -rf "$W"