. "$BST_LIB" # BST-TIMEOUT: 1800 # ONE python3 start, and that is most of the cost. Every # # assertion below either reads the filesystem/dpkg with # # shell builtins or greps the single probe run's output. The # # probe itself is dominated by the five 32-bit-form vectors, # # which pack and unpack 65536 elements or bytes each; on an # # x86-64 control that is 0.08s in total, and this bound is # # the 1200 the other python3 suites use with a margin for a # # wasm32 interpreter being two orders of magnitude slower at # # a per-element pure-Python loop. # # NOT MEASURED ON TARGET YET: this port's first on-target run # # is the batch verify, and the measured elapsed replaces this # # note in the test plan rather than in this comment, which # # would cost an image build to keep the shipped suite # # byte-identical. # # python3-u-msgpack-python 2.8.0 (meta-openembedded meta-python # recipes-devtools/python/python3-u-msgpack-python_2.8.0.bb, `inherit pypi # setuptools3 ptest') on the wasm32-hwjs glibc image -- A COMPLETE MessagePack # SERIALISER AND DESERIALISER IN ONE PURE-PYTHON MODULE. `packb' turns a Python # object into MessagePack bytes and `unpackb' reads them back; `pack'/`unpack' # are the same codecs over a file-like object. # # ============================================================================ # WHY THE EXPECTATIONS HERE ARE SPEC LITERALS RATHER THAN CONTROL OUTPUT. # ============================================================================ # # Almost every other python3 suite in this corpus pins a value BECAUSE A CONTROL # HOST PRODUCED IT. This one does not have to. MessagePack is a wire format with # a published specification, and the specification fixes the bytes exactly: one # format byte per form, big-endian lengths, and a stated boundary at which each # form gives way to the next. So `enc-uint16-min' asserting `cd0100' is asserting # what msgpack/msgpack spec.md says a uint16 of 256 IS -- not what x86-64 # happened to emit. # # That is what makes this member the round's CONTROL. It is pure Python with no # accelerator, no optional C half and no build step, so there is nothing between # the source and the bytes that could explain a difference away. Anything wrong # here is wrong in the interpreter or in the arithmetic underneath it, and it # would be wrong in the same way for every other codec in the round. # # ============================================================================ # THE VECTORS SIT ON THE ENCODING BOUNDARIES, BOTH SIDES OF EACH. # ============================================================================ # # A serialiser that wrote `<' where the specification says `<=' produces correct # bytes for almost every value and the wrong form for exactly one. So each form # is driven at its first value and at its last -- 127/128 for fixint->uint8, # 65535/65536 for uint16->uint32, 31/32 for fixstr->str8, 255/256 for # str8->str16, 15/16 for fixarray->array16 -- through every form the format has: # the fixint and uint8/16/32/64 and int8/16/32/64 integers, float32 and float64, # fixstr through str8/16/32, bin8/16/32, fixarray through array16/32, fixmap # through map16/32, the nil and boolean singletons, and fixext1/2/4/8/16 through # ext8/16/32. # # EACH VECTOR IS ASSERTED TWICE, AND THE TWO HALVES ANSWER DIFFERENT QUESTIONS. # `enc-*' says the ENCODER agrees with the specification; `rt-*' says the DECODER # reads those bytes back to the value that went in. A library whose encoder and # decoder were wrong in the same direction round-trips perfectly and reds on # `enc-*', which is the whole reason the literal bytes are here at all. # # THE LONG VECTORS ARE ASSERTED BY PREFIX, LENGTH AND DIGEST. str8-max and its # five relatives carry a 255- or 256-byte constant payload, and the five 32-bit # forms carry 65536; a 500-character hex literal would say nothing the format # byte and the big-endian length do not already say, and a 65 kB one could not be # read at all. The prefix is the part the specification fixes. # # ============================================================================ # WHY THIS PORT IS POSSIBLE AT ALL, AND WHAT THE .so CASES ARE REALLY ASSERTING. # ============================================================================ # # This image's python3 is ONE STATIC BINARY with an EMPTY lib-dynload and no # dynamic loader (BRINTOS-1476; 99-python3 asserts both directly). A module # carrying a compiled extension therefore could not be imported here -- not # "would be slow", could not be imported. u-msgpack-python declares no # ext_modules and ships no `.so': `packages=['umsgpack']' and nothing else, which # is the entire reason it ports. # # So `pkg-no-so' is not boilerplate. It is a REGRESSION TRIPWIRE on a version # bump: this package grows no extension today, and the day a release adds a # compiled accelerator this case goes red at INSTALL time rather than at `import # umsgpack' on somebody else's ticket three rounds later. That is the shape # python3-cbor2 (BRINTOS-2039) had to decline an extension for in this same round. # # ============================================================================ # IT IS A PACKAGE, NOT A MODULE -- A DIRECTORY HOLDING ONE FILE. # ============================================================================ # # setup.py declares `packages=['umsgpack']', so what lands in site-packages is # the directory `umsgpack/' containing `__init__.py' and NO top-level # `umsgpack.py'. `package-dir', `module-file' and `no-flat-module' assert all # three: a future release that flattened to a single module would still import, # and every behavioural case below would still pass, while the packaging this # image ships had silently changed shape. # # THE TYPE STUB SHIPS AND THE PEP 561 MARKER DOES NOT, AND THAT IS UPSTREAM'S. # setup.py says `package_data={'umsgpack': ['*.pyi', 'py.typed']}' but the 2.8.0 # sdist carries no `py.typed' file, so the glob matches `__init__.pyi' alone. A # type checker therefore IGNORES the stub, because without the marker the package # is not declared typed. `stub-file' and `no-py-typed' pin both halves; neither is # a defect of this port, and a release that added the marker should be seen here # rather than inferred. # # ============================================================================ # NO BBAPPEND AND NO PATCH OF OURS -- MEASURED, NOT ASSUMED. # ============================================================================ # # The recipe is taken from meta-python exactly as written. There is no configure # and no compile step -- setuptools3 over a PyPI sdist whose entire payload is one # 43 617-byte Python file -- so there is nothing on this arm for an override to # reach. It built at recipe scope unchanged (1603 of 1603 tasks, all succeeded, # zero ERROR lines) and `umsgpack/__init__.py' and `umsgpack/__init__.pyi' # unpacked from the built .deb are BYTE-IDENTICAL to the sdist the SRC_URI sha256 # pins. That is the measurement behind "no patch" rather than a reading of the # recipe. # # ============================================================================ # THE RECIPE INHERITS ptest AND THIS IMAGE PRODUCES NO -ptest PACKAGE. # ============================================================================ # # The recipe does `inherit ptest' and RDEPENDS its `-ptest' split on # python3-pytest and python3-unittest-automake-output, neither of which is on this # image. That is not a problem to work around: the absent `ptest' feature in # brintos.conf leaves ptest off, and the recipe-scope build produced exactly three # .debs -- the runtime package, `-dev' and `-dbg' -- and NO `-ptest'. So the # unsatisfiable dependency is never resolved because the package that carries it # is never built. `no-ptest-package' states that from the target's own side, so an # image that later turned the feature on and dragged pytest in reds here. # # The consequence is that upstream's `test_umsgpack.py' does not ship, and this # suite is what the port closes on. It was written against that file as the # reference for WHAT to assert -- the boundary vectors, the option behaviours and # the documented exceptions are the same ground -- and asserts them through the # module's public API rather than through pytest. # # ============================================================================ # THE ONE PLATFORM PROPERTY THIS PORT DEPENDS ON. # ============================================================================ # # `float-precision'. The module's initialiser reads `sys.float_info.mant_dig' and # packs a Python float as IEEE-754 float64 when it is 53 and as float32 # otherwise. An interpreter whose double were not binary64 would therefore change # THE WIRE FORMAT of every float this image writes, silently and correctly by the # library's own logic. That is why the auto-detected value is asserted as a case # and not merely emitted. # # ============================================================================ # WHY THE RECIPE RDEPENDS ON python3-datetime, DEMONSTRATED RATHER THAN READ. # ============================================================================ # # `import datetime' is at MODULE scope in umsgpack/__init__.py, and the module # packs a `datetime.datetime' as the reserved ext type -1 -- the MessagePack # timestamp extension. poky's python3-manifest.json puts datetime.py in the # `datetime' split, not in `core', so on an image without it the package would # fail at IMPORT, not at the first timestamp. `pkg-depends-datetime' asserts the # declared dependency, `enc-timestamp32'/`enc-timestamp64' assert the behaviour # that needs it, and `closure-datetime' asserts that the import actually happened. # # ============================================================================ # WHAT IS NOT COVERED, ON PURPOSE. # ============================================================================ # # - THE `huge string' / `huge binary string' / `huge array' / `huge map' # GUARDS. Each needs an object of 2**32 elements or bytes -- 4 GiB -- which # this image cannot allocate, so they are unreachable here and are not # asserted. The INTEGER guards are reachable (2**64 is a small object) and are # driven, as `err-huge-uint' and `err-huge-int'. # - `compatibility' MODE, the pre-2013 raw-string encoding. It is a module-wide # global whose only correct use is talking to an implementation older than the # str/bin split; setting it would change the meaning of every string vector # above, so the suite asserts it is OFF and leaves it there. # - `ext_serializable', the class-registration decorator. It mutates # module-global registries that `_pack3' consults BEFORE every other type # test, so a registration made here would change the encoding of the class it # names for the rest of the interpreter's life. `ext_handlers' is the # per-call form of the same feature and is driven instead; the decorator's # presence on the API surface is asserted, its side effect is not invoked. # - upstream's `test_umsgpack.py' -- see above; the recipe's ptest split is not # built on this image. # # THIS PORT CLOSES ON THIS SUITE: non-interactive, no manual browser pass, no # follow-on ticket under BRINTOS-780. P=python3-u-msgpack-python # THE CANARY, which asserts nothing about python3-u-msgpack-python. # # A suite's FIRST BST result line is lost between the guest's printf and the # host's read -- per-suite, deterministic, and NOT a defect in the host tool # (BRINTOS-1149 established that and filed it as linux-in-the-browser#177). The # harness computes its pass count as total-minus-failures, so THE DROPPED LINE # IS COUNTED AS A PASS. A nothing-case therefore stands in that position and the # eaten line costs nothing. It sits ahead of the `require' guards deliberately, # because a `require' that skips would emit its skip as the first line and be # eaten too. # # Remove this case when linux-in-the-browser#177 is fixed. bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/umsgpack MOD=$PKG/__init__.py STUB=$PKG/__init__.pyi DI=$SITE/u_msgpack_python-2.8.0.dist-info PKGLIST=/var/lib/dpkg/info/python3-u-msgpack-python.list PROBE=$BST_DATA/python3-u-msgpack-python-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-umsgpack.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------- the payload --- # Where the package has to land. `import umsgpack' further down is the functional # statement; these are the ones that say WHICH copy answered, so a module that # imported out of some other path is distinguishable from one that imported out # of the package this image installed. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" check $P/module-file test -f "$MOD" # ...AND NO FLAT MODULE. setup.py declares `packages=['umsgpack']', so a # top-level `umsgpack.py' here would mean the upstream shape changed under a # recipe that still says otherwise -- invisible to every behavioural case below. check_eq $P/no-flat-module "absent" \ "$(if [ -f "$SITE/umsgpack.py" ]; then echo present; else echo absent; fi)" # THE TYPE STUB SHIPS AND THE PEP 561 MARKER DOES NOT. See the header: setup.py's # package_data names both, the 2.8.0 sdist carries only the stub, and a checker # therefore ignores it. Upstream's, not this port's -- pinned so a release that # added the marker is seen rather than inferred. check $P/stub-file test -f "$STUB" check_eq $P/no-py-typed "absent" \ "$(if [ -f "$PKG/py.typed" ]; then echo present; else echo absent; fi)" # NO COMPILED EXTENSION ANYWHERE THE PACKAGE PUT ANYTHING. See the header: on # this image a .so could not be loaded at all, so this is the case that fails a # version bump which grew one -- loudly, here, instead of at `import' on a later # ticket. # # The list is copied to tmpfs first and never redirected from the rootfs: a # top-level stdin redirect on the virtiofs root makes dup2(2) return 513 and ash # EXITS, ending the suite silently (brintos/linux#73). cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # EVERY REGULAR FILE THE PACKAGE SHIPS IS PYTHON, A STUB OR ITS METADATA. The # case above names the failure that is expected; this one catches the shape # nobody predicted -- a binary, a .pyd, a stray README -- without having to # enumerate what it might be. # # A dpkg .list names DIRECTORIES too, and without a trailing slash, so a # name-only filter would report every directory in the package as an unexpected # shape; each entry is therefore tested for being a regular file first. _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc|*.pyi) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" # ----------------------------------------------------------- the dist-info --- # The installed metadata, which is the second, independent statement of the # version -- `$P/dist-version' below reads it through importlib.metadata, and the # directory name carries it a third time in $DI itself. A package whose module # landed and whose metadata did not would import fine and serve every behavioural # case while reporting no installed version at all. # # The directory is `u_msgpack_python-2.8.0.dist-info', not the recipe's or the # distribution's own spelling: PEP 503/427 normalisation turns the hyphens of # `u-msgpack-python' into underscores. Asserting the normalised name is what # makes `dist-version' reachable at all. check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 2.8.0" cat "$DI/METADATA" # --------------------------------------------------------------- the package -- # dpkg KNOWS the package. Every case above would look identical if some other # package had installed the same files, so this is the one that says # python3-u-msgpack-python is a real package this image installed rather than an # IMAGE_INSTALL line that resolved to nothing. # # The stanza is read from a tmpfs copy for the same reason as the file list, and # the anchor is exact -- an unanchored match would also take a -ptest or -dev # stanza of the same family. sed -n '/^Package: python3-u-msgpack-python$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # THE -ptest SPLIT IS NOT ON THIS IMAGE. See the header: the recipe inherits # ptest and RDEPENDS that split on python3-pytest, which this image does not # carry -- and the absent `ptest' feature means the split is never built, so # the dependency is never resolved. This is the case that reds if the feature # is turned on and pytest is dragged in behind it. check_eq $P/no-ptest-package "absent" \ "$(if grep -q '^Package: python3-u-msgpack-python-ptest$' /var/lib/dpkg/status 2>/dev/null; \ then echo present; else echo absent; fi)" # THE RUNTIME DEPENDENCIES THE RECIPE DECLARES, as dpkg recorded them. Both are # sub-packages of the ported python3 (BRINTOS-1476) and both are real: # python3-datetime carries the `datetime' the module imports at scope and packs # as the timestamp extension. # # ONE CASE PER DEPENDENCY, against the extracted line, rather than one case # asserting the whole list as a string. The recipe's RDEPENDS order is not dpkg's # to preserve and the separator spacing is dpkg's own, so an exact-string case # here would be asserting a FORMAT this phase cannot measure. Split, each red also # names which dependency went missing instead of printing the whole line back. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-core "python3-core" cat "$W/dep" check_out $P/pkg-depends-datetime "python3-datetime" cat "$W/dep" # ---------------------------------------------------------- the interpreter --- # THE IMPORT, ON ITS OWN, BEFORE THE PROBE. If umsgpack cannot be imported then # every probe case below reds at once and each names a different symptom; this # case is what says the cause is the import. It is also the cheapest possible # statement of the deliverable. check $P/import python3 -c 'import umsgpack' # ...and `datetime' separately, first in the failure order that matters: umsgpack # imports it at module scope, so an image without the python3-datetime split # presents AS a broken umsgpack. This case is what tells the two apart. check $P/datetime-import python3 -c 'import datetime' # ---------------------------------------------------------------- the probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present # ONE interpreter start for everything that follows. bst_probe bounds it and # keeps BOTH streams: a probe that dies early otherwise leaves every case below # reporting got:'' and naming a symptom each, with the actual traceback nowhere. # The bound is generous because a python3 start on this arm is tens of seconds # before any work begins, and the five 32-bit-form vectors pack 65536 elements # each in a pure-Python loop. bst_probe $P/probe-run "$W/probe" 1500 python3 "$PROBE" OUT="$W/probe" # Read a single key. The probe emits each key at most once, so head -n 1 is # defensive rather than load-bearing; the anchors keep `version' distinguishable # from `version_tuple', and `enc_fixext1' from `enc_fixext16'. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE EVIDENCE PLANE. If the probe emitted nothing, every case below is reading # an absent file: each would red with got:'' and the set would read exactly like # a broken library, when what actually happened is that no measurement was taken. # The word `unscorable' is PARSED -- brintos_local_verify re-measures a suite that # prints it on a fresh guest rather than trusting a shared boot's numbers # (BRINTOS-1644) -- so it goes in this structural check, not in a comment. # # The cases below still print their reds: withdraw, do not hide. This case is # what says what those reds measure. if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # ------------------------------------------------------------ the identity --- # THE VERSION THE RECIPE PINS, three times and independently: the module's # `__version__' string, its `version' tuple, and importlib.metadata over the # installed dist-info. Any two disagreeing means a mixed install. check_eq $P/version "2.8.0" "$(pv version)" check_eq $P/version-tuple "2.8.0" "$(pv version_tuple)" check_eq $P/dist-version "2.8.0" "$(pv dist_version)" # The package answered out of the directory this image installed, not from a # stray copy on sys.path. This is what makes the payload cases above and the # behaviour cases below statements about the SAME umsgpack. check_eq $P/module-answered "$MOD" "$(pv module_file)" check_eq $P/package-answered "$PKG" "$(pv package_dir)" # THE PUBLIC SURFACE, WHOLE. umsgpack declares no `__all__', so this is `dir()' # minus the underscored names. It is NOT a static list: `__init()' runs at import # and BINDS pack/packb/unpack/unpackb, their dump/dumps/load/loads aliases, # `compatibility' and `xrange' as module globals. So this case is also the # statement that the module's own initialiser RAN -- a package that imported # without it would carry the exception classes and none of the functions. # # The names it imports at scope (collections, datetime, io, struct, sys, Hashable) # are deliberately in the list: they are what `from umsgpack import *' binds, and # a release that stopped importing one of them has changed its codec rather than # its formatting. `KeyDuplicateException' and `KeyNotPrimitiveException' are # upstream's deprecated aliases and are here for the same reason. check_eq $P/api-surface \ "DuplicateKeyException,Ext,Hashable,InsufficientDataException,InvalidString,InvalidStringException,KeyDuplicateException,KeyNotPrimitiveException,PackException,ReservedCodeException,UnhashableKeyException,UnpackException,UnsupportedTimestampException,UnsupportedTypeException,collections,compatibility,datetime,dump,dumps,ext_serializable,io,load,loads,pack,packb,struct,sys,unpack,unpackb,version,xrange" \ "$(pv api_surface)" # WHICH IMPLEMENTATION `__init()' PICKED. On Python 3 these must be the `3' # variants; the Python 2 arm returns `str' from packb and would fail every # encoding case below with a type error rather than with a wrong byte. check_eq $P/packb-impl "_packb3" "$(pv packb_impl)" check_eq $P/unpackb-impl "_unpackb3" "$(pv unpackb_impl)" # THE ONE PLATFORM PROPERTY THIS PORT DEPENDS ON -- see the header. A mant_dig # other than 53 would make the library pack every float as float32, correctly by # its own logic and wrongly for every peer on the wire. check_eq $P/float-precision "double" "$(pv float_precision)" check_eq $P/mant-dig "53" "$(pv mant_dig)" # The old-spec toggle, off. Every string vector below is written against the # CURRENT specification; a build shipping this on would encode str as the # pre-2013 raw form and red across them. check_eq $P/compatibility-default "False" "$(pv compatibility_default)" # ====================================================================== SPEC == # THE ENCODING VECTORS. Every expectation below is the MessagePack specification's # own answer for that value -- see the header. `enc-*' is the encoder against the # spec; `rt-*' is the decoder against the encoder. check_eq $P/enc-fixint-min "00" "$(pv enc_fixint_min)" check_eq $P/rt-fixint-min "yes" "$(pv rt_fixint_min)" check_eq $P/enc-fixint-max "7f" "$(pv enc_fixint_max)" check_eq $P/rt-fixint-max "yes" "$(pv rt_fixint_max)" check_eq $P/enc-uint8-min "cc80" "$(pv enc_uint8_min)" check_eq $P/rt-uint8-min "yes" "$(pv rt_uint8_min)" check_eq $P/enc-uint8-max "ccff" "$(pv enc_uint8_max)" check_eq $P/rt-uint8-max "yes" "$(pv rt_uint8_max)" check_eq $P/enc-uint16-min "cd0100" "$(pv enc_uint16_min)" check_eq $P/rt-uint16-min "yes" "$(pv rt_uint16_min)" check_eq $P/enc-uint16-max "cdffff" "$(pv enc_uint16_max)" check_eq $P/rt-uint16-max "yes" "$(pv rt_uint16_max)" check_eq $P/enc-uint32-min "ce00010000" "$(pv enc_uint32_min)" check_eq $P/rt-uint32-min "yes" "$(pv rt_uint32_min)" check_eq $P/enc-uint32-max "ceffffffff" "$(pv enc_uint32_max)" check_eq $P/rt-uint32-max "yes" "$(pv rt_uint32_max)" check_eq $P/enc-uint64-min "cf0000000100000000" "$(pv enc_uint64_min)" check_eq $P/rt-uint64-min "yes" "$(pv rt_uint64_min)" check_eq $P/enc-uint64-max "cfffffffffffffffff" "$(pv enc_uint64_max)" check_eq $P/rt-uint64-max "yes" "$(pv rt_uint64_max)" check_eq $P/enc-negfixint-max "ff" "$(pv enc_negfixint_max)" check_eq $P/rt-negfixint-max "yes" "$(pv rt_negfixint_max)" check_eq $P/enc-negfixint-min "e0" "$(pv enc_negfixint_min)" check_eq $P/rt-negfixint-min "yes" "$(pv rt_negfixint_min)" check_eq $P/enc-int8-max "d0df" "$(pv enc_int8_max)" check_eq $P/rt-int8-max "yes" "$(pv rt_int8_max)" check_eq $P/enc-int8-min "d080" "$(pv enc_int8_min)" check_eq $P/rt-int8-min "yes" "$(pv rt_int8_min)" check_eq $P/enc-int16-max "d1ff7f" "$(pv enc_int16_max)" check_eq $P/rt-int16-max "yes" "$(pv rt_int16_max)" check_eq $P/enc-int16-min "d18000" "$(pv enc_int16_min)" check_eq $P/rt-int16-min "yes" "$(pv rt_int16_min)" check_eq $P/enc-int32-max "d2ffff7fff" "$(pv enc_int32_max)" check_eq $P/rt-int32-max "yes" "$(pv rt_int32_max)" check_eq $P/enc-int32-min "d280000000" "$(pv enc_int32_min)" check_eq $P/rt-int32-min "yes" "$(pv rt_int32_min)" check_eq $P/enc-int64-max "d3ffffffff7fffffff" "$(pv enc_int64_max)" check_eq $P/rt-int64-max "yes" "$(pv rt_int64_max)" check_eq $P/enc-int64-min "d38000000000000000" "$(pv enc_int64_min)" check_eq $P/rt-int64-min "yes" "$(pv rt_int64_min)" check_eq $P/enc-nil "c0" "$(pv enc_nil)" check_eq $P/rt-nil "yes" "$(pv rt_nil)" check_eq $P/enc-false "c2" "$(pv enc_false)" check_eq $P/rt-false "yes" "$(pv rt_false)" check_eq $P/enc-true "c3" "$(pv enc_true)" check_eq $P/rt-true "yes" "$(pv rt_true)" check_eq $P/enc-float64-one "cb3ff0000000000000" "$(pv enc_float64_one)" check_eq $P/rt-float64-one "yes" "$(pv rt_float64_one)" check_eq $P/enc-float64-half "cb3fe0000000000000" "$(pv enc_float64_half)" check_eq $P/rt-float64-half "yes" "$(pv rt_float64_half)" check_eq $P/enc-float64-neg "cbc000000000000000" "$(pv enc_float64_neg)" check_eq $P/rt-float64-neg "yes" "$(pv rt_float64_neg)" check_eq $P/enc-fixstr-empty "a0" "$(pv enc_fixstr_empty)" check_eq $P/rt-fixstr-empty "yes" "$(pv rt_fixstr_empty)" check_eq $P/enc-fixstr-max "bf61616161616161616161616161616161616161616161616161616161616161" "$(pv enc_fixstr_max)" check_eq $P/rt-fixstr-max "yes" "$(pv rt_fixstr_max)" check_eq $P/enc-str8-min "d9206161616161616161616161616161616161616161616161616161616161616161" "$(pv enc_str8_min)" check_eq $P/rt-str8-min "yes" "$(pv rt_str8_min)" check_eq $P/enc-fixstr-utf8 "a6c3a9c3a8c3aa" "$(pv enc_fixstr_utf8)" check_eq $P/rt-fixstr-utf8 "yes" "$(pv rt_fixstr_utf8)" check_eq $P/enc-bin8-empty "c400" "$(pv enc_bin8_empty)" check_eq $P/rt-bin8-empty "yes" "$(pv rt_bin8_empty)" check_eq $P/enc-fixarray-empty "90" "$(pv enc_fixarray_empty)" check_eq $P/rt-fixarray-empty "yes" "$(pv rt_fixarray_empty)" check_eq $P/enc-fixarray-max "9fc0c0c0c0c0c0c0c0c0c0c0c0c0c0c0" "$(pv enc_fixarray_max)" check_eq $P/rt-fixarray-max "yes" "$(pv rt_fixarray_max)" check_eq $P/enc-array16-min "dc0010c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0" "$(pv enc_array16_min)" check_eq $P/rt-array16-min "yes" "$(pv rt_array16_min)" check_eq $P/enc-array-nested "9301920203a161" "$(pv enc_array_nested)" check_eq $P/rt-array-nested "yes" "$(pv rt_array_nested)" check_eq $P/enc-fixmap-empty "80" "$(pv enc_fixmap_empty)" check_eq $P/rt-fixmap-empty "yes" "$(pv rt_fixmap_empty)" check_eq $P/enc-fixmap-one "81a16101" "$(pv enc_fixmap_one)" check_eq $P/rt-fixmap-one "yes" "$(pv rt_fixmap_one)" check_eq $P/enc-map16-min "de001000c001c002c003c004c005c006c007c008c009c00ac00bc00cc00dc00ec00fc0" "$(pv enc_map16_min)" check_eq $P/rt-map16-min "yes" "$(pv rt_map16_min)" check_eq $P/enc-fixext1 "d40501" "$(pv enc_fixext1)" check_eq $P/rt-fixext1 "yes" "$(pv rt_fixext1)" check_eq $P/enc-fixext2 "d5050102" "$(pv enc_fixext2)" check_eq $P/rt-fixext2 "yes" "$(pv rt_fixext2)" check_eq $P/enc-fixext4 "d60501010101" "$(pv enc_fixext4)" check_eq $P/rt-fixext4 "yes" "$(pv rt_fixext4)" check_eq $P/enc-fixext8 "d7050101010101010101" "$(pv enc_fixext8)" check_eq $P/rt-fixext8 "yes" "$(pv rt_fixext8)" check_eq $P/enc-fixext16 "d80501010101010101010101010101010101" "$(pv enc_fixext16)" check_eq $P/rt-fixext16 "yes" "$(pv rt_fixext16)" check_eq $P/enc-ext8-three "c70305010101" "$(pv enc_ext8_three)" check_eq $P/rt-ext8-three "yes" "$(pv rt_ext8_three)" check_eq $P/enc-fixext1-negtype "d4fe01" "$(pv enc_fixext1_negtype)" check_eq $P/rt-fixext1-negtype "yes" "$(pv rt_fixext1_negtype)" # THE LONG VECTORS, by prefix, length and digest. Their payload is a 255- or # 256-byte constant run, so the literal would say nothing the format byte and the # big-endian length do not. The prefix carries exactly the part the specification # fixes: `d9ff' is str8 with an 8-bit length of 255, `da0100' is str16 with a # 16-bit length of 256, and so on. check_eq $P/enc-str8-max-prefix "d9ff61616161" "$(pv pfx_str8_max)" check_eq $P/enc-str8-max-len "257" "$(pv len_str8_max)" check_eq $P/enc-str8-max-sha "68f96e81cde41c2a" "$(pv sha_str8_max)" check_eq $P/rt-str8-max "yes" "$(pv rt_str8_max)" check_eq $P/enc-str16-min-prefix "da0100616161" "$(pv pfx_str16_min)" check_eq $P/enc-str16-min-len "259" "$(pv len_str16_min)" check_eq $P/enc-str16-min-sha "b54aed6be9f96cf2" "$(pv sha_str16_min)" check_eq $P/rt-str16-min "yes" "$(pv rt_str16_min)" check_eq $P/enc-bin8-max-prefix "c4ff01010101" "$(pv pfx_bin8_max)" check_eq $P/enc-bin8-max-len "257" "$(pv len_bin8_max)" check_eq $P/enc-bin8-max-sha "03a4a368b69a0531" "$(pv sha_bin8_max)" check_eq $P/rt-bin8-max "yes" "$(pv rt_bin8_max)" check_eq $P/enc-bin16-min-prefix "c50100010101" "$(pv pfx_bin16_min)" check_eq $P/enc-bin16-min-len "259" "$(pv len_bin16_min)" check_eq $P/enc-bin16-min-sha "9a215693ccf5ffd7" "$(pv sha_bin16_min)" check_eq $P/rt-bin16-min "yes" "$(pv rt_bin16_min)" check_eq $P/enc-ext8-max-prefix "c7ff05010101" "$(pv pfx_ext8_max)" check_eq $P/enc-ext8-max-len "258" "$(pv len_ext8_max)" check_eq $P/enc-ext8-max-sha "5ef1ac01c00d7967" "$(pv sha_ext8_max)" check_eq $P/rt-ext8-max "yes" "$(pv rt_ext8_max)" check_eq $P/enc-ext16-min-prefix "c80100050101" "$(pv pfx_ext16_min)" check_eq $P/enc-ext16-min-len "260" "$(pv len_ext16_min)" check_eq $P/enc-ext16-min-sha "148b5f1f6d4a479d" "$(pv sha_ext16_min)" check_eq $P/rt-ext16-min "yes" "$(pv rt_ext16_min)" # THE 32-BIT FORMS. Each needs 65536 elements or bytes -- the first value at # which the 16-bit length no longer fits -- so each is asserted by its prefix # (format byte plus big-endian uint32 length), its total encoded length, and a # digest of the whole. These are the vectors that cost the probe its run time. check_eq $P/big-str32-prefix "db00010000" "$(pv big_str32_prefix)" check_eq $P/big-str32-len "65541" "$(pv big_str32_len)" check_eq $P/big-str32-sha "5f97d047d40a13f5" "$(pv big_str32_sha)" check_eq $P/big-str32-rt "yes" "$(pv big_str32_rt)" check_eq $P/big-bin32-prefix "c600010000" "$(pv big_bin32_prefix)" check_eq $P/big-bin32-len "65541" "$(pv big_bin32_len)" check_eq $P/big-bin32-sha "a9076426b653d1aa" "$(pv big_bin32_sha)" check_eq $P/big-bin32-rt "yes" "$(pv big_bin32_rt)" check_eq $P/big-array32-prefix "dd00010000" "$(pv big_array32_prefix)" check_eq $P/big-array32-len "65541" "$(pv big_array32_len)" check_eq $P/big-array32-sha "70cb3d160eaae986" "$(pv big_array32_sha)" check_eq $P/big-array32-rt "yes" "$(pv big_array32_rt)" check_eq $P/big-map32-prefix "df00010000" "$(pv big_map32_prefix)" check_eq $P/big-map32-len "261765" "$(pv big_map32_len)" check_eq $P/big-map32-sha "227bf08057b7aa70" "$(pv big_map32_sha)" check_eq $P/big-map32-rt "yes" "$(pv big_map32_rt)" check_eq $P/big-ext32-prefix "c90001000005" "$(pv big_ext32_prefix)" check_eq $P/big-ext32-len "65542" "$(pv big_ext32_len)" check_eq $P/big-ext32-sha "cdd32c5c8d998ce9" "$(pv big_ext32_sha)" check_eq $P/big-ext32-rt "yes" "$(pv big_ext32_rt)" # THE BOOL/INT DISTINCTION, WHICH `rt-true' CANNOT SEE. In Python `True == 1', so # a decoder that returned the integer 1 for 0xc3 passes `rt-true'. These are the # cases that say the TYPE came back too -- and it is not pedantry: `_pack3' tests # `isinstance(obj, bool)' BEFORE `isinstance(obj, int)' precisely because the # reverse order would encode True as fixint 1. check_eq $P/rt-true-type "bool" "$(pv rt_true_type)" check_eq $P/rt-zero-type "int" "$(pv rt_zero_type)" check_eq $P/enc-true-not-int "yes" "$(pv enc_true_not_int)" # THE FORCED FLOAT PRECISION. `force_float_precision' is the one packing option # that changes the FORMAT BYTE, so it is driven rather than left to the # auto-detected default: 1.0 as float32 is `ca 3f800000', and forcing "double" # on this arm must reproduce the default `cb 3ff0000000000000' exactly. check_eq $P/enc-float32-one "ca3f800000" "$(pv enc_float32_one)" check_eq $P/enc-float32-forced-double "cb3ff0000000000000" "$(pv enc_float32_forced_double)" # ------------------------------------------------------- the timestamp ext --- # WHY THIS RECIPE RDEPENDS ON python3-datetime, AS BEHAVIOUR. The specification # reserves ext type -1 (0xff) for timestamps: the 32-bit form is `d6 ff' followed # by a big-endian uint32 of seconds, so 2020-01-01T00:00:00Z = 1577836800 = # 0x5e0be100. The 64-bit form (`d7 ff') appears the moment microseconds are # non-zero, packing nanoseconds in the top 30 bits and seconds in the low 34. check_eq $P/enc-timestamp32 "d6ff5e0be100" "$(pv enc_timestamp32)" check_eq $P/rt-timestamp32 "yes" "$(pv rt_timestamp32)" check_eq $P/enc-timestamp64 "d7ff773594005e0be100" "$(pv enc_timestamp64)" check_eq $P/rt-timestamp64 "yes" "$(pv rt_timestamp64)" # The decoded object is TIMEZONE-AWARE UTC, not naive -- a decoder that dropped # the tzinfo would compare equal above only by luck of the local zone, and this # image's is UTC. check_eq $P/timestamp-tzinfo "UTC" "$(pv timestamp_tzinfo)" # ------------------------------------------------------------- the Ext type --- # Equality is by (type, data) and NOT by identity, and the hash agrees with it -- # which is what lets an Ext be a dict key. `__ne__' is written out separately in # this module rather than derived from `__eq__', so it is asserted separately too, # including against an object of another type entirely. check_eq $P/ext-eq "yes" "$(pv ext_eq)" check_eq $P/ext-not-identical "yes" "$(pv ext_not_identical)" check_eq $P/ext-ne-type "yes" "$(pv ext_ne_type)" check_eq $P/ext-ne-data "yes" "$(pv ext_ne_data)" check_eq $P/ext-ne-other "yes" "$(pv ext_ne_other)" check_eq $P/ext-hash-eq "yes" "$(pv ext_hash_eq)" # Two equal Exts collapse to ONE dict entry. This is the case that says the hash # and the equality agree operationally rather than merely returning equal ints. check_eq $P/ext-hash-usable "yes" "$(pv ext_hash_usable)" check_eq $P/ext-str "Ext Object (Type: 5, Data: 0x01 0x02)" "$(pv ext_str)" check_eq $P/ext-type-attr "5" "$(pv ext_type_attr)" check_eq $P/ext-data-attr "0102" "$(pv ext_data_attr)" # The constructor's own guards. The wire format has ONE SIGNED BYTE for the ext # type, so an out-of-range value would be silently truncated into a different # type code if the constructor did not refuse it -- which is a data-corruption # bug, not an ergonomics one. Note the two exception classes differ: a wrong TYPE # is TypeError, an out-of-RANGE int is ValueError. check_eq $P/ext-reject-type-not-int "TypeError" "$(pv ext_reject_type_not_int)" check_eq $P/ext-reject-type-too-big "ValueError" "$(pv ext_reject_type_too_big)" check_eq $P/ext-reject-type-too-small "ValueError" "$(pv ext_reject_type_too_small)" check_eq $P/ext-reject-data-not-bytes "TypeError" "$(pv ext_reject_data_not_bytes)" # --------------------------------------------------------- option: ext_handlers # The documented route for a type THE FORMAT DOES NOT DEFINE. A handler maps a # Python class to an Ext on the way out and an ext type code back to the class on # the way in. A Point(3,4) becomes ext type 0x10 carrying `packb([3,4])' -- two # bytes of payload plus the fixarray header is three, which has no fixext form and # must fall to ext8: `c7 03 10' then `92 03 04'. check_eq $P/ext-handler-enc "c70310920304" "$(pv ext_handler_enc)" check_eq $P/ext-handler-rt "yes" "$(pv ext_handler_rt)" check_eq $P/ext-handler-rt-type "Point" "$(pv ext_handler_rt_type)" # WITHOUT the unpack handler the same bytes decode to a plain Ext rather than # failing -- the ext payload is opaque to the decoder, which is the property that # makes an unknown extension forward-compatible instead of fatal. check_eq $P/ext-handler-absent-type "Ext" "$(pv ext_handler_absent_type)" check_eq $P/ext-handler-absent-code "16" "$(pv ext_handler_absent_code)" # ...and without a PACK handler the object is refused rather than encoded to # something lossy. check_eq $P/ext-handler-required "UnsupportedTypeException" "$(pv ext_handler_required)" # ------------------------------------------ option: use_ordered_dict/use_tuple # Both change the TYPE the decoder builds, not the value, so each is asserted by # type name -- an `==' comparison cannot see either: an OrderedDict equals the # dict with the same items, and a tuple never equals a list, so a value # comparison would report the wrong answer in both directions. check_eq $P/ordered-default-type "dict" "$(pv ordered_default_type)" check_eq $P/ordered-opt-type "OrderedDict" "$(pv ordered_opt_type)" check_eq $P/ordered-opt-keys "a,b" "$(pv ordered_opt_keys)" check_eq $P/tuple-default-type "list" "$(pv tuple_default_type)" check_eq $P/tuple-opt-type "tuple" "$(pv tuple_opt_type)" # ...and it is DEEP: the nested array is a tuple too, which is what makes the # result usable as a dict key and is the option's actual purpose. check_eq $P/tuple-opt-nested-type "tuple" "$(pv tuple_opt_nested_type)" check_eq $P/tuple-opt-value "(1, (2, 3))" "$(pv tuple_opt_value)" # ------------------------------------------------------- the documented errors # Each of these is behaviour the module DOCUMENTS, and each is asserted as the # NAMED exception rather than tolerated. A decoder that returned a partial value # for any of them would be a silent data-corruption bug in a serialiser -- which # is the one class of defect a wire-format library must not have. # # The bytes each case drives are spelled out so a red can be reproduced by hand: # truncated cd 01 -- uint16 declares two payload bytes, one follows # truncated-empty (nothing) -- not even a format byte # reserved-code c1 -- reserved by the specification, never on the wire # unhashable-key 81 80 c0 -- fixmap(1) whose KEY is fixmap(0): a dict # duplicate-key 82 a1 61 01 a1 61 02 -- fixmap(2) with "a" twice # invalid-utf8 a1 ff -- fixstr(1) whose byte is not UTF-8 # bad-timestamp d5 ff 00 00 -- ext type -1 with a 2-byte payload, which is # none of the 32-, 64- or 96-bit forms check_eq $P/err-truncated "InsufficientDataException" "$(pv err_truncated)" check_eq $P/err-truncated-empty "InsufficientDataException" "$(pv err_truncated_empty)" check_eq $P/err-reserved-code "ReservedCodeException" "$(pv err_reserved_code)" check_eq $P/err-unhashable-key "UnhashableKeyException" "$(pv err_unhashable_key)" check_eq $P/err-duplicate-key "DuplicateKeyException" "$(pv err_duplicate_key)" check_eq $P/err-invalid-utf8 "InvalidStringException" "$(pv err_invalid_utf8)" check_eq $P/err-bad-timestamp "UnsupportedTimestampException" "$(pv err_bad_timestamp)" # The two reachable size guards -- see the header for why the 4 GiB ones are not. check_eq $P/err-huge-uint "UnsupportedTypeException" "$(pv err_huge_uint)" check_eq $P/err-huge-int "UnsupportedTypeException" "$(pv err_huge_int)" # The argument guard: unpackb takes bytes, and a str would otherwise be iterated # as characters somewhere deep inside io.BytesIO. check_eq $P/err-unpackb-str "TypeError" "$(pv err_unpackb_str)" # THE EXCEPTION HIERARCHY, WHICH IS THE PART A CALLER WRITES `except' AGAINST. # `except UnpackException' has to catch every decode failure above. The module # would pass every case above with a FLAT hierarchy and break every caller that # relied on the documented one, so the hierarchy is its own assertion. check_eq $P/exc-pack-base "yes" "$(pv exc_pack_base)" check_eq $P/exc-unpack-base "yes" "$(pv exc_unpack_base)" # `allow_invalid_utf8' is the documented escape from InvalidStringException: the # bytes come back wrapped in InvalidString, a bytes SUBCLASS, so a caller can # inspect what actually arrived instead of losing it. check_eq $P/invalid-utf8-type "InvalidString" "$(pv invalid_utf8_type)" check_eq $P/invalid-utf8-is-bytes "yes" "$(pv invalid_utf8_is_bytes)" check_eq $P/invalid-utf8-value "ff" "$(pv invalid_utf8_value)" # ------------------------------------------------------ the stream interface --- # `pack'/`unpack' are the same codecs over a file-like object rather than over a # buffer, and they are what `dump'/`load' alias. Driven over io.BytesIO so the # assertion is about the API and not about the filesystem. check_eq $P/stream-enc "81a161920102" "$(pv stream_enc)" check_eq $P/stream-rt "yes" "$(pv stream_rt)" # The four aliases are the SAME objects, not re-implementations that could drift. check_eq $P/alias-dumps "yes" "$(pv alias_dumps)" check_eq $P/alias-loads "yes" "$(pv alias_loads)" check_eq $P/alias-dump "yes" "$(pv alias_dump)" check_eq $P/alias-load "yes" "$(pv alias_load)" # ------------------------------------------------------- the module closure --- # WHAT THIS PACKAGE PULLS IN, MEASURED RATHER THAN READ OFF THE RECIPE. This # image's python3 is split into sub-packages (BRINTOS-1476), so a module reached # at import scope that lives outside python3-core is a runtime dependency the # recipe has to declare. Every name is scored against `sys.stdlib_module_names', # so a third-party import could not hide in the list. check_eq $P/closure-present "collections,datetime,io,struct,sys" "$(pv closure_present)" check_eq $P/closure-all-stdlib "yes" "$(pv closure_all_stdlib)" # ...and `datetime' by name, because it is the one that is NOT in python3-core and # is the reason the recipe's RDEPENDS line reads as it does. check_eq $P/closure-datetime "yes" "$(pv datetime_imported)" # The probe ran to the end. Every case above greps a key, so a probe that died # halfway would leave the remaining keys absent and each case would red with # got:'' -- this case is the one that says the cause was a truncated run. check_eq $P/probe-complete "yes" "$(pv done)" rm -rf "$W"