. "$BST_LIB" # BST-TIMEOUT: 2400 # FIVE python3 starts, and they are the whole cost. # # Everything else below reads the filesystem or dpkg with # # shell builtins, or greps the output of one of the five. # # # # The five are: a bare `import\'; a second import whose # # STDERR is the assertion (`import-quiet\'); the probe, # # which takes ONE start for 137 measurements; and TWO runs # # of `python3 -m ubjson\', which is a separate entry point # # (ubjson/__main__.py) and so is not foldable into the # # probe -- reaching the module AS A PROGRAM is what those # # two cases measure. # # # # 2400 is 480s per start, against the ~60s a python3 start # # has measured at on this arm -- the same budget # # 99-python3-cbor2 carries for the same five-start shape. # # # # NOTHING HERE IS QUADRATIC IN ANYTHING, and nothing here # # is deep: the largest input is a 256-byte string and the # # deepest is 16 nested arrays. See "WHAT IS DELIBERATELY # # NOT COVERED" below for why the depth is 16 and not 400. # # # # NOT MEASURED ON TARGET YET: this port\'s first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-py-ubjson 0.16.1 (meta-openembedded/meta-python # recipes-devtools/python/python3-py-ubjson_0.16.1.bb, `inherit pypi # setuptools3\', PyPI sdist, no patches, no DEPENDS, RDEPENDS python3-numbers) # on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE DOES. # ============================================================================ # # py-ubjson implements UBJSON draft-12 -- JSON\'s data model in a binary # framing, with type tags and the optimised container forms UBJSON adds on top. # `dumpb\'/`loadb\' over bytes, `dump\'/`load\' over a file object, and a # `python3 -m ubjson\' converter between json and ubjson. # # ============================================================================ # THE ONE THING THIS PORT DECIDED, AND WHY EVERY CASE BELOW DEPENDS ON IT. # ============================================================================ # # py-ubjson ships TWO implementations of the same four callables: the # pure-Python ubjson/encoder.py + ubjson/decoder.py, and `_ubjson\', a top-level # CPython extension built from src/*.c. ubjson/__init__.py picks between them # with `try: from _ubjson import ...\' guarded by `except ImportError\', and # publishes which one won as `EXTENSION_ENABLED\'. # # THIS IMAGE SHIPS THE PURE-PYTHON HALF AND NO EXTENSION AT ALL. A CPython C # extension links on this arm and cannot be LOADED -- hwjs-dlopen refuses it # with `policy -> ELIBBAD -80\' on unresolved env.* imports, because this # image\'s python3.12 exports 35 symbols, none of them Py*, with no # libpython3.12.so shipped to bind against instead (BRINTOS-2046, first measured # on python3-bitarray in BRINTOS-2032, then on python3-cbor2 and # python3-crcmod). So python3-py-ubjson_%.bbappend declines to build it. # # `pkg-no-so\', `no-c-ext\' and `ext-disabled\' below are therefore not hygiene # cases: they are the assertion that the bbappend still takes. A run where they # went red would mean every behavioural case in this suite had silently changed # which code it was measuring. # # THE BBAPPEND IS NOT SPELLED THE WAY python3-cbor2_%.bbappend IS, and the # difference is measured rather than stylistic: cbor2\'s opt-out is read for a # VALUE, while py-ubjson\'s is a PRESENCE test -- # # BUILD_EXTENSIONS = \'PYUBJSON_NO_EXTENSION\' not in os.environ and ... # # -- so ANY value disables it, "0" included, and bitbake exports a variable # whose value is empty. Writing it cbor2-style would have disabled the # accelerator on native and nativesdk too, where it builds and works and is # worth having, and nothing would ever have reported it. The bbappend\'s header # carries the whole reasoning. # # ============================================================================ # WHERE THE EXPECTED VALUES CAME FROM. # ============================================================================ # # Every expectation in this file is the OUTPUT of a control run of # data/python3-py-ubjson-probe.py -- py-ubjson 0.16.1, pure-Python, on # CPython 3.12.13 taken from THIS build\'s own python3-native sysroot rather # than from a distro python, which is the version pair the target runs -- and # was then read back against the UBJSON draft-12 spec by hand. # # The encode cases pin LITERAL BYTES and the round-trip cases pin the value. # Both directions are needed: a round-trip alone cancels a symmetric error out, # so an encoder writing the wrong width paired with a decoder reading it back # the same way would pass every round-trip and still be wrong on the wire. # # ============================================================================ # FIVE PLACES py-ubjson DEPARTS FROM WHAT A READER WOULD GUESS. # ============================================================================ # # Each is pinned below as MEASURED BEHAVIOUR, not asserted as correctness, and # each is a case a reviewer would otherwise read as a bug in this port: # # * A single-BYTE string encodes as CHAR (`C\'), with no length prefix, so # `dumpb("a")\' is 2 bytes and `dumpb("ab")\' is 5 (`str-enc-char\'). # * 0.0 encodes as FLOAT32 even under the default no_float32=True, because # __encode_float64\'s zero arm short-circuits (`float-enc-zero\'). # * inf and nan encode as NULL. The value is DESTROYED by its own encoder and # `float-inf-rt\' records that rather than pretending it survives. # * An integer past 2**63 goes out as high-precision `H\' and comes BACK as a # Decimal, so the round-trip is not type-preserving at the top of the range # (`int-rt-u9223372036854775808\'). # * Integer widths are ASYMMETRIC about zero -- a positive takes uint8 to 255 # then int16, a negative takes int8 only to -128 -- so +200 is `U\' and # -200 is `I\'. The int-enc-* pairs below are the last value of one width # and the first of the next, on each side separately, which is where an # off-by-one shows and nowhere else. # # ============================================================================ # WHAT IS DELIBERATELY NOT COVERED, AND ONE TICKET CORRECTION. # ============================================================================ # # THE TICKET\'S DESCRIPTION ASKS FOR THREE THINGS THIS PACKAGE DOES NOT DO, and # they are listed here so the next reader does not go hunting for the cases: # # * "a recursion depth past the configured limit raises the module\'s decoder # error" -- the pure-Python decoder has NO configured limit. Depth is # bounded by CPython\'s own stack, so nesting past it raises RecursionError, # not DecoderException. (The C extension does have one; it is not shipped.) # Nothing here tests it: the recursion is one frame per level and this # guest\'s stack is not the host\'s, so a deep-nesting probe risks WEDGING # THE MACHINE for every other suite in a shared batch image while measuring # nothing about this package. Depth 16 exercises the recursive descent and # stops well short of characterising the limit, which is not this port\'s to # characterise. # * "a typed array without a count" as a form to DECODE -- draft-12 requires # `#\' whenever `$\' is present, and py-ubjson refuses it. It is covered as # a failure case (`opt-typed-no-count\'), not a decode. # * "a strongly typed array with a count" as an ENCODER output -- the encoder # emits `$\' for exactly one input, a bytes object, and never for a list. # The typed forms are therefore fed to the DECODER as literals # (`opt-dec-typed-*\'), which is the only direction they occur in. # # The sdist ships test/test.py, but the recipe does not inherit `ptest\' and the # image has no pytest, so the suite drives the API directly instead of running # upstream\'s tests. There is no ptest case here. # ============================================================================ P=python3-py-ubjson # `suite-canary\' is emitted FIRST and unconditionally. A suite that dies before # its first result line is indistinguishable from one that was never scheduled, # so this one line is what separates "ran and found nothing" from "did not run". bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/ubjson DI=$SITE/py_ubjson-0.16.1.dist-info PKGLIST=/var/lib/dpkg/info/python3-py-ubjson.list PROBE=$BST_DATA/python3-py-ubjson-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-py-ubjson.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ---------------------------------------------------------------- packaging -- # WHERE THE PACKAGING SAYS THE MODULE LANDS. These run before any import, so a # red here says the .deb is wrong rather than that the codec misbehaves. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" check $P/package-init test -s "$PKG/__init__.py" # It is a PACKAGE, not a single module. A release that flattened it would still # import and still pass every behavioural case, so the shape is asserted. check $P/no-flat-module test ! -f "$SITE/ubjson.py" # The five modules the public names come out of. `__init__.py\' does # `from .encoder import ...\', so a truncated install fails at import rather # than here -- naming them individually is what turns "the package is complete" # into a measurement the packaging can be judged by. `__main__.py\' is the # `python3 -m ubjson\' entry point the cli-* cases below reach. for _m in compat decoder encoder markers __main__; do check $P/module-$_m test -s "$PKG/$_m.py" done # ---------------------------------------------------- THE EXTENSION IS ABSENT -- # THE ASSERTION THE WHOLE PORT TURNS ON. `_ubjson.cpython-312.so\' is what the # stock recipe builds and what python3-py-ubjson_%.bbappend declines; it cannot # bind this interpreter\'s C API (BRINTOS-2046) and would ship a file whose best # outcome is being swallowed by an except-clause. Both the exact path and the # shape of the package list are asserted, because the second one catches a .so # landing under any other name. check $P/no-c-ext-so test ! -e "$SITE/_ubjson.cpython-312.so" check $P/no-c-ext-any sh -c "! ls $SITE/_ubjson*.so >/dev/null 2>&1" # THIS PACKAGE INSTALLS NO EXECUTABLE. Unlike python3-cbor2 and like # python3-pyparsing, setup.py carries no `entry_points\'/`console_scripts\', so # there is no /usr/bin/ubjson and the CLI is reached only as `python3 -m\'. check $P/no-bin-script test ! -e /usr/bin/ubjson # --------------------------------------------------------- the dist-info ------ check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 0.16.1" cat "$DI/METADATA" check_out $P/dist-info-name "Name: py-ubjson" cat "$DI/METADATA" check $P/dist-info-wheel test -f "$DI/WHEEL" # Root-Is-Purelib: true IS THE BBAPPEND\'S SIGNATURE IN THE METADATA. With the # extension built this wheel is platlib and the tag is a platform tag; with it # declined the wheel is pure and the tag is py3-none-any. So these two lines # say, from the packaging side rather than from a file check, that the build # produced a pure-Python distribution. check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag "Tag: py3-none-any" cat "$DI/WHEEL" # top_level.txt names ONLY `ubjson\'. With the extension built it names # `_ubjson\' as well, so this one-word file is the shortest statement that the # accelerator was not packaged. check_eq $P/dist-info-top-level "ubjson" "$(cat "$DI/top_level.txt" 2>/dev/null)" # ------------------------------------------------------------ the .deb list --- cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # NO SHARED OBJECT ANYWHERE IN THE PACKAGE. The file check above names the one # path the extension would take; this one catches it under any name. check_eq $P/pkg-no-so "" \ "$(grep -c '\.so$' "$W/pkgf" 2>/dev/null | grep -v '^0$')" # ------------------------------------------------------------------ imports --- check $P/import python3 -c "import ubjson" # STDERR MUST BE EMPTY. With the extension present and unloadable, the loader\'s # refusal is what would land here; with it declined the `except ImportError\' # arm is taken silently. So this line is a statement about the extension as much # as about the module. check_eq $P/import-quiet "" \ "$(python3 -c 'import ubjson' 2>&1 >/dev/null)" # ------------------------------------------------------------------ probe ----- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "probe missing at $PROBE" exit 0 fi bst_ok $P/probe-present # 1200s is half the suite budget for the one start that does 137 measurements. # On overrun bst_probe names the last line the probe printed, which names the # case it stopped at. bst_probe $P/probe-run "$W/probe" 1200 python3 "$PROBE" OUT="$W/probe" # One `grep\' per value, split on the FIRST `=\' so a value containing one is # safe. Every key the probe emits is [a-z0-9_]+. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE PROBE RAN TO THE END AND EMITTED EVERYTHING. Without these two, a probe # that died half way would report every case after the death as a mismatch # against an empty string, which reads as 137 behavioural failures instead of # one truncated transcript. check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "137" "$(pv emitted_count)" # ---------------------------------------------------------------- identity -- # WHICH COPY ANSWERED, AND WHICH OF THE TWO IMPLEMENTATIONS IT IS. # `impl-module\' is `ubjson.encoder\' on the pure-Python path and `_ubjson\' on # the accelerated one, because the `except ImportError\' arm binds the four # callables from the pure-Python modules. It is the one-word answer to which # half of the codec every case below is about. check_eq $P/module-answered "$PKG/__init__.py" "$(pv module_file)" check_eq $P/is-package "yes" "$(pv is_package)" check_eq $P/version "0.16.1" "$(pv version)" check_eq $P/all-names "DecoderException,EXTENSION_ENABLED,EncoderException,dump,dumpb,load,loadb" "$(pv all_names)" # EXTENSION_ENABLED IS UPSTREAM\'S OWN PUBLIC ANSWER to which implementation # bound dump/dumpb/load/loadb, and `no\' here is the bbappend taking effect at # runtime rather than at packaging time. With the extension shipped and # unloadable this would still be `no\' -- the try/except would swallow the # refusal -- which is why the packaging cases above assert the file is ABSENT # and this one asserts the consequence. check_eq $P/ext-disabled "no" "$(pv ext_enabled)" check_eq $P/impl-module "ubjson.encoder" "$(pv impl_module)" check_eq $P/c-ext-import-fails "ModuleNotFoundError: No module named '_ubjson'" "$(pv c_ext_import)" # The exception types come from the pure-Python modules in BOTH arms, so these # say which module object answered rather than merely that a name exists. Their # BASES are part of the public contract: EncoderException is a TypeError and # DecoderException a ValueError, so `except ValueError\' catches a decode # failure in caller code that never imported the module\'s own names. check_eq $P/encoder-exc-module "ubjson.encoder" "$(pv encoder_exc_module)" check_eq $P/decoder-exc-module "ubjson.decoder" "$(pv decoder_exc_module)" check_eq $P/encoder-exc-base "TypeError" "$(pv encoder_exc_base)" check_eq $P/decoder-exc-base "ValueError" "$(pv decoder_exc_base)" # ------------------------------------------------- constants: Z, T, F, noop -- # The three no-data types are one byte each, and a `N\' no-op INSIDE a container # is skipped rather than decoded -- the one UBJSON marker that carries no value # and occupies no slot. check_eq $P/enc-null "5a" "$(pv enc_null)" check_eq $P/enc-true "54" "$(pv enc_true)" check_eq $P/enc-false "46" "$(pv enc_false)" check_eq $P/dec-null "None" "$(pv dec_null)" check_eq $P/dec-true "True" "$(pv dec_true)" check_eq $P/dec-false "False" "$(pv dec_false)" check_eq $P/dec-noop-in-array "[1]" "$(pv dec_noop_in_array)" # -------------------------------------------- integer widths and their edges -- # THE BOUNDARY OF EVERY INTEGER ENCODING, and the point of the whole group: the # widths are ASYMMETRIC about zero. A positive takes uint8 (`U\') to 255 and # then int16 (`I\') to 2**15-1; a negative takes int8 (`i\') only to -128 and # then int16. So each pair below is the LAST value of one width and the FIRST of # the next, on each side of zero SEPARATELY, which is where an off-by-one in the # width selection shows and nowhere else. # # The two 2**63 cases are the top of the range: past int64 the encoder falls # back to high-precision `H\', a length-prefixed DECIMAL STRING, so the value # survives and its TYPE does not -- an int goes out and a Decimal comes back. # `int-rt-u9223372036854775808\' records that. check_eq $P/int-enc-u0 "5500" "$(pv int_enc_u0)" check_eq $P/int-enc-u127 "557f" "$(pv int_enc_u127)" check_eq $P/int-enc-u255 "55ff" "$(pv int_enc_u255)" check_eq $P/int-enc-u256 "490100" "$(pv int_enc_u256)" check_eq $P/int-enc-u32767 "497fff" "$(pv int_enc_u32767)" check_eq $P/int-enc-u32768 "6c00008000" "$(pv int_enc_u32768)" check_eq $P/int-enc-u2147483647 "6c7fffffff" "$(pv int_enc_u2147483647)" check_eq $P/int-enc-u2147483648 "4c0000000080000000" "$(pv int_enc_u2147483648)" check_eq $P/int-enc-u9223372036854775807 "4c7fffffffffffffff" "$(pv int_enc_u9223372036854775807)" check_eq $P/int-enc-u9223372036854775808 "48551339323233333732303336383534373735383038" "$(pv int_enc_u9223372036854775808)" check_eq $P/int-enc-n1 "69ff" "$(pv int_enc_n1)" check_eq $P/int-enc-n128 "6980" "$(pv int_enc_n128)" check_eq $P/int-enc-n129 "49ff7f" "$(pv int_enc_n129)" check_eq $P/int-enc-n32768 "498000" "$(pv int_enc_n32768)" check_eq $P/int-enc-n32769 "6cffff7fff" "$(pv int_enc_n32769)" check_eq $P/int-enc-n2147483648 "6c80000000" "$(pv int_enc_n2147483648)" check_eq $P/int-enc-n2147483649 "4cffffffff7fffffff" "$(pv int_enc_n2147483649)" check_eq $P/int-enc-n9223372036854775808 "4c8000000000000000" "$(pv int_enc_n9223372036854775808)" check_eq $P/int-enc-n9223372036854775809 "4855142d39323233333732303336383534373735383039" "$(pv int_enc_n9223372036854775809)" # THE SAME NINETEEN VALUES THROUGH dumpb->loadb. A round-trip alone would cancel # a symmetric width error out, which is why the literals above come first; these # say the value is preserved as well as the bytes. check_eq $P/int-rt-u0 "0" "$(pv int_rt_u0)" check_eq $P/int-rt-u127 "127" "$(pv int_rt_u127)" check_eq $P/int-rt-u255 "255" "$(pv int_rt_u255)" check_eq $P/int-rt-u256 "256" "$(pv int_rt_u256)" check_eq $P/int-rt-u32767 "32767" "$(pv int_rt_u32767)" check_eq $P/int-rt-u32768 "32768" "$(pv int_rt_u32768)" check_eq $P/int-rt-u2147483647 "2147483647" "$(pv int_rt_u2147483647)" check_eq $P/int-rt-u2147483648 "2147483648" "$(pv int_rt_u2147483648)" check_eq $P/int-rt-u9223372036854775807 "9223372036854775807" "$(pv int_rt_u9223372036854775807)" check_eq $P/int-rt-u9223372036854775808 "Decimal('9223372036854775808')" "$(pv int_rt_u9223372036854775808)" check_eq $P/int-rt-n1 "-1" "$(pv int_rt_n1)" check_eq $P/int-rt-n128 "-128" "$(pv int_rt_n128)" check_eq $P/int-rt-n129 "-129" "$(pv int_rt_n129)" check_eq $P/int-rt-n32768 "-32768" "$(pv int_rt_n32768)" check_eq $P/int-rt-n32769 "-32769" "$(pv int_rt_n32769)" check_eq $P/int-rt-n2147483648 "-2147483648" "$(pv int_rt_n2147483648)" check_eq $P/int-rt-n2147483649 "-2147483649" "$(pv int_rt_n2147483649)" check_eq $P/int-rt-n9223372036854775808 "-9223372036854775808" "$(pv int_rt_n9223372036854775808)" check_eq $P/int-rt-n9223372036854775809 "Decimal('-9223372036854775809')" "$(pv int_rt_n9223372036854775809)" # Decoding each width from its LITERAL bytes -- the direction a round-trip # cannot check, because it never exercises a form the encoder does not emit. check_eq $P/dec-uint8 "255" "$(pv dec_uint8)" check_eq $P/dec-int8 "-128" "$(pv dec_int8)" check_eq $P/dec-int16 "32767" "$(pv dec_int16)" check_eq $P/dec-int32 "2147483647" "$(pv dec_int32)" check_eq $P/dec-int64 "9223372036854775807" "$(pv dec_int64)" # ------------------------------------------------------------------- floats -- # 0.0 IS A FLOAT32 EVEN THOUGH no_float32 DEFAULTS TRUE. Measured, not assumed: # __encode_float64 short-circuits zero to the float32 packer, so the one value # a reader would expect to be the safest is the one that takes the narrow form. check_eq $P/float-enc-zero "6400000000" "$(pv float_enc_zero)" check_eq $P/float-enc-1p5 "443ff8000000000000" "$(pv float_enc_1p5)" check_eq $P/float-enc-1p5-f32 "643fc00000" "$(pv float_enc_1p5_f32)" check_eq $P/float-enc-0p1 "443fb999999999999a" "$(pv float_enc_0p1)" check_eq $P/float-enc-0p1-f32 "643dcccccd" "$(pv float_enc_0p1_f32)" # float32 IS LOSSY AND THE ROUND-TRIP IS NOT THE IDENTITY. This pair is the # whole reason no_float32 defaults to True, so both halves are recorded: 0.1 # survives as 0.1 by default and comes back as 0.10000000149011612 under # no_float32=False. A suite that asserted round-trip identity for both would be # asserting something false about IEEE 754, not about this port. check_eq $P/float-rt-0p1 "0.1" "$(pv float_rt_0p1)" check_eq $P/float-rt-0p1-f32 "0.10000000149011612" "$(pv float_rt_0p1_f32)" # inf/nan ENCODE AS NULL: the value does not survive its own encoder. That is # upstream\'s documented choice -- UBJSON draft-12 has no infinity or NaN type -- # and `float-inf-rt\' pins the CONSEQUENCE (None comes back) rather than # pretending the value round-trips. check_eq $P/float-enc-inf "5a" "$(pv float_enc_inf)" check_eq $P/float-enc-ninf "5a" "$(pv float_enc_ninf)" check_eq $P/float-enc-nan "5a" "$(pv float_enc_nan)" check_eq $P/float-inf-rt "None" "$(pv float_inf_rt)" check_eq $P/dec-float32 "1.5" "$(pv dec_float32)" check_eq $P/dec-float64 "1.5" "$(pv dec_float64)" # ------------------------------------------------------------ high precision -- # `H\' is a length-prefixed DECIMAL STRING, which is how UBJSON carries a number # no fixed width holds -- including the past-int64 integers above. It decodes to # a Decimal, so `hp-rt-type\' is what says the precision is actually preserved # rather than collapsed into a float on the way back. check_eq $P/hp-enc-decimal "485503312e35" "$(pv hp_enc_decimal)" check_eq $P/hp-enc-big "48551e313233343536373839303132333435363738393031323334353637383930" "$(pv hp_enc_big)" check_eq $P/hp-rt-decimal "Decimal('1.5')" "$(pv hp_rt_decimal)" check_eq $P/hp-rt-type "Decimal" "$(pv hp_rt_type)" check_eq $P/dec-high-prec "Decimal('1.5')" "$(pv dec_high_prec)" check_eq $P/hp-enc-nan "5a" "$(pv hp_enc_nan)" # --------------------------------------------------------- strings and chars -- # ONE BYTE IS A CHAR, TWO OR MORE IS A STRING. `str-enc-char\' is 2 bytes total # (`C\' + the byte, no length prefix) and `str-enc-two\' is 5. A reader expecting # `S\' for both would read the first as a bug, which is why it is pinned. # # The empty string takes the STRING form with a zero length, not the char form. # # U+00E9 is ONE codepoint and TWO utf-8 bytes, so it also takes the string form: # the length prefix counts BYTES, not characters. It is asserted as hex, never # printed as text -- the guest runs under the C locale and a suite that printed # U+00E9 would red on a stdout coercion rather than on ubjson. check_eq $P/str-enc-char "4361" "$(pv str_enc_char)" check_eq $P/str-enc-two "5355026162" "$(pv str_enc_two)" check_eq $P/str-enc-empty "535500" "$(pv str_enc_empty)" check_eq $P/str-enc-nonascii "535502c3a9" "$(pv str_enc_nonascii)" check_eq $P/str-rt-nonascii "c3a9" "$(pv str_rt_nonascii)" # THE 255/256 LENGTH BOUNDARY, which is the same width selection the integers # take: under 256 the length prefix is a one-byte `U\', at 256 it becomes `I\'. # Reported as head + total length rather than as 516 hex characters of `x\'. check_eq $P/str-enc-len255-head "5355ff" "$(pv str_enc_len255_head)" check_eq $P/str-enc-len255-len "258" "$(pv str_enc_len255_len)" check_eq $P/str-enc-len256-head "53490100" "$(pv str_enc_len256_head)" check_eq $P/str-enc-len256-len "260" "$(pv str_enc_len256_len)" check_eq $P/dec-char "'a'" "$(pv dec_char)" check_eq $P/dec-string "'ab'" "$(pv dec_string)" # -------------------------------------------------------------------- bytes -- # bytes take UBJSON\'s ONE optimised form this encoder ever emits: a strongly # typed uint8 array WITH a count, `[$U#\'. It is the only place `$\' appears in # encoder output at all -- a list never gets one, however uniform its contents. # # `bytes-rt-type\' is the case that matters: the typed uint8 array decodes back # to a bytes object rather than to a list of ints, so the type survives the # round-trip. no_bytes=True asks for the same payload as a list instead, which # is the documented way to get the other reading. check_eq $P/bytes-enc "5b2455235503010203" "$(pv bytes_enc)" check_eq $P/bytes-rt-type "bytes" "$(pv bytes_rt_type)" check_eq $P/bytes-rt "010203" "$(pv bytes_rt)" check_eq $P/bytes-enc-empty "5b2455235500" "$(pv bytes_enc_empty)" check_eq $P/bytes-dec-no-bytes "[1, 2, 3]" "$(pv bytes_dec_no_bytes)" # -------------------------------------------------- containers, plain and not -- # The plain forms are delimited (`[\' ... `]\', `{\' ... `}\') and the counted # forms replace the terminator with a `#\' count up front, which is what # container_count=True selects. Note the counted forms have NO terminator at # all: `arr-enc-count\' ends with the last value. check_eq $P/arr-enc "5b550155025d" "$(pv arr_enc)" check_eq $P/arr-enc-empty "5b5d" "$(pv arr_enc_empty)" check_eq $P/arr-enc-count "5b23550255015502" "$(pv arr_enc_count)" check_eq $P/arr-rt "[1, 2]" "$(pv arr_rt)" check_eq $P/obj-enc "7b55016155017d" "$(pv obj_enc)" check_eq $P/obj-enc-empty "7b7d" "$(pv obj_enc_empty)" check_eq $P/obj-enc-count "7b2355015501615501" "$(pv obj_enc_count)" check_eq $P/obj-rt "{'a': 1}" "$(pv obj_rt)" # sort_keys IS WHAT MAKES A MULTI-KEY OBJECT\'S BYTES DETERMINISTIC; without it # the encoder walks the dict in insertion order. Both are pinned, and they # differ, so a release that started sorting unconditionally would be caught -- # and every OTHER object case in this file uses a single key so that none of # them depends on this. check_eq $P/obj-enc-sorted "7b550161550255016255017d" "$(pv obj_enc_sorted)" check_eq $P/obj-enc-unsorted "7b550162550155016155027d" "$(pv obj_enc_unsorted)" # ------------------------------------------- THE OPTIMISED CONTAINER FORMS ---- # WHAT DISTINGUISHES UBJSON FROM A PLAIN JSON TRANSCODING, and -- apart from the # bytes case above -- forms this implementation only ever READS. So they are fed # to the decoder as LITERALS; there is no encoder output to compare them to. # # A typed array of a no-data type (`Z\', `T\') carries the count and NO payload # at all: `[$Z#U\x03\' is five bytes and decodes to three Nones. That is the # form UBJSON exists for, and the one a round-trip could never have reached. check_eq $P/opt-dec-typed-uint8 "b'\\x01\\x02\\x03'" "$(pv opt_dec_typed_uint8)" check_eq $P/opt-dec-typed-int16 "[1, 2]" "$(pv opt_dec_typed_int16)" check_eq $P/opt-dec-typed-null "[None, None, None]" "$(pv opt_dec_typed_null)" check_eq $P/opt-dec-typed-true "[True, True]" "$(pv opt_dec_typed_true)" check_eq $P/opt-dec-count-only "[1, 2]" "$(pv opt_dec_count_only)" check_eq $P/opt-dec-obj-count "{'a': 1}" "$(pv opt_dec_obj_count)" check_eq $P/opt-dec-typed-obj "{'a': 5}" "$(pv opt_dec_typed_obj)" # A TYPED ARRAY WITHOUT A COUNT IS REFUSED. The ticket\'s description lists it # as a form to decode; draft-12 requires `#\' whenever `$\' is present, and the # implementation raises. Pinned as the failure it is. check_eq $P/opt-typed-no-count "DecoderException: ('Container type without count (at byte 3)', 3)" "$(pv opt_typed_no_count)" # ------------------------------------------------------------ failure cases --- # EACH IS A DISTINCT MALFORMATION WITH A DISTINCT MESSAGE, so a decoder that # collapsed them into one generic error would be caught here and nowhere else. # The byte OFFSET is asserted along with the message: it is what a caller uses # to locate the damage in a stream, and an off-by-one in it is invisible to a # case that only checks the exception type. # # `err-count-gt-data\' is the declared-count-longer-than-the-data case the # ticket asks for: the decoder runs out of input mid-container and reports the # empty marker it read, rather than returning a short container. check_eq $P/err-truncated-int16 "DecoderException: ('Failed to unpack int16 (at byte 2)', 2)" "$(pv err_truncated_int16)" check_eq $P/err-unknown-marker "DecoderException: ('Invalid marker (at byte 1)', 1)" "$(pv err_unknown_marker)" check_eq $P/err-empty-input "DecoderException: ('Invalid marker (at byte 0)', 0)" "$(pv err_empty_input)" check_eq $P/err-count-gt-data "DecoderException: ('Invalid marker within array (at byte 6)', 6)" "$(pv err_count_gt_data)" check_eq $P/err-bad-container-type "DecoderException: ('Invalid container type (at byte 3)', 3)" "$(pv err_bad_container_type)" check_eq $P/err-bytes-too-short "DecoderException: ('Container bytes array too short (at byte 8)', 8)" "$(pv err_bytes_too_short)" check_eq $P/err-string-too-short "DecoderException: ('String too short (at byte 5)', 5)" "$(pv err_string_too_short)" check_eq $P/err-unterminated-array "DecoderException: ('Invalid marker within array (at byte 3)', 3)" "$(pv err_unterminated_array)" check_eq $P/err-unterminated-object "DecoderException: ('Invalid marker within object (at byte 4)', 4)" "$(pv err_unterminated_object)" check_eq $P/err-negative-count "DecoderException: ('Negative count/length unexpected (at byte 4)', 4)" "$(pv err_negative_count)" # THE ENCODER\'S REFUSALS, the other half of the pair. Note the third: a # circular reference raises a bare ValueError, NOT EncoderException, so a # caller catching only the module\'s own exception type would not catch it. # That is upstream\'s inconsistency and it is recorded, not corrected. check_eq $P/err-enc-unsupported "EncoderException: Cannot encode item of type " "$(pv err_enc_unsupported)" check_eq $P/err-enc-nonstring-key "EncoderException: Mapping keys can only be strings" "$(pv err_enc_nonstring_key)" check_eq $P/err-enc-circular "ValueError: Circular reference detected" "$(pv err_enc_circular)" # ------------------------------------------------------ the round-trip corpus -- # NINETEEN PAYLOADS THROUGH dumpb->loadb, COUNTED RATHER THAN MERELY COLLECTED. # # `rt-disagree\' alone would be satisfied by a corpus that never ran -- an empty # set of disagreements reads green vacuously, which is the idiom trap three # lanes of this Epic hit independently (BRINTOS-2081). So the count is asserted # too: `rt-total\' and `rt-agree\' both have to be 19 for this to pass, and a # probe that skipped the loop fails the count instead of passing the emptiness. check_eq $P/rt-total "19" "$(pv rt_total)" check_eq $P/rt-agree "19" "$(pv rt_agree)" check_eq $P/rt-disagree "" "$(pv rt_disagree)" # THE SAME CORPUS THROUGH THE FILE-OBJECT API. dump/load over a stream is a # different code path from dumpb/loadb over bytes, and this asserts the two # agree BYTE FOR BYTE as well as in the value they recover. Counted for the same # reason as above. check_eq $P/fs-total "19" "$(pv fs_total)" check_eq $P/fs-disagree "" "$(pv fs_disagree)" # Nesting exercises the recursive descent at a depth the guest\'s stack is not # in question at. See the header for why it is 16 and not 400. check_eq $P/nested16-rt "[[[[[[[[[[[[[[[[1]]]]]]]]]]]]]]]]" "$(pv nested16_rt)" check_eq $P/nested16-enc-head "5b5b5b5b" "$(pv nested16_enc_head)" # ------------------------------------------------------------ decoder hooks ---- # The three documented ways to change what an object literal decodes TO. # object_pairs_hook takes precedence over object_hook and preserves key order, # which is the only way a caller can see the wire order of a UBJSON object. check_eq $P/hook-object-pairs "[('a', 1), ('b', 2)]" "$(pv hook_object_pairs)" check_eq $P/hook-object "[('a', 1)]" "$(pv hook_object)" check_eq $P/hook-intern-keys "{'a': 1}" "$(pv hook_intern_keys)" # ---------------------------------------------------------------------- CLI --- # ubjson/__main__.py IS A REAL ENTRY POINT and the only way this package is # reachable as a program: there is no console script, so `python3 -m ubjson\' is # the whole of its command-line surface. These two cases are the last two of the # five interpreter starts. # # The converter round-trips json -> ubjson -> json. `from_json\' encodes with # sort_keys=True, so the emitted bytes are deterministic and are asserted # literally; `to_json\' then writes json with sorted keys, so the recovered # document is deterministic too. # # Both files live under $W, i.e. ${TMPDIR:-/tmp} -- tmpfs. Nothing here # redirects stdin from the rootfs, which is what ends a suite mid-run on the # virtiofs root (brintos/linux#73). printf '%s' '{"b":1,"a":[1,2]}' > "$W/in.json" check $P/cli-fromjson python3 -m ubjson fromjson "$W/in.json" "$W/out.ubj" check_eq $P/cli-fromjson-bytes "7b5501615b550155025d55016255017d" \ "$(od -An -tx1 -v "$W/out.ubj" 2>/dev/null | tr -d ' \n')" check $P/cli-tojson python3 -m ubjson tojson "$W/out.ubj" "$W/back.json" check_eq $P/cli-tojson-value '{"a":[1,2],"b":1}' "$(cat "$W/back.json" 2>/dev/null)" # Called with no arguments it prints its usage to STDERR and exits 1, which is # what says the module was reached AS A PROGRAM rather than merely imported. # # NOT check_out: that helper requires rc=0 AND the needle, and this command # exits 1 BY DESIGN, so check_out could never pass however right the text was. # The first stderr line is captured and compared instead. check_rc $P/cli-usage-rc 1 python3 -m ubjson check_eq $P/cli-usage-text \ "USAGE: ubjson (fromjson|tojson) (INFILE|-) [OUTFILE]" \ "$(python3 -m ubjson 2>&1 >/dev/null | head -n 1)" rm -rf "$W"