. "$BST_LIB" # BST-TIMEOUT: 1200 # THREE python3 starts, and they are the whole cost. # # Every other assertion below reads the filesystem or dpkg # # with shell builtins, or greps the output of one of those # # three. # # # # The three are: a bare `import' (the platform gate), the # # probe, which takes ONE start for forty-odd measurements, # # and one start under CRC32C_SW_MODE=none, which cannot be # # folded into the probe because the module reads that # # variable at IMPORT and a process can only import once. # # # # The work inside each start is a CRC over at most 512 # # bytes -- microseconds beside an interpreter start on this # # arm, which is tens of seconds before the first line of the # # program runs. # # # # 1200 is 400s per start, the same budget # # 99-python3-pyparsing gives its own three and roughly six # # times the ~60s a python3 start has measured at here. # # Nothing below is quadratic in anything: the largest input # # is 256 bytes and the bytewise loop is 256 one-byte calls. # # # # A suite-timeout red is a batch-wide cost, so if the three # # starts prove slower than that on target the RIGHT fix is # # this number, not fewer cases. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-crc32c 2.3 (meta-openembedded/meta-python # recipes-devtools/python/python3-crc32c_2.3.bb, `inherit pypi setuptools3', # PyPI sdist) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE DOES. # ============================================================================ # # One function: CRC-32C, the Castagnoli polynomial, which is what iSCSI, SCTP, # Btrfs and Google Cloud Storage checksum with. It is NOT zlib's crc32 -- same # width, different polynomial, different values for every input. # # The package ships TWO implementations of it and picks between them at BUILD # time: a hardware path over the SSE 4.2 and ARMv8 CRC instructions, and a # portable slice-by-8 software path. Neither instruction set exists on this arm, # so the software path is the one that must be selected -- and `hardware_based' # below is asserted rather than assumed, so a later toolchain change that # silently flipped it is visible as a red. # # ============================================================================ # THE PLATFORM GATE: THIS IS A PURE C EXTENSION, WITH NO PYTHON HALF. # ============================================================================ # # This is the load-bearing difference from every python3 port that has landed # here so far, and it is why `import' below is a gate rather than a formality. # # python3-pyparsing, -wcwidth, -smartypants and the rest are PURE PYTHON: they # declare no ext_modules and ship no `.so'. python3-protobuf ships a C # accelerator AND a documented pure-Python codec, so declining to build the # accelerator still leaves a working package -- which is what that port did. # # crc32c has no such half. `crc32c' IS `_crc32c.c'; the sdist contains no .py at # all. So the port either loads a side module or it delivers nothing, and there # is no fallback to fall back to. # # WHAT WAS MEASURED AT BUILD TIME, and it is more favourable than the recipe # comments elsewhere in this layer would predict: # # * The extension DOES link. `bitbake python3-crc32c' completed 1603 of 1603 # tasks with zero ERROR lines and packaged # /usr/lib/python3.12/site-packages/crc32c.cpython-312.so, 33686 bytes, a # genuine wasm side module carrying a `dylink.0' section and this arm's # `hwjs.abi' section. It is NOT the refusal python3-protobuf's `_upb' # extension got ("-shared side-module link received NATIVE wasm input(s)"): # crc32c's six translation units are plain C with no native inputs to # absorb, so they all arrive as bitcode and the link is accepted. # # * Its `dylink.0' declares NO NEEDED entries -- it depends on no other shared # library -- and its 55 wasm imports are the HWJS coroutine ABI, `getenv', # `strcmp', and TWELVE CPython API symbols: PyArg_ParseTuple, # PyBuffer_Release, PyErr_SetString, PyErr_WarnEx, PyExc_DeprecationWarning, # PyExc_ImportError, PyLong_FromUnsignedLong, PyModule_AddIntConstant, # PyModule_AddObject, PyModule_Create2, _Py_FalseStruct and PyInit_crc32c. # # * Those twelve are what the import turns on. They must resolve against the # MAIN module -- this image's python3, which is one static binary with an # empty lib-dynload because configure picks MODULE_BUILDTYPE=static from # host_cpu=wasm32 (BRINTOS-1476). The interpreter's own pyconfig-32.h does # define HAVE_DYNAMIC_LOADING, HAVE_DLOPEN and HAVE_DLFCN_H, so CPython will # ATTEMPT the dlopen rather than refusing the suffix outright; and the arm # has had a working side-module substrate since BRINTOS-1442, which stages # shared libraries content-addressed on the loader's needed search path and # is what lets six converted libraries bind at exec today. # # But that substrate resolves a NEEDED entry at exec. A Python extension is a # runtime dlopen(3) of an absolute path in site-packages -- a different # mechanism, against a main module that was linked whole-program -- and # nothing in this layer has exercised it before. WHETHER IT WORKS IS # GENUINELY OPEN, and this suite is the first thing that will answer it. # # SO THE SUITE IS BUILT TO BE INFORMATIVE EITHER WAY. `import' is one case. If it # fails, the probe reports `import_ok=no' with the exception, that one case reds # carrying the reason, and every behavioural case is SKIPPED with the same reason # attached rather than reded -- forty reds all naming one platform fact would # bury it rather than report it. If it succeeds, every behavioural case runs and # this port is complete with no attribution needed at all. # # ============================================================================ # WHERE THE EXPECTED VALUES COME FROM. # ============================================================================ # # The four checksums for `123456789', `23456789', `1234567890' and the fox # phrase are UPSTREAM 2.3's OWN published vectors, copied from its # test/test_crc32c.py. The empty-input 0 is upstream's `test_zero'. # # The all-256-byte fixture -- crc32c(bytes(range(256))) = 0x9c44184b -- is not # published anywhere, so it was computed from the CRC-32C definition (reflected # polynomial 0x82F63B78) by a reference implementation that was FIRST checked # against those four published vectors. It covers every byte value, which is what # reaches all eight slice-by-8 tables rather than only the first. # # MEASURED ON AN x86-64 CONTROL AT THE SAME VERSION -- crc32c 2.3 built from the # same sdist -- every expected value below is that control's OUTPUT rather than a # reading of the source. The control was run TWICE: once as built (hardware # path), and once under CRC32C_SW_MODE=force, which is the SOFTWARE path and # therefore the same code the target runs. Both produced byte-identical results # for every vector, every incremental split and every buffer type; the second # also reported `hardware_based=no', which is the shape the target must show. # # Two control results are worth naming because they are not what the source # reads like: # # * A NEGATIVE `crc' argument is ACCEPTED and wraps. The format is `I', which # is PyLong_AsUnsignedLongMask -- it masks rather than range-checks. Measured # `accepted', not an exception. # # * A STRIDED memoryview raises BufferError, not TypeError. `y*' asks for a # C-contiguous buffer and it is the buffer machinery that refuses, before # the argument parser has anything to say. # # The cases that could NOT be measured there are the packaging ones, whose paths # and control stanza differ on a Debian host by construction. P=python3-crc32c # ============================================================================ # The suite proper. # ============================================================================ # # `suite-canary' is emitted FIRST and unconditionally. A suite that dies before # its first result line is indistinguishable from one that was never scheduled, # so this one line is what separates "ran and found nothing" from "did not run". bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages SO=$SITE/crc32c.cpython-312.so DI=$SITE/crc32c-2.3.dist-info PKGLIST=/var/lib/dpkg/info/python3-crc32c.list PROBE=$BST_DATA/python3-crc32c-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-crc32c.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ---------------------------------------------------------------- packaging -- # WHERE THE PACKAGING SAYS THE MODULE LANDS. These run before any import, so a # red here says the .deb is wrong rather than that the module misbehaves -- which # matters more than usual for this port, since a missing .so and an unloadable # one both end at the same ImportError. check $P/site-packages-dir test -d "$SITE" check $P/extension-present test -f "$SO" check $P/extension-nonempty test -s "$SO" # IT IS AN EXTENSION AND NOTHING ELSE. The sdist has no .py, so a `crc32c.py' or # a `crc32c/' package directory here would mean a release that grew a # pure-Python half -- which would change what this port is and wants noticing. check $P/no-py-module test ! -f "$SITE/crc32c.py" check $P/no-py-package test ! -d "$SITE/crc32c" # THE SONAME CARRIES NO PLATFORM TRIPLE. Upstream names an extension # `crc32c.cpython-312-.so'; poky's python3 reports an empty # EXT_SUFFIX triple on this arm, so the file is plain `crc32c.cpython-312.so'. # The import resolves through importlib.machinery.EXTENSION_SUFFIXES, so the two # must agree -- and if a future poky restored the triple, this case reds before # the import does and says why. check_eq $P/extension-name "crc32c.cpython-312.so" \ "$(cd "$SITE" 2>/dev/null && echo crc32c.cpython-312*.so)" # NO EXECUTABLE. setup.py declares no console_scripts, so nothing lands in # /usr/bin. Asserted so a future release that started shipping one is noticed. check $P/no-bin-script test ! -e /usr/bin/crc32c # --------------------------------------------------------------- dist-info --- check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 2.3" cat "$DI/METADATA" check_out $P/dist-info-name "Name: crc32c" cat "$DI/METADATA" check $P/dist-info-wheel test -f "$DI/WHEEL" # NOT PURELIB, stated by the packaging itself. `Root-Is-Purelib: false' is the # wheel's own claim that it contains something architecture-specific -- the # opposite of what every pure-Python port here asserts, and the one-line summary # of why this port is different. check_out $P/dist-info-not-purelib "Root-Is-Purelib: false" cat "$DI/WHEEL" # THE WHEEL TAG IS WRONG, AND IT IS UPSTREAM SETUPTOOLS RATHER THAN THIS PORT. # bdist_wheel computes the tag from the BUILD HOST, and the cross build never # tells it otherwise, so the wheel says `cp312-cp312-linux_x86_64' for a wasm32 # module. Nothing reads it at runtime -- the .deb is what installs the file and # dpkg has the right architecture -- so this is recorded rather than repaired: # fixing it means patching the tag out of a metadata file no consumer on this # image opens. The case pins the current value so that a poky which started # setting the tag correctly is a visible change and not a silent one. check_out $P/dist-info-tag-is-host "Tag: cp312-cp312-linux_x86_64" cat "$DI/WHEEL" # The four LICENSE files the recipe's four-way LICENSE reflects. They are the # vendored implementations' own terms and their presence is the port's evidence # that the license declaration is not just a string in a recipe. for _l in LICENSE LICENSE.google-crc32c LICENSE.slice-by-8 AUTHORS.google-crc32c; do check $P/license-$_l test -s "$DI/$_l" done # ------------------------------------------------------------ the .deb file --- # The installed file list, read once into $W. The copy is what the loop below # redirects from: a stdin redirect the top-level shell performs on the virtiofs # root makes dup2(2) return 513 and ENDS the suite silently (brintos/linux#73), # and $W is on tmpfs. cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # EXACTLY ONE SHARED OBJECT SHIPS. Everything else in the package is dist-info # metadata. A second .so would mean the build produced something this port did # not expect and did not measure. check_eq $P/pkg-one-so "1" \ "$(grep -c '\.so$' "$W/pkgf" 2>/dev/null)" # Every payload path is under site-packages and is the .so or a dist-info file. _bad="" while IFS= read -r _f; do case "$_f" in ''|/usr/lib/python3.12/site-packages) continue ;; */) continue ;; esac [ -d "$_f" ] && continue case "$_f" in *.so|*.py|*.pyc) ;; */METADATA|*/WHEEL|*/RECORD|*/INSTALLER|*/LICENSE|*/LICENSE.*) ;; */AUTHORS.*|*/entry_points.txt|*/top_level.txt) ;; *) _bad="$_bad $_f" ;; esac done < "$W/pkgf" check_eq $P/pkg-only-expected "" "$_bad" # ------------------------------------------------------------- dpkg status --- sed -n '/^Package: python3-crc32c$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # THE DEPENDENCY SET IS THE PORT'S CLAIM THAT IT WAITS ON NOTHING. The recipe # carries no DEPENDS and RDEPENDS only python3-core, and the wasm shlibs pass # derived nothing further -- the extension's dylink.0 declares no NEEDED entry, # so there is no library dependency to derive. python3-core is therefore the # whole of it, and it is asserted by name: a control stanza that had lost it # would surface as an ImportError that looked like this port's fault. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-python3-core "python3-core" cat "$W/dep" # The package's architecture is this arm's, which is the half the wrong wheel tag # above might otherwise be read as contradicting. check_out $P/pkg-arch "Architecture: wasm32" cat "$W/st" # ------------------------------------------------------------------ import --- # THE PLATFORM GATE, and the first of three python3 starts. See the header: this # is a dlopen of a side module in site-packages whose twelve CPython API imports # must resolve against a statically linked main module, and nothing in this layer # has exercised that before. check $P/import python3 -c 'import crc32c' # ------------------------------------------------------------------- probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "probe missing at $PROBE" exit 0 fi bst_ok $P/probe-present # THE SECOND START, and the one that carries the suite. Bounded at 900s inside # the 1200s suite budget so a hung probe is reported as a probe overrun with the # last line it printed, rather than as a suite timeout that says nothing about # where it stopped. # # The probe EXITS 0 even when the import fails, so this case passing means "the # probe ran", not "the module works" -- `import-ok' below is what carries that. bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" # pv : the value the probe emitted for , or empty. `head -n 1' because # a key is emitted once and a duplicate would otherwise concatenate; `cut -f2-' # because a value may legitimately contain `='. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # The probe reached its end. Every case below reads a value it printed, so a # probe that stopped early would otherwise turn into a run of empty-string # mismatches that all name the wrong thing. check_eq $P/probe-complete "yes" "$(pv probe_done)" # ------------------------------------------------------- the gate's verdict --- # ONE case for the platform fact, carrying the exception if there is one. _imp=$(pv import_ok) check_eq $P/import-ok "yes" "$_imp" if [ "$_imp" != "yes" ]; then # The extension did not load. Skip -- not red -- every behavioural case, each # with the reason attached: they are all one fact, and forty reds would bury it # rather than report it. `import' and `import-ok' above are the two reds that # carry the verdict, and the design doc attributes them. _why="crc32c.cpython-312.so did not load: $(pv import_error)" for _c in module-answered is-extension dist-version no-version-attr \ hardware-based hardware-based-present hardware-based-bool \ big-endian-present big-endian-little has-crc32c has-crc32 \ vec-empty vec-numbers1 vec-numbers2 vec-numbers3 vec-phrase \ vec-allbytes inc-full inc-split-0 inc-split-1 inc-split-7 \ inc-split-8 inc-split-17 inc-split-128 inc-split-255 \ inc-split-256 inc-zero-chunk inc-bytewise buf-bytes \ buf-bytearray buf-memoryview buf-strided arg-str \ arg-negative-crc arg-none result-type result-in-range \ dep-value dep-warn-count dep-warn-class dep-agrees \ sw-mode-unset probe-count sw-mode-none; do bst_skip $P/$_c "$_why" done exit 0 fi check_eq $P/probe-count "46" "$(pv emitted_count)" # ---------------------------------------------------------------- identity --- # WHICH COPY ANSWERED. Without this the vectors below would be statements about # whatever was first on sys.path. check_eq $P/module-answered "$SO" "$(pv module_file)" check_eq $P/is-extension "yes" "$(pv is_extension)" check_eq $P/dist-version "2.3" "$(pv dist_version)" # The module adds only `hardware_based' and `big_endian' to its dict -- there is # no `__version__' -- so its absence is asserted rather than left looking like a # probe that forgot to read it. check_eq $P/no-version-attr "no" "$(pv has_version_attr)" # ------------------------------------------------ which implementation ran --- # THE TICKET'S OWN QUESTION. Two independent gates select the software path here # and `hardware_based' is where either one flipping becomes visible: # # setup.py lowercases the `platform' the recipe's do_compile:prepend writes # into setup.cfg -- ${TARGET_ARCH}, so `wasm32' -- and tests it against # ('aarch64','armv8b','armv8l','universal2') and # ('x86_64','amd64','i386','i686','universal2'). `wasm32' is in neither, so the # extra compile args are plain ['-O3']: no -msse4.2, no -march=armv8-a+crc. # Upstream 2.3 does NOT fail on an unrecognised platform -- it falls through to # the portable case, which is the answer to the ticket's first question. # # common.h then defines IS_INTEL only for x86/x86_64 and IS_ARM only for # __aarch64__/_M_ARM64, so on wasm32 the hardware branches of MOD_INIT are # preprocessed out entirely. This is the DECISIVE gate -- it would still select # software even if setup.cfg named an ARM platform. # # CONFIRMED IN THE BUILT ARTEFACT, not only in the source: the side module's wasm # imports name `_crc32c_sw_slicing_by_8' and the eight `crc_tableil8_o*' tables, # and name neither `_crc32c_hw_adler' nor `_crc32c_hw_arm64'. check_eq $P/hardware-based "no" "$(pv hardware_based)" check_eq $P/hardware-based-present "yes" "$(pv has_hardware_based)" check_eq $P/hardware-based-bool "bool" "$(pv hardware_based_type)" # wasm32 is little-endian. `big_endian' is computed at module init off the first # byte of a uint32 1, so this reads the RUNTIME's byte order rather than a # build-time assumption -- and the slice-by-8 tables are indexed differently on # a big-endian host, so a 1 here would invalidate every vector below. check_eq $P/big-endian-present "yes" "$(pv has_big_endian)" check_eq $P/big-endian-little "0" "$(pv big_endian)" check_eq $P/has-crc32c "yes" "$(pv has_crc32c)" check_eq $P/has-crc32 "yes" "$(pv has_crc32)" # ------------------------------------------------------------- the vectors --- # Upstream 2.3's own published values; the empty input is its `test_zero'. check_eq $P/vec-empty "00000000" "$(pv vec_empty)" check_eq $P/vec-numbers1 "e3069283" "$(pv vec_numbers1)" check_eq $P/vec-numbers2 "bfe92a83" "$(pv vec_numbers2)" check_eq $P/vec-numbers3 "f3dbd4fe" "$(pv vec_numbers3)" check_eq $P/vec-phrase "22620404" "$(pv vec_phrase)" # Every byte value, which is what reaches all eight slice-by-8 tables rather # than only the first. Computed from the polynomial, cross-checked against the # four published vectors above, and confirmed on the x86-64 control. check_eq $P/vec-allbytes "9c44184b" "$(pv vec_allbytes)" # ---------------------------------------------------------- incremental use --- # `crc32c(data, crc)' takes a PREVIOUS RESULT as `crc' -- MOD_INIT XORs # 0xffffffff in and out around the call, so a result feeds straight back with no # fixups by the caller. Every split must reproduce the one-shot value over the # same input; a slice-by-8 that mishandled a partial block would agree at some # offsets and not others, which is why eight are taken and not one. 7, 8 and 17 # straddle the eight-byte stride deliberately. check_eq $P/inc-full "9c44184b" "$(pv inc_full)" for _o in 0 1 7 8 17 128 255 256; do check_eq $P/inc-split-$_o "9c44184b" "$(pv inc_split_$_o)" done # A ZERO-LENGTH CHUNK mid-stream must be the identity. It is the case a length # check written as `if (length)' rather than as a loop bound gets wrong, and no # split above reaches it -- none of them feeds an empty slice through the second # call. check_eq $P/inc-zero-chunk "9c44184b" "$(pv inc_zero_chunk)" # Upstream's own `test_piece_by_piece', and the harshest form of the same claim: # 256 consecutive one-byte calls. check_eq $P/inc-bytewise "9c44184b" "$(pv inc_bytewise)" # --------------------------------------------------------- buffer protocol --- # `crc32c' parses with `y*' -- a Py_buffer fill, not a bytes copy -- so bytes, # bytearray and memoryview reach the same C code by three different paths. A # fresh extension build is where those diverge, so each is measured. check_eq $P/buf-bytes "9c44184b" "$(pv buf_bytes)" check_eq $P/buf-bytearray "9c44184b" "$(pv buf_bytearray)" check_eq $P/buf-memoryview "9c44184b" "$(pv buf_memoryview)" # A NON-CONTIGUOUS memoryview must be REFUSED rather than silently checksummed # over the wrong bytes. Getting a number here would be a wrong answer, not an # error. BufferError rather than TypeError, measured on the control: `y*' asks # for a C-contiguous buffer and the buffer machinery refuses before the argument # parser has anything to say. check_eq $P/buf-strided "BufferError" "$(pv buf_strided)" # -------------------------------------------------------- argument handling --- # `str' is refused under py3: the format is `y*', so text has no defined encoding # to checksum and TypeError is the documented answer. check_eq $P/arg-str "TypeError" "$(pv arg_str)" check_eq $P/arg-none "TypeError" "$(pv arg_none)" # A NEGATIVE `crc' is ACCEPTED and wraps -- `I' is PyLong_AsUnsignedLongMask, # which masks rather than range-checks. That is not what the format's name # suggests and it is not this platform's doing: measured `accepted' on the # x86-64 control at the same version. Pinned so the wrap is a documented # property rather than something a later reader rediscovers. check_eq $P/arg-negative-crc "accepted" "$(pv arg_negative_crc)" check_eq $P/result-type "int" "$(pv result_type)" check_eq $P/result-in-range "yes" "$(pv result_in_range)" # ------------------------------------------------------- deprecated alias --- # `crc32' is `crc32c' behind a DeprecationWarning. Both halves are asserted: a # release that dropped the warning and one that broke the alias are different # regressions. check_eq $P/dep-value "e3069283" "$(pv dep_value)" check_eq $P/dep-warn-count "1" "$(pv dep_warn_count)" check_eq $P/dep-warn-class "DeprecationWarning" "$(pv dep_warn_class)" check_eq $P/dep-agrees "yes" "$(pv dep_agrees)" # The probe's own start saw no CRC32C_SW_MODE, so the `hardware-based' reading # above is the UNFORCED default rather than a forced software mode. Without this # the strongest case in the suite would be ambiguous. check_eq $P/sw-mode-unset "" "$(pv sw_mode_env)" # -------------------------------------------------------------- SW_MODE=none --- # THE THIRD START, and it cannot be folded into the probe: the module reads # CRC32C_SW_MODE at IMPORT and a process imports once. # # `none' means "fail if no hardware support is found". On a machine WITH the # instructions it imports fine -- measured on the x86-64 control, which imported # and reported hardware_based=True. Here there is no hardware path compiled in at # all, so MOD_INIT must fall to its last branch and raise ImportError. # # This is the INVERTED case that proves `hardware-based=no' above is a real # reading rather than a default: if the hardware path ever became available on # this arm, this import would start succeeding and the case would red. check_out $P/sw-mode-none "ImportError" \ sh -c 'CRC32C_SW_MODE=none python3 -c "import crc32c" 2>&1; exit 0'