. "$BST_LIB" # BST-TIMEOUT: 1200 # ONE python3 start, and that is nearly the whole cost. # # Everything else reads the filesystem and dpkg with shell # # builtins or greps the single probe run's output. The # # probe's own work is six sha256 digests and two sha384 # # digests over eight small files -- the largest is the # # 282 kB unminified jQuery and the whole payload is under # # 400 kB -- plus two copies of a 87 kB file into a tmpdir. # # That is nothing beside the interpreter load. It matches # # 99-python3-alabaster's 1200 for the same reason: the # # start dominates. # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-sphinxcontrib-jquery 4.1 (poky/meta recipes-devtools/python/ # python3-sphinxcontrib-jquery_4.1.bb, `inherit pypi python_flit_core', NO # meta-brintos bbappend) on the wasm32-hwjs glibc image -- THE SPHINX EXTENSION # THAT PUTS jQUERY BACK. Sphinx bundled jQuery until 6.0 and stopped; this # extension re-registers it, together with a compatibility shim, for the themes # and doc trees that still expect it. # # ============================================================================ # THE ONE FACT THAT SHAPES THIS WHOLE SUITE: THE MODULE CANNOT BE IMPORTED HERE. # ============================================================================ # # sphinxcontrib/jquery/__init__.py line 4 is a plain `import sphinx' at MODULE # scope. This image does not install Sphinx -- python3-sphinx exists in poky # (recipes-devtools/python/python3-sphinx_7.2.6.bb) and is in neither # core-image-brintos-emcraft nor any port ticket. So on this rootfs # # python3 -c 'import sphinxcontrib.jquery' # # raises ModuleNotFoundError: No module named 'sphinx', and `import-needs-sphinx' # below asserts exactly that. THAT CASE IS GREEN ON PURPOSE. It is not a defect # being tolerated and it is not a platform gap: it is the measured, expected # state of an image that installs an extension without installing the # application the extension extends, and it is the case that will go RED -- and # so demand that this decision be revisited -- on the day python3-sphinx lands. # # What is asserted instead is the package's CONTRACT WITH SPHINX, against a stub # application object, which is the whole of what an extension is: the dict # setup() returns, the config value it declares, the event it connects to, and # the exact add_js_file() calls it then makes. None of that needs Sphinx to be # installed, and all of it is what would break in a port that had gone wrong. # See the probe's header for why a stub is the right instrument here and why it # is deliberately minimal. # # ============================================================================ # WHY THERE IS NO bbappend, AND WHY THAT IS A DECISION RATHER THAN AN OMISSION. # ============================================================================ # # The built .deb reads `Depends: python3-core' -- python_flit_core's base # contributes that and nothing else. Upstream is NOT silent about the shortfall # the way alabaster's pyproject.toml is: sphinxcontrib_jquery-4.1.dist-info/ # METADATA carries `Requires-Dist: Sphinx>=1.8', so the dependency OE dropped is # one upstream states. `pkg-depends-core' and `dist-requires' below are the two # halves of that disagreement, each measured on target. # # The obvious repair -- `RDEPENDS:${PN} += "python3-sphinx"' in a bbappend -- is # NOT taken here, and the reason is not that it would be wrong about the package # but that this ticket cannot pay for it. python3-alabaster's bbappend added # python3-pygments to a rootfs that ALREADY installed python3-pygments, so it # corrected the package and changed no image. Declaring python3-sphinx does the # opposite: it drags Sphinx and its entire dependency tree into the rootfs, and # several of those dependencies -- imagesize (BRINTOS-2105), snowballstemmer # (BRINTOS-2103) -- are being ported in this same Epic and are not yet in any # image. A dependency line that pulls unported code into a shared image is a # broken build for every package that shares it, not a correction. # # So the recipe is taken unmodified, the shortfall is recorded here and in the # image recipe rather than papered over, and `pkg-depends-no-sphinx' is the # tripwire: it asserts that the Depends line does NOT yet name python3-sphinx, # and it goes red together with `import-needs-sphinx' the moment Sphinx is # ported -- which is exactly when this decision should be reopened. # # ============================================================================ # WHY THIS PORT IS POSSIBLE AT ALL. # ============================================================================ # # This image's python3 is ONE STATIC BINARY with an EMPTY lib-dynload and no # dynamic loader (BRINTOS-1476; 99-python3 asserts both directly), and its # exported symbol table carries no Py* API at all (BRINTOS-2046). A module # carrying a compiled extension therefore could not be imported here -- not # "would be slow", could not be imported. This package is one .py file and three # JavaScript assets, which is the entire reason it ports, and BRINTOS-2046 does # not reach it. The wheel says so itself: `Root-Is-Purelib: true', tags # `py2-none-any' and `py3-none-any', generator `flit 3.9.0'. # # So `pkg-no-so' is not boilerplate. It is a REGRESSION TRIPWIRE on a version # bump, and it goes red at INSTALL time rather than at somebody else's import # three rounds later. # # ============================================================================ # EVERY VALUE BELOW WAS MEASURED, NOT PREDICTED. # ============================================================================ # # The expectations come from the tree unpacked from the .deb this recipe built # (python3-sphinxcontrib-jquery_4.1-r0_wasm32.deb), read on the x86-64 control # with the probe below run against it verbatim. Nothing here is arm-dependent: # the wheel is `py3-none-any', the digests are the wheel's and the package's # own, and the single .py file is the same bytes on both machines. The paths are # the only substitution, and they are built from $SITE rather than written out. # # The runner's read drops the first line a suite emits, per-suite and # deterministic -- not a defect in the host tool (BRINTOS-1149, filed as # linux-in-the-browser#177). The harness computes its pass count as # total-minus-failures, so THE DROPPED LINE IS COUNTED AS A PASS. A nothing-case # stands in that position so the eaten line costs nothing. It sits ahead of the # `require' guard deliberately, because a `require' that skips would emit its # skip as the first line and be eaten too. # # Remove this case when linux-in-the-browser#177 is fixed. P=python3-sphinxcontrib-jquery bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages NSDIR=$SITE/sphinxcontrib PKGDIR=$NSDIR/jquery DI=$SITE/sphinxcontrib_jquery-4.1.dist-info PKGLIST=/var/lib/dpkg/info/python3-sphinxcontrib-jquery.list PROBE=$BST_DATA/python3-sphinxcontrib-jquery-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-scjquery.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------- the payload --- # Where the package has to land. The probe's `module-answered' further down is # what says the interpreter agreed; these say what the filesystem holds, so a # package that imported out of some other path on sys.path is distinguishable # from one that imported out of what this image installed. check $P/site-packages-dir test -d "$SITE" # THE NAMESPACE DIRECTORY, AND THE FILE THAT MUST NOT BE IN IT. `sphinxcontrib' # is a PEP 420 implicit namespace package: it is a directory with NO # __init__.py, which is how the whole sphinxcontrib-* family -- jsmath, # serializinghtml, applehelp, devhelp, htmlhelp and qthelp, every one of them a # sibling ticket in this Epic -- shares one top-level name without any single # distribution owning it. A release that started shipping sphinxcontrib/ # __init__.py would shadow all of them, and would pass every other case here. check $P/namespace-dir test -d "$NSDIR" check $P/namespace-no-init test ! -f "$NSDIR/__init__.py" check $P/package-dir test -d "$PKGDIR" # THE ONE SOURCE FILE. Everything that executes in this package is in it: # setup(), which Sphinx calls to register the extension, and add_js_files(), # which setup() connects to `config-inited'. check $P/src-init test -s "$PKGDIR/__init__.py" # THE THREE JAVASCRIPT ASSETS, each by name. jquery.js is the minified jQuery # 3.6.0 the extension registers; _sphinx_javascript_frameworks_compat.js is the # shim Sphinx used to ship that re-adds jQuery.urldecode and friends; and # jquery-3.6.0.js is the UNMINIFIED source of the same release, carried for # provenance and registered by nothing. The probe's `full-jquery-*' cases state # that last relationship, so a reader does not find it "missing" from the # registration and go looking for a bug. check $P/asset-jquery-min test -s "$PKGDIR/jquery.js" check $P/asset-jquery-full test -s "$PKGDIR/jquery-3.6.0.js" check $P/asset-compat-shim test -s "$PKGDIR/_sphinx_javascript_frameworks_compat.js" # The precompiled bytecode. OE byte-compiles the tree with compileall, which # does NOT import -- which matters here more than usual, since importing this # module needs a Sphinx that is not on the image. So the .pyc exists despite # the module being unimportable, and that is correct rather than surprising. check $P/pycache-init test -s "$PKGDIR/__pycache__/__init__.cpython-312.pyc" # NO COMPILED EXTENSION. See the header -- this is the regression tripwire, not # boilerplate. # # The list is copied to tmpfs first and never redirected from the rootfs: a # top-level stdin redirect on the virtiofs root makes dup2(2) return 513 and ash # EXITS, ending the suite silently (brintos/linux#73). cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # NOT ONE FILE IN THIS PACKAGE IS EXECUTABLE, AND NONE IS UNDER /usr/bin. A # Sphinx extension installs no program. The case above names the failure that is # expected; this one catches the shape nobody predicted. check_eq $P/pkg-no-bin "" \ "$(grep -E '^/usr/(bin|sbin)/' "$W/pkgf" 2>/dev/null)" # EVERY REGULAR FILE THE PACKAGE SHIPS IS PYTHON, JAVASCRIPT OR ITS METADATA -- # without having to enumerate what an unexpected one might be. dpkg's .list # marks no directory with a trailing slash, so the entries are sieved by asking # the filesystem what each one is. _bad= while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc|*.js) continue ;; */sphinxcontrib_jquery-4.1.dist-info/METADATA) continue ;; */sphinxcontrib_jquery-4.1.dist-info/RECORD) continue ;; */sphinxcontrib_jquery-4.1.dist-info/WHEEL) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-expected "" "$_bad" # ------------------------------------------------------------ the dist-info -- # The installed distribution metadata. The directory name carries the version, # so a mixed install -- new files beside an old dist-info -- shows up as the two # disagreeing rather than as neither being wrong. check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -s "$DI/METADATA" check $P/dist-info-record test -s "$DI/RECORD" check $P/dist-info-wheel test -s "$DI/WHEEL" # NO entry_points.txt, AND ITS ABSENCE IS THE DESIGN. A Sphinx THEME is # discovered through the `sphinx.html_themes' entry-point group -- python3- # alabaster ships one for exactly that. An EXTENSION is not discovered at all: # the user names it in conf.py's `extensions' list and Sphinx imports it. So # there is nothing to declare, and a file here would mean the package had # changed shape. check $P/no-entry-points test ! -f "$DI/entry_points.txt" # PURE PYTHON, AS THE WHEEL ITSELF DECLARES IT. `Root-Is-Purelib: true' is the # packaging-side statement of the header's claim -- no platform tag, no # extension module, nothing that could need the absent dynamic loader. Both tags # are asserted: this wheel is universal, py2 as well as py3, which is unusual # enough in 2024 that a release dropping the py2 tag is worth seeing. check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag-py3 "Tag: py3-none-any" cat "$DI/WHEEL" check_out $P/dist-info-tag-py2 "Tag: py2-none-any" cat "$DI/WHEEL" check_out $P/dist-info-generator "Generator: flit 3.9.0" cat "$DI/WHEEL" # The distribution name is HYPHENATED and the import name is DOTTED -- # `sphinxcontrib-jquery' installs `sphinxcontrib.jquery'. flit writes no # top_level.txt, so METADATA's Name is the only packaging-side statement of the # name, and it is not the name anyone imports. check_out $P/dist-info-name "Name: sphinxcontrib-jquery" cat "$DI/METADATA" check_out $P/dist-info-version "Version: 4.1" cat "$DI/METADATA" # THE DEPENDENCY UPSTREAM DECLARES AND OE DROPS, read as text. The probe reads # the same fact through importlib.metadata; both are here because they fail # differently -- a corrupt file reds here, and a metadata cache that cannot see # a perfectly good file reds there. See the header for why the .deb is left # disagreeing with this line. check_out $P/dist-info-requires-sphinx "Requires-Dist: Sphinx>=1.8" cat "$DI/METADATA" # ------------------------------------------------------------- the package --- # dpkg KNOWS the package. Every case above would look identical if some other # package had installed the same files, so this is the one that says # python3-sphinxcontrib-jquery is a real package this image installed rather # than an IMAGE_INSTALL line that resolved to nothing. # # The stanza is read from a tmpfs copy for the same reason as the file list, and # the anchor is exact -- an unanchored match would also take the -dev or -dbg # stanza of the same family. sed -n '/^Package: python3-sphinxcontrib-jquery$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" check_out $P/pkg-version "Version: 4.1-r0" cat "$W/st" check_out $P/pkg-arch "Architecture: wasm32" cat "$W/st" # THE RUNTIME DEPENDENCIES, as dpkg recorded them. python3-core is what # python_flit_core's base contributes and it is the whole of the line. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-core "python3-core" cat "$W/dep" # ...AND THE DEPENDENCY THAT IS NOT ON IT. This is the tripwire the header # describes: upstream's METADATA says `Requires-Dist: Sphinx>=1.8', the module # imports sphinx at module scope, and the .deb declares neither. Asserting the # ABSENCE keeps the disagreement visible and dated instead of leaving it to a # comment, and makes the day python3-sphinx is ported the day this case goes red # and the bbappend decision is reopened. See the header for why declaring it now # would break the shared image rather than fix the package. check_eq $P/pkg-depends-no-sphinx "" \ "$(grep -E 'python3-sphinx([ ,]|$)' "$W/dep" 2>/dev/null)" # THE IMAGE TOOK THE RUNTIME PACKAGE ALONE. The recipe BBCLASSEXTENDs to native # and nativesdk and OE splits a -dev and a -dbg from every recipe; none of the # three belongs on a target rootfs. check_eq $P/no-dev-package "" \ "$(sed -n '/^Package: python3-sphinxcontrib-jquery-dev$/p' /var/lib/dpkg/status 2>/dev/null)" check_eq $P/no-dbg-package "" \ "$(sed -n '/^Package: python3-sphinxcontrib-jquery-dbg$/p' /var/lib/dpkg/status 2>/dev/null)" # --------------------------------------------------------- the interpreter --- # THE IMPORT FAILS, AND THAT IS THE MEASURED STATE OF THIS IMAGE, NOT A DEFECT. # See the header. `import sphinx' at module scope, no python3-sphinx in the # rootfs, so the interpreter raises ModuleNotFoundError and exits 1. This case # is what pins the fact; the probe's `bare-import' names the missing module, so # a DIFFERENT import failure -- a genuinely broken package -- is distinguishable # from this one rather than hiding behind the same red. check_rc $P/import-needs-sphinx 1 python3 -c 'import sphinxcontrib.jquery' # ...AND SPHINX ITSELF IS WHAT IS ABSENT, asserted separately so the cause is # stated rather than inferred. A red HERE and a pass above would mean Sphinx had # arrived, which is the event that reopens the bbappend decision. check_rc $P/sphinx-absent 1 python3 -c 'import sphinx' # THE NAMESPACE PACKAGE ITSELF IMPORTS, THOUGH -- it is a directory with no code # in it, so nothing it does can fail. This is what separates "the submodule # needs Sphinx" from "sphinxcontrib is not installed at all", which would look # the same from the case above. check $P/import-namespace python3 -c 'import sphinxcontrib' # ---------------------------------------------------------------- the probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present # ONE interpreter start for everything that follows. bst_probe bounds it and # keeps BOTH streams: a probe that dies early otherwise leaves every case below # reading an empty file with no traceback to say why. bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" # Read a single key. The probe emits each key at most once, so head -n 1 is # defensive rather than load-bearing; the anchors keep `version' distinguishable # from `version_info' and `dist_version'. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE EVIDENCE PLANE. If the probe emitted nothing, every case below is reading # an absent file: each would red with got:'' and the set would read exactly like # a broken package, when what actually happened is that no measurement was taken. # The word `unscorable' is PARSED -- brintos_local_verify re-measures a suite that # prints it on a fresh guest rather than trusting a shared boot's numbers # (BRINTOS-1644) -- so it goes in this structural check, not in a comment. # # The cases below still print their reds: withdraw, do not hide. if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # THE BARE IMPORT, MEASURED INSIDE THE PROBE TOO, and naming the module that is # missing. `check_rc' above says the import exits 1; this says WHY, and it is the # case that would distinguish a package broken some other way -- a SyntaxError, # a missing file -- from the absent Sphinx this image is expected to have. check_eq $P/bare-import "ModuleNotFoundError:sphinx" "$(pv bare_import)" # THE VERSION THE RECIPE PINS, three times and independently: the package's own # `__version__', its `version_info' tuple, and importlib.metadata over the # installed dist-info. Two of these disagreeing means a mixed install. check_eq $P/version "4.1" "$(pv version)" check_eq $P/version-info "4.1" "$(pv version_info)" check_eq $P/dist-version "4.1" "$(pv dist_version)" check_eq $P/dist-name "sphinxcontrib-jquery" "$(pv dist_name)" # The package answered out of the files this image installed, not from a stray # copy on sys.path. This is what makes the payload cases above and the metadata # cases below statements about the SAME package. check_eq $P/module-answered "$PKGDIR/__init__.py" "$(pv module_file)" check_eq $P/root-dir "$PKGDIR" "$(pv root_dir)" # Everything it loaded is .py or .pyc -- the interpreter's own view of the # header's claim, complementing `pkg-no-so', which is the packaging view. check_eq $P/all-pure-python "yes" "$(pv all_pure_python)" # ------------------------------------------------------ the metadata plane --- # UPSTREAM DECLARES SPHINX. Read through importlib.metadata rather than grepped, # so what is asserted is what the interpreter resolves out of the installed # dist-info -- the other half of `dist-info-requires-sphinx' above, and the # statement that makes `pkg-depends-no-sphinx' a disagreement rather than an # opinion. check_eq $P/dist-requires "Sphinx>=1.8" "$(pv dist_requires)" # AND DECLARES NO ENTRY POINTS. See `no-entry-points' above: an extension is # imported by name from conf.py, never discovered. The zero is asserted because # a non-zero would mean the package had changed shape. check_eq $P/ep-count "0" "$(pv ep_count)" # ------------------------------------------------- the namespace package --- # THE PEP 420 FACTS, from the interpreter rather than the filesystem. # `sphinxcontrib' has __file__ None and a _NamespacePath for __path__ -- that is # what an implicit namespace package IS, and it is the mechanism every sibling # sphinxcontrib-* port in this Epic depends on. `namespace-no-init' above says # no __init__.py is on disk; these say the interpreter agreed and built a # namespace rather than a regular package. check_eq $P/ns-file "None" "$(pv ns_file)" check_eq $P/ns-path-type "_NamespacePath" "$(pv ns_path_type)" check_eq $P/ns-path "$NSDIR" "$(pv ns_path)" # ------------------------------------------------------------------ the API --- # THE PUBLIC SURFACE, sorted and read off the module -- it declares no __all__. # `makedirs', `path', `shutil' and `sphinx' are in it and that is not a mistake # in this expectation: __init__.py does `from os import makedirs, path', `import # shutil' and `import sphinx' at module scope, so each name is a module # attribute like any other. They are stated rather than filtered out, because a # release that changed its import form would change the surface and this is the # case that would say so. `sphinx' being visible here is also the plainest # possible statement of why the bare import fails. check_eq $P/api-surface \ "add_js_files,makedirs,path,setup,shutil,sphinx,version_info" \ "$(pv api_surface)" # The two functions the package defines, plus `makedirs', which is a callable it # imported. setup() is what Sphinx calls to register the extension; # add_js_files() is what setup() connects to `config-inited'. check_eq $P/api-callables "add_js_files,makedirs,setup" "$(pv api_callables)" # ------------------------------------------------- the registered assets --- # _FILES IS THE PACKAGE'S OWN LIST OF WHAT IT REGISTERS: two entries, in the # order they are registered, which is the order the