. "$BST_LIB" # BST-TIMEOUT: 1200 # THREE python3 starts, and they are the whole cost. # # Every other assertion below reads the filesystem or dpkg # # with shell builtins, or greps the output of one of those # # three. # # # # The three are: a bare `import', a second import whose # # STDERR is the assertion (`import-quiet'), and the probe, # # which takes ONE start for fifty-odd measurements. # # # # There is NO fourth for a shipped script. Unlike # # python3-smartypants, this package installs no executable # # at all -- flit_core builds it from a `[project]' table # # with no `[project.scripts]' -- so there is nothing to # # reach through a shebang and nothing that could be folded # # into the probe in the first place. `no-bin-script' below # # asserts that absence rather than leaving it unstated. # # # # The work inside each start is a handful of grammars built # # and run over strings under 100 bytes -- microseconds # # beside an interpreter start on this arm, which is tens of # # seconds before the first line of the program runs. # # # # 1200 is 400s per start, matching 99-python3-wcwidth's # # budget for its own three and roughly six times the ~60s a # # python3 start has measured at on this arm. Nothing here is # # quadratic in anything: the packrat table is ten # # expressions of under ten tokens each. # # # # A suite-timeout red is a batch-wide cost, so if the three # # starts prove slower than that on target the RIGHT fix is # # this number, not fewer cases. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-pyparsing 3.1.1 (poky/meta # recipes-devtools/python/python3-pyparsing_3.1.1.bb, `inherit pypi # python_flit_core', PyPI sdist) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE DOES. # ============================================================================ # # pyparsing builds a grammar out of PYTHON EXPRESSIONS rather than out of a # separate grammar file: `Word(alphas) + Suppress('=') + quoted' is the parser, # constructed by operator overloading, and there is no generator step and no # build-time artefact. Results come back as a `ParseResults' -- indexable like a # list AND addressable by name, which is the half a purely positional test never # reaches and which `kv-name-*' and `kv-as-dict' below exercise on purpose. # # ============================================================================ # THE THREE GRAMMARS, AND WHY EACH ONE IS HERE. # ============================================================================ # # ARITHMETIC, via `infix_notation' -- precedence and associativity. This is the # package's most-used helper and the one whose output shape surprises people: # # A RIGHT-associative run NESTS. `2^3^2' gives ['2','^',['3','^','2']], and the # tree states unambiguously that the right-hand `^' binds first. # # A LEFT-associative run DOES NOT. `a-b-c' comes back FLAT -- # ['a','-','b','-','c'] -- and NOT as [['a','-','b'],'-','c']. pyparsing groups # a whole same-precedence left-associative run into ONE group and leaves the # folding to the caller. For a LEFT level the tree therefore does not encode # the associativity at all, and a consumer walking it as a binary tree would # read `a-b-c' as a single three-operand node. That is upstream 3.1.1's # documented behaviour, it is not this platform's, and `arith-*-flat' records # it so a reader who inferred a left-leaning tree from the words # "left associative" is corrected here rather than after shipping. # # PRECEDENCE, unlike associativity, IS visible in the shape, because it nests # the levels against each other -- `1+2*3' and `1*2+3' each put the `*' in the # inner group. # # A KEY-VALUE CONFIGURATION grammar -- `Word', `QuotedString', `Suppress', # `Group', `delimited_list' and `set_results_name', asserted BOTH positionally # and by name, with `as_dict()' compared against a literal. Two of its findings # are the ones callers trip over: `QuotedString("\"")' STRIPS the quotes, and # `Word(nums)' returns a STRING -- `8080' is text, not an int, because Word # matches characters and converts nothing. # # A NESTED-LIST grammar -- whitespace and comment skipping, `Optional' with a # default, `OneOrMore' against `ZeroOrMore', and `Forward' for the recursion. # `Forward' is what makes a recursive grammar expressible at all here: the name # is bound before its definition exists and filled in afterwards with `<<='. # `ignore(python_style_comment)' is attached to the TOP element and applies # throughout the recursion, which is why the fixture puts its comments INSIDE # the nesting; `nested-comments-transparent' asserts that skipping removed the # comments and changed nothing else. # # ============================================================================ # THE FAILURE PATH IS A FIRST-CLASS CASE, AND IT IS MULTI-LINE ON PURPOSE. # ============================================================================ # # Input the grammar does not admit raises `ParseException', and the exception # must say WHERE. A parser that rejected bad input without pointing at it would # be useless for the diagnostics every caller builds on top of it. # # The fixture spans three lines deliberately. On a single-line input `lineno' is # always 1 and `line' is the whole input, so the fields cannot be told apart and # an implementation that returned the ENTIRE text as `line' would still pass. # Here the error is on the third line, so `err-lineno' must be 3, `err-line' must # be that line ALONE, and `err-col' must be the column within it -- while # `err-loc' stays an offset into the WHOLE string. Confusing those two coordinate # systems is the standing mistake, so they are asserted together. # # `err-loc-char' is `' rather than a character, and that is the finding # rather than a gap in it: the unclosed `(' means the parser consumed the entire # input and still wanted a `)', so the offset lands exactly at end of input. # # The SECOND failure (`err2-*') is reached differently -- the grammar matched a # shorter prefix perfectly well, and what rejects the input is `parse_all=True' # requiring nothing be left over. `partial-without-parse-all' runs the same input # WITHOUT that flag and asserts the silent partial parse it returns, because a # caller who omits it gets no error at all and that is the other standing # mistake. # # ============================================================================ # PACKRAT MUST CHANGE THE RESULTS NOT AT ALL. # ============================================================================ # # `ParserElement.enable_packrat()' turns on a per-(element, location) memo cache. # The whole of its contract is that it is INVISIBLE in the output, so the probe # runs the arithmetic table twice -- once plain, once memoised -- and # `packrat-identical' asserts the two agree tree for tree. This is a live # assertion and not a formality: a grammar whose parse actions carry state # between calls is a real and well-known way for memoisation to change answers. # # The switch is CLASS-LEVEL and ONE-WAY -- there is no `disable_packrat' -- which # is why the probe runs it last and why `packrat-before' is asserted False before # it is touched. # # ============================================================================ # WHY THIS PORT NEEDED NO BBAPPEND AND NO PATCH. # ============================================================================ # # PURE PYTHON, and this image's python3 is ONE STATIC BINARY with an empty # lib-dynload and no dynamic loader (BRINTOS-1476), so a module carrying a # compiled extension could not be imported here at all. pyparsing has no # ext_modules and ships no `.so'; `pkg-no-so' asserts that off the installed file # list rather than trusting the source. # # Its RDEPENDS name only sub-packages of the ported python3 -- datetime, # debugger, html, json, netclient, pprint, stringold, threading -- so this port # waits on no other package. That set was checked against what the source # actually imports and it is complete. The one apparent third-party import, # `import regex' in `core.py', is inside a DOCSTRING and is never executed; # `import' below is the case that would catch it if it were not. # # `pyparsing.diagram' IS SHIPPED AND CANNOT BE IMPORTED, and that is correct # rather than broken: the subpackage renders a grammar as a railroad diagram and # needs the `railroad' distribution, which is not a dependency of this recipe and # is not in the image. `diagram-import' asserts the ModuleNotFoundError and names # the missing module, so the failure is recorded as an optional extra declining # to load rather than being mistaken for a gap in the port. P=python3-pyparsing # ============================================================================ # The suite proper. # ============================================================================ # # `suite-canary' is emitted FIRST and unconditionally. A suite that dies before # its first result line is indistinguishable from one that was never scheduled, # so this one line is what separates "ran and found nothing" from "did not run". bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/pyparsing DI=$SITE/pyparsing-3.1.1.dist-info PKGLIST=/var/lib/dpkg/info/python3-pyparsing.list PROBE=$BST_DATA/python3-pyparsing-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-pyparsing.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ---------------------------------------------------------------- packaging -- # WHERE THE PACKAGING SAYS THE MODULE LANDS. These run before any import, so a # red here says the .deb is wrong rather than that the module misbehaves. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" check $P/package-init test -s "$PKG/__init__.py" # It is a PACKAGE, not a single module -- the opposite shape from # python3-smartypants' one file. A release that flattened it would still import # and still pass every behavioural case, so the shape is asserted directly. check $P/no-flat-module test ! -f "$SITE/pyparsing.py" # The submodules the public names are re-exported from. `__init__.py' does # `from .core import *', so a truncated install fails at import rather than here # -- naming them individually is what turns "the package is complete" into a # measurement the packaging can be judged by. for _m in actions common core exceptions helpers results testing unicode util; do check $P/module-$_m test -s "$PKG/$_m.py" done # The typing marker, which flit ships as package data. Its absence would not # break a single runtime case, which is exactly why it needs its own assertion. check $P/py-typed test -f "$PKG/py.typed" # The diagram subpackage SHIPS even though it cannot import -- see the header. check $P/diagram-dir test -d "$PKG/diagram" check $P/diagram-init test -s "$PKG/diagram/__init__.py" # NO EXECUTABLE. flit builds this from a `[project]' table with no # `[project.scripts]', so nothing lands in /usr/bin. Asserted so that a future # release which started shipping one is noticed here. check $P/no-bin-script test ! -e /usr/bin/pyparsing # --------------------------------------------------------------- dist-info --- # python_flit_core writes METADATA and WHEEL; both are asserted, and the version # is read from METADATA rather than from the module so that a mismatch between # what the packaging claims and what imports is visible as a mismatch. check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 3.1.1" cat "$DI/METADATA" check_out $P/dist-info-name "Name: pyparsing" cat "$DI/METADATA" check $P/dist-info-wheel test -f "$DI/WHEEL" # PURE PYTHON, stated by the packaging itself. `Root-Is-Purelib: true' is the # wheel's own claim that nothing in it is architecture-specific, which is the # property that makes this package portable to this arm at all. check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-flit "Generator: flit" cat "$DI/WHEEL" # ------------------------------------------------------------ the .deb file --- # The installed file list, read once into $W. The copy is what the loop below # redirects from: a stdin redirect the top-level shell performs on the virtiofs # root makes dup2(2) return 513 and ENDS the suite silently (brintos/linux#73), # and $W is on tmpfs. cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # NO COMPILED EXTENSION, asserted off the shipped file list rather than off the # source. This image's python3 has an empty lib-dynload and no dynamic loader # (BRINTOS-1476), so a single .so here would mean a package that installs # cleanly and cannot import. check_eq $P/pkg-no-so "" \ "$(grep -c '\.so$' "$W/pkgf" 2>/dev/null | grep -v '^0$')" # Every payload path the package ships is under site-packages, and is a .py, a # .pyc, or the typing marker. Anything else is a file this port did not expect # to be shipping and wants looking at. _bad="" while IFS= read -r _f; do case "$_f" in ''|/usr/lib/python3.12/site-packages) continue ;; */) continue ;; esac [ -d "$_f" ] && continue case "$_f" in *.py|*.pyc|*/py.typed) ;; # dist-info metadata. Which of these flit_core writes is a property of the # BUILD BACKEND rather than of this package, so the whole standard set is # allowed: pinning the exact list would red on a flit version change that # says nothing about pyparsing. `pkg-no-so' above is the case that carries # the real claim -- nothing compiled ships -- and it is exact. */METADATA|*/WHEEL|*/RECORD|*/INSTALLER|*/LICENSE|*/entry_points.txt|*/top_level.txt) ;; *) _bad="$_bad $_f" ;; esac done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" # ------------------------------------------------------------- dpkg status --- sed -n '/^Package: python3-pyparsing$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # THE DEPENDENCY SET IS THE PORT'S CLAIM THAT IT WAITS ON NOTHING. Every name # here is a sub-package of the ported python3 (BRINTOS-1476), so each is asserted # individually -- a bare "the Depends line exists" would pass on a control stanza # that had lost one of them, and the loss would surface as an ImportError on # whichever grammar first reached that stdlib module. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" for _d in python3-core python3-datetime python3-debugger python3-html \ python3-json python3-netclient python3-pprint python3-stringold \ python3-threading; do check_out $P/pkg-depends-$_d "$_d" cat "$W/dep" done # ------------------------------------------------------------------ import --- # THE FIRST OF THREE PYTHON STARTS. If the RDEPENDS set above were short by one # sub-package, this is the case that would say so. check $P/import python3 -c 'import pyparsing' # THE SECOND. A package that imports but WARNS on the way in is a different # defect from one that fails, and it is invisible to the case above because the # exit status is 0 either way. The assertion is that stderr is EMPTY -- pyparsing # 3.1.1 issues deprecation warnings for several of its own compatibility names, # and a bare import must trigger none of them. check_eq $P/import-quiet "" \ "$(python3 -c 'import pyparsing' 2>&1 >/dev/null)" # ------------------------------------------------------------------- probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "probe missing at $PROBE" exit 0 fi bst_ok $P/probe-present # THE THIRD START, and the one that carries the suite. Bounded at 900s inside the # 1200s suite budget so that a hung probe is reported as a probe overrun with the # last line it printed, rather than as a suite timeout that says nothing about # where it stopped. bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" # pv : the value the probe emitted for , or empty. `head -n 1' because # a key is emitted once and a duplicate would otherwise concatenate; `cut -f2-' # because several values legitimately contain `='. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # The probe reached its end. Every case below reads a value the probe printed, so # a probe that stopped early would otherwise turn into a run of empty-string # mismatches that all name the wrong thing. check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "57" "$(pv emitted_count)" # ---------------------------------------------------------------- identity --- # WHICH COPY ANSWERED. Without this the behavioural cases below would be # statements about whatever was first on sys.path. check_eq $P/module-answered "$PKG/__init__.py" "$(pv module_file)" check_eq $P/is-package "yes" "$(pv is_package)" check_eq $P/version-attr "3.1.1" "$(pv version_attr)" check_eq $P/dist-version "3.1.1" "$(pv dist_version)" # The two are read by different mechanisms -- an attribute on the module and # importlib.metadata off the dist-info -- so their agreement is a real assertion # about the install and not a tautology. check_eq $P/version-agrees "$(pv version_attr)" "$(pv dist_version)" for _s in core results helpers exceptions; do check_eq $P/submodule-$_s "yes" "$(pv submodule_$_s)" done # The optional extra that is EXPECTED to fail, naming the module it wants. See # the header: this is an attributed non-defect, and asserting the exact # ModuleNotFoundError means an image that gained railroad-diagrams reds here and # gets looked at rather than silently changing the suite's meaning. check_eq $P/diagram-import "ImportError:railroad" "$(pv diagram_import)" # ------------------------------------------------- grammar one: arithmetic --- # PRECEDENCE. Both cases nest the `*' inside the `+', from opposite orderings in # the source text, so a grammar that merely preserved input order would fail one. check_eq $P/arith-mul-binds-tighter-right \ "[['1', '+', ['2', '*', '3']]]" "$(pv arith_mul_binds_tighter_right)" check_eq $P/arith-mul-binds-tighter-left \ "[[['1', '*', '2'], '+', '3']]" "$(pv arith_mul_binds_tighter_left)" # RIGHT ASSOCIATIVITY NESTS, and the nesting is the assertion. check_eq $P/arith-pow-right-assoc \ "[['2', '^', ['3', '^', '2']]]" "$(pv arith_pow_right_assoc)" check_eq $P/arith-pow-right-assoc-vars \ "[['a', '^', ['b', '^', 'c']]]" "$(pv arith_pow_right_assoc_vars)" # THE UNARY OPERATOR, and its precedence against `^'. `-2^2' binds the power # first -- ['-', ['2','^','2']] -- which is the mathematical reading and the one # a level ordering could easily get backwards. check_eq $P/arith-unary-minus \ "[[['-', 'a'], '+', 'b']]" "$(pv arith_unary_minus)" check_eq $P/arith-unary-minus-below-pow \ "[['-', ['2', '^', '2']]]" "$(pv arith_unary_minus_below_pow)" # LEFT ASSOCIATIVITY IS FLAT. These three are the suite's most useful statement # about infix_notation and the one most likely to be read as a bug in the port: # a same-precedence left-associative run becomes ONE group holding every operand # and operator, NOT a left-leaning binary tree. Upstream 3.1.1's behaviour. check_eq $P/arith-sub-left-assoc-is-flat \ "[['a', '-', 'b', '-', 'c']]" "$(pv arith_sub_left_assoc_is_flat)" check_eq $P/arith-add-left-assoc-is-flat \ "[['1', '+', '2', '+', '3']]" "$(pv arith_add_left_assoc_is_flat)" check_eq $P/arith-muldiv-same-level-flat \ "[['a', '*', 'b', '/', 'c']]" "$(pv arith_muldiv_same_level_flat)" # EXPLICIT PARENTHESES override the precedence, and infix_notation handles them # itself -- they are not in the operator table. check_eq $P/arith-parens-override \ "[[['1', '+', '2'], '*', '3']]" "$(pv arith_parens_override)" # ------------------------------------------- grammar two: key-value config --- # The compatibility spelling the contract names resolves to the 3.1 class. This # is the value that changes when upstream finally removes the alias. check_eq $P/delimited-list-type "DelimitedList" "$(pv delimited_list_type)" # POSITIONALLY: the whole result, then one pair reached by index. `Suppress' is # as much the subject here as `Word' -- the `=' and `;' are required in the input # and absent from the result. check_eq $P/kv-as-list \ "[['host', 'example.net'], ['port', '8080'], ['mode', 'fast']]" "$(pv kv_as_list)" check_eq $P/kv-pos1 "['port', '8080']" "$(pv kv_pos1)" # BY NAME: the same two fields through the names `set_results_name' attached. check_eq $P/kv-name1 "port" "$(pv kv_name1)" check_eq $P/kv-value1 "8080" "$(pv kv_value1)" # `as_dict()' against a literal -- the named structure projected into plain # Python dicts, which is the form most callers actually consume. check_eq $P/kv-as-dict \ "{'pairs': [{'name': 'host', 'value': 'example.net'}, {'name': 'port', 'value': '8080'}, {'name': 'mode', 'value': 'fast'}]}" \ "$(pv kv_as_dict)" # The top level carries exactly the one name set on it, and reaching the list # through that name gives the same thing indexing gave. check_eq $P/kv-keys "pairs" "$(pv kv_keys)" check_eq $P/kv-byname-same "yes" "$(pv kv_byname_same)" # THE TWO SURPRISES, each on its own so neither can hide inside the whole-result # comparison above. QuotedString STRIPS its quotes... check_eq $P/kv-quote-stripped "example.net" "$(pv kv_quote_stripped)" # ...and Word(nums) returns TEXT. `8080' is a str: Word matches characters and # converts nothing, so a caller expecting an int gets a string. check_eq $P/kv-numeric-type "str" "$(pv kv_numeric_type)" # ----------------------------------------- grammar three: recursion et al. --- # `Forward' carrying the recursion, with comments and irregular indentation # INSIDE the nesting. check_eq $P/nested-tree \ "[['a', 'b', ['c', ['d']], 'e']]" "$(pv nested_tree)" # ...and the claim that makes the comment skipping mean something: the same # grammar over the same input without comments gives the identical tree. check_eq $P/nested-comments-transparent "yes" "$(pv nested_comments_transparent)" # `Optional' WITH A DEFAULT supplies a real token when the element is absent, so # the result LENGTH is the same either way. A bare Optional would simply produce # a shorter list, and that difference is the whole point of the argument. check_eq $P/optional-present "['ab', '12']" "$(pv optional_present)" check_eq $P/optional-absent "['ab', '0']" "$(pv optional_absent)" # `OneOrMore' AGAINST `ZeroOrMore' on empty input -- the one case that separates # them, and it separates them by RAISING rather than by returning differently. check_eq $P/oneormore-empty "ParseException" "$(pv oneormore_empty)" check_eq $P/zeroormore-empty "[]" "$(pv zeroormore_empty)" # ------------------------------------------------------- the failure path --- check_eq $P/err-raised "yes" "$(pv err_raised)" check_eq $P/err-type "ParseException" "$(pv err_type)" # A caller catching broadly relies on the base class, so the relationship is # asserted rather than assumed from the name. check_eq $P/err-is-base "yes" "$(pv err_is_base)" # THE TWO COORDINATE SYSTEMS. `loc' is an offset into the WHOLE input; `col' and # `lineno' locate the same point within its line. The fixture is three lines so # the two cannot be confused with each other, and `err-line' must be the third # line ALONE -- an implementation returning the entire text would fail here and # would have passed on a single-line fixture. check_eq $P/err-loc "16" "$(pv err_loc)" check_eq $P/err-col "5" "$(pv err_col)" check_eq $P/err-lineno "3" "$(pv err_lineno)" # Bracketed by the probe because the line is INDENTED and bare leading spaces do # not survive the shell transport intact; the length is asserted beside it so a # whitespace difference cannot pass unnoticed. check_eq $P/err-line "< e)>" "$(pv err_line)" check_eq $P/err-line-len "4" "$(pv err_line_len)" # `' is the finding, not a gap: the unclosed `(' means the parser consumed # the whole input and still wanted a `)', so the offset lands at end of input. check_eq $P/err-loc-char "" "$(pv err_loc_char)" # THE SECOND FAILURE, reached through `parse_all' rather than through the grammar # -- the grammar itself matched a shorter prefix happily. check_eq $P/err2-raised "yes" "$(pv err2_raised)" check_eq $P/err2-loc "9" "$(pv err2_loc)" check_eq $P/err2-col "10" "$(pv err2_col)" check_eq $P/err2-msg "Expected end of text" "$(pv err2_msg)" check_eq $P/err2-loc-char "';'" "$(pv err2_loc_char)" # ...and the same input WITHOUT `parse_all', which returns the partial parse # silently. This is what makes the case above a statement about `parse_all' and # not about the grammar, and it is the mistake a caller makes exactly once. check_eq $P/partial-without-parse-all \ "[['host', 'ok']]" "$(pv partial_without_parse_all)" # ----------------------------------------------------------------- packrat --- # Memoisation is OFF by default and the switch is one-way, which is why the probe # takes it last. check_eq $P/packrat-before "False" "$(pv packrat_before)" check_eq $P/packrat-after "True" "$(pv packrat_after)" check_eq $P/packrat-cache-type "_FifoCache" "$(pv packrat_cache_type)" # THE CONTRACT: the same ten expressions, re-parsed memoised, agree tree for tree # with the unmemoised run. A single disagreement makes this `no', and the # `arith-*' cases above are what then say which expression moved. check_eq $P/packrat-table-size "10" "$(pv packrat_table_size)" check_eq $P/packrat-identical "yes" "$(pv packrat_identical)" rm -rf "$W"