. "$BST_LIB" # BST-TIMEOUT: 2000 # FIVE python3 starts, and they are the whole cost. Every # # other assertion below reads the filesystem or dpkg with # # shell builtins, or greps a line out of the transcript the # # third start left behind. # # # # The five are: a bare `import', a second one whose STDERR # # is the assertion (`import-quiet'), the probe -- which # # takes ONE start for 159 measurements -- and two runs of # # /usr/bin/python-cson, the package's only executable. # # # # The work inside each start is tokenising and re-parsing # # documents under 200 bytes. The one exception is # # `array-500-len', a 500-element array, which is there # # because the tokeniser is a character-at-a-time Python # # loop and the only case in the suite whose cost is visible # # at all. Microseconds beside an interpreter start on this # # arm, which is tens of seconds before the first line of # # the program runs. # # # # 2000 is 400s per start, the same per-start budget # # 99-python3-imagesize sets for its three, and roughly six # # times the ~60s a python3 start has measured at here. # # # # A suite-timeout red is a batch-wide cost, so if the five # # starts prove slower than that on target the RIGHT fix is # # this number, not fewer cases. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-cson 1.0.9+1.0.10 (git 69090778, meta-openembedded/meta-python # recipes-extended/python-cson/python3-cson_git.bb -- `inherit setuptools3', a # git SRC_URI and one upstream patch that stops setup.py shelling out to # `git describe') on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE IS, WHICH IS NOT WHAT ITS NAME PROMISES. # ============================================================================ # # CSON is CoffeeScript Object Notation: JSON with the punctuation removed -- # unquoted keys, comments, no commas, and NESTING BY INDENTATION. `cson.py' is # one flat 462-line module that reads a relaxed dialect and hands the result to # `json.loads', and the dialect it reads IS NOT THAT ONE. Indentation is # discarded by the tokeniser, so nesting requires braces; the multi-line form is # a `|' verbatim block rather than a triple-quoted heredoc; and the `@' # reference -- the one thing the module has that JSON does not -- raises on # every input. What is left is a SUPERSET OF JSON: every JSON document parses, # and it parses identically through both parsers, which is the compatibility # claim asserted here in three cases. # # So this suite is mostly a map of the distance between the name and the module. # Where BRINTOS-2204 describes a feature this release does not have, the case is # written at the value the release actually produces, so a future version that # closes the gap turns the suite red at the case that names it. # # ============================================================================ # THE FOUR FINDINGS. # ============================================================================ # # indent-nesting The defining feature of CSON does not work. `a:' followed # by an indented `b: 1' becomes `{"a":"b":1}' and dies in # `json'. `brace-nesting' is the same document with braces # and passes, so the pair states the shape rather than a # lone failure. # # ref-* The `@.a' reference RAISES `KeyError: ''' on every input, # because the marker `toJSON' writes prepends two `@' to a # token that already starts with one and the resolver strips # three. `csons2py' runs the resolver unconditionally, so a # document that merely CONTAINS a string beginning `@@@' # takes the same path -- `ref-lookalike-string'. # # leading-zero `isNumber' is `float(s)' in a try, so it accepts spellings # and siblings JSON refuses. `007', `+5', `.5', `1_000', `nan' and `inf' # are each left unquoted as numbers and then refused by # `json', so the input is LOST rather than read as the # string it plainly is. Quoting the same token works, which # is the workaround and the proof of where the loss is. # # unterminated-* An unterminated quote does NOT raise. It is consumed to # end of input and the partial string is returned, which is # exactly the "partial structure" BRINTOS-2204 asks the # package not to produce. # # And one absence that is a fact about the release rather than a defect: # `dump' and `dumps' do not exist. BRINTOS-2204 names them; `__all__' is # `('loads',)' and no name in the module serialises a Python object. The # round-trip the ticket wanted is asserted from the other side instead -- # `json.dumps' output reads back through `cson.loads', and `toJSON' output reads # back through both parsers and is stable across calls. # # ============================================================================ # WHY THE FIXTURES ARE BUILT IN THE PROBE. # ============================================================================ # # Every document this parser reads is under 200 bytes and its SHAPE is the # assertion -- a case exists to vary one character of syntax at a time. Writing # them in the probe beside the case that reads them keeps the shape legible and # keeps ~40 one-line fixture files out of the layer. Only the file-API cases # touch the filesystem, and they write into $TMPDIR. P=python3-cson # The suite reached its first line. Everything after this is a real result, so a # run that ends without this case ended before the suite started. bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages MOD=$SITE/cson.py DI=$SITE/python_cson-1.0.9+1.0.10.dist-info BIN=/usr/bin/python-cson PKGLIST=/var/lib/dpkg/info/python3-cson.list PROBE=$BST_DATA/python3-cson-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-cson.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------ what shipped ---- # A FLAT MODULE, NOT A PACKAGE. `py_modules = ["cson"]' installs one file, so # there is no `cson/' directory and no `__init__.py' anywhere -- `no-package-dir' # is the case that would catch a future release restructuring into a package # while every import case below stayed green. check $P/site-packages-dir test -d "$SITE" check $P/module-present test -s "$MOD" check $P/no-package-dir test ! -d "$SITE/cson" check $P/pycache-module test -s "$SITE/__pycache__/cson.cpython-312.pyc" # `version.py' IS WRITTEN AT BUILD TIME AND NOT INSTALLED. The recipe's # do_configure:prepend writes it so setup.py can read a version out of it; it is # not in py_modules, so it must not reach the image -- where it would be a # top-level module called `version' on every interpreter's sys.path. check $P/no-version-module test ! -e "$SITE/version.py" cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # NO SHARED OBJECT AT ANY PATH. The package is pure Python -- setup.py declares # no ext_modules and there is no C in the tree -- so a `.so' in the file list # would mean something else was packaged into it. check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # AND NOTHING BUT PYTHON AND THE ONE SCRIPT. The list is walked rather than # counted, so a file of a kind nobody expected is named in the red rather than # changing a total. _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; "$BIN") continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" # ------------------------------------------------------------- the dist-info -- # THE DISTRIBUTION AND THE IMPORT HAVE DIFFERENT NAMES: the distribution is # `python-cson' (directory `python_cson-…'), the import is `cson'. Both are # asserted, because `importlib.metadata' answers only to the first and a caller # only ever types the second. check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 1.0.9+1.0.10" cat "$DI/METADATA" check_out $P/dist-info-name "Name: python-cson" cat "$DI/METADATA" check $P/dist-info-license test -s "$DI/LICENSE" check $P/dist-info-record test -s "$DI/RECORD" check_out $P/dist-info-generator "Generator: bdist_wheel" cat "$DI/WHEEL" check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag-py3 "Tag: py3-none-any" cat "$DI/WHEEL" check_eq $P/dist-info-tag-only-py3 "1" \ "$(grep -c '^Tag: ' "$DI/WHEEL" 2>/dev/null)" check_out $P/dist-info-top-level "cson" cat "$DI/top_level.txt" # THE DECLARED DEPENDENCY `py' IS NOT ON THE IMAGE. setup.py's install_requires # names it and nothing in cson.py imports it, so it is stale metadata rather # than a hole -- the METADATA half is asserted here and the import half in the # probe, so neither can go green alone. check_out $P/dist-info-requires-py "Requires-Dist: py " cat "$DI/METADATA" # ------------------------------------------------------------- the console ---- check $P/bin-present test -s "$BIN" check $P/bin-executable test -x "$BIN" check_out $P/bin-shebang "#!/usr/bin/env python3" head -n 1 "$BIN" check_out $P/entry-points-console "[console_scripts]" cat "$DI/entry_points.txt" check_out $P/entry-points-target "python-cson = cson:main" cat "$DI/entry_points.txt" # ----------------------------------------------------------------- the dpkg --- sed -n '/^Package: python3-cson$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # THE PACKAGE VERSION CARRIES THE SRCREV, and the recipe's PV does not. The # recipe pins `SRCREV = "69090778…"' with `PV = "1.0.9+1.0.10"', so the .deb # version is the PV with the revision appended and is NOT the version the # dist-info reports. Both are asserted, at their own values, because a reader # who saw only one would take it for the other. check_out $P/pkg-version "Version: 1.0.9+1.0.100+69090778bc-r0" cat "$W/st" # RDEPENDS IS EXACTLY WHAT THE RECIPE DECLARES PLUS python3-core. cson.py # imports `argparse', `copy', `json' and `sys'; the first two and the last are # in python3-core and `json' is the recipe's one explicit RDEPENDS. Asserted # exactly rather than by substring, so a dependency appearing OR disappearing is # a red. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_eq $P/pkg-depends-exact \ "Depends: python3-core, python3-json" \ "$(cat "$W/dep")" # ------------------------------------------------------------- the import ----- check $P/import python3 -c 'import cson' # AND IT SAYS NOTHING WHILE DOING IT. cson.py runs `from __future__ import ...' # and three stdlib imports at module scope; a DeprecationWarning from any of # them would land on the stderr of every caller. check_eq $P/import-quiet "" \ "$(python3 -c 'import cson' 2>&1)" # ------------------------------------------------------------ the executable -- # THE TWO CLI FORMS THE README DOCUMENTS BOTH FAIL. It shows # `python-cson ' printing to stdout and `cat | python-cson -f # ' reading stdin; argparse is built with `-f' REQUIRED and `input_file' # positional and required, so the first exits 2 and the second has no way to # name an input at all. The form that works is the one the README does not lead # with, and it is asserted first so the red on the other reads as a # documentation gap rather than as a broken package. printf '# a document\na: 1\nb: "two"\nc: [1, 2]\n' > "$W/in.cson" check $P/cli-convert python3 "$BIN" -f "$W/out.json" "$W/in.cson" check $P/cli-output-present test -s "$W/out.json" check_eq $P/cli-output '{"a": 1, "b": "two", "c": [1, 2]}' \ "$(cat "$W/out.json" 2>/dev/null)" check_rc $P/cli-readme-form 2 python3 "$BIN" "$W/in.cson" # ---------------------------------------------------------------- the probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # The probe reached its last line, and emitted every value it meant to. A short # transcript would otherwise read as a run of empty `pv' results, each red at # the case rather than at the cause. check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "159" "$(pv emitted_count)" # ---------------------------------------------------- module identity -------- # The module is FLAT -- `cson.py' beside site-packages, not a package directory # -- because setup.py declares `py_modules = ["cson"]'. The distribution is # `python-cson' and the import is `cson', so the two names differ and # `importlib.metadata' has to be asked by the distribution name. check_eq $P/module-file /usr/lib/python3.12/site-packages/cson.py "$(pv module_file)" check_eq $P/is-flat-module yes "$(pv is_flat_module)" check_eq $P/has-path-attr no "$(pv has_path_attr)" check_eq $P/dist-version 1.0.9+1.0.10 "$(pv dist_version)" check_eq $P/dist-files-named yes "$(pv dist_files_named)" # `Requires-Dist: py >=1.4.25' IS DECLARED AND NOT SATISFIED. setup.py names # `py' in install_requires, the recipe's RDEPENDS is python3-json alone, and # nothing in cson.py imports it -- so this is stale metadata rather than a # missing dependency. Both halves are asserted so an image that happened to ship # `py' could not turn the second green in silence. # # EXPECTED, NOT YET MEASURED (phase A cannot run the suite): `py-importable' is # derived from the image's package list -- python3-py is not on EMCRAFT_TOOLS # and nothing here pulls it -- rather than from a run on target. check_eq $P/dist-requires-count 2 "$(pv dist_requires_count)" check_eq $P/dist-requires-py yes "$(pv dist_requires_py)" check_eq $P/py-importable '!ModuleNotFoundError' "$(pv py_importable)" # THE VERSION IS NOT AVAILABLE FROM THE MODULE. setup.py writes a `version.py' # beside cson.py at build time and `py_modules' does not install it, so neither # `cson.__version__' nor the module it would have come from exists on the image. # `importlib.metadata' is the only way to ask, which is why `dist-version' above # is the version case and there is no attribute case. check_eq $P/has-version-attr no "$(pv has_version_attr)" check_eq $P/version-module '!ModuleNotFoundError' "$(pv version_module)" # ------------------------------------------------------- the public names ---- # `__all__' IS ONE NAME. `from cson import *' gives a caller `loads' and nothing # else -- not `load', not `toJSON' -- so every other entry point has to be # reached through the module object. check_eq $P/all-names loads "$(pv all_names)" check_eq $P/all-count 1 "$(pv all_count)" check_eq $P/has-loads yes "$(pv has_loads)" check_eq $P/has-load yes "$(pv has_load)" check_eq $P/has-csons2py yes "$(pv has_csons2py)" check_eq $P/has-cson2py yes "$(pv has_cson2py)" check_eq $P/has-csons2json yes "$(pv has_csons2json)" check_eq $P/has-toJSON yes "$(pv has_toJSON)" check_eq $P/has-tokenize yes "$(pv has_tokenize)" check_eq $P/has-main yes "$(pv has_main)" # THERE IS NO WRITER. BRINTOS-2204 names `dump' and `dumps' among the # `json'-shaped four this package exposes; 1.0.9+1.0.10 has neither, and no # other name in the module serialises a Python object either. The absence is # asserted rather than noted, so a release that adds a writer turns this suite # red at the case that names it. What the ticket wanted from `dumps' -- output # that reads back -- is asserted further down, from the other side, through # `json.dumps' and `toJSON'. check_eq $P/has-dumps no "$(pv has_dumps)" check_eq $P/has-dump no "$(pv has_dump)" check_eq $P/dumps-absent yes "$(pv dumps_absent)" check_eq $P/dump-absent yes "$(pv dump_absent)" # `loads' AND `load' ARE ALIASES ASSIGNED AT THE FOOT OF THE MODULE, not defs, # so the identity is what says the two spellings cannot drift apart. check_eq $P/loads-is-csons2py yes "$(pv loads_is_csons2py)" check_eq $P/load-is-cson2py yes "$(pv load_is_cson2py)" check_eq $P/csons2json-calls-tojson yes "$(pv csons2json_calls_tojson)" # ------------------------------------------------------------- the scalars --- # Every JSON scalar, and the whole set in one document. check_eq $P/int '{"a":1}' "$(pv int)" check_eq $P/negative '{"a":-1}' "$(pv negative)" check_eq $P/float '{"a":1.5}' "$(pv float)" check_eq $P/exponent '{"a":1000.0}' "$(pv exponent)" check_eq $P/bigint '{"a":12345678901234567890}' "$(pv bigint)" check_eq $P/true '{"a":true}' "$(pv true)" check_eq $P/false '{"a":false}' "$(pv false)" check_eq $P/null '{"a":null}' "$(pv null)" check_eq $P/all-scalars '{"f":1.5,"i":1,"n":null,"s":"word","t":true,"x":false}' "$(pv all_scalars)" # A BARE WORD IS A STRING; `true', `false' AND `null' ARE NOT. The tokeniser # quotes any value token that is not a number, a boolean, a null or a `@' # reference, so those four keywords are the entire set of unquoted non-strings # -- and quoting one turns it back into a string. check_eq $P/bare-word '{"a":"word"}' "$(pv bare_word)" check_eq $P/quoted-true '{"a":"true"}' "$(pv quoted_true)" check_eq $P/quoted-number '{"a":"1"}' "$(pv quoted_number)" # ------------------------------------------------ the CSON relaxations ------- # The relaxations BRINTOS-2204 names, one case each: unquoted keys, `=' as well # as `:' for the separator, `#' line comments, and trailing commas in both # containers. Commas are DELETED by the tokeniser rather than parsed, which is # why a trailing one is accepted and why a document with no commas at all parses # the same as one with them. check_eq $P/unquoted-key '{"a":1}' "$(pv unquoted_key)" check_eq $P/equals-separator '{"a":1}' "$(pv equals_separator)" check_eq $P/mixed-separators '{"a":1,"b":2}' "$(pv mixed_separators)" check_eq $P/key-with-dash '{"a-b":1}' "$(pv key_with_dash)" check_eq $P/key-with-dot '{"a.b":1}' "$(pv key_with_dot)" check_eq $P/key-unicode '{"ключ":1}' "$(pv key_unicode)" check_eq $P/quoted-key-with-space '{"a b":1}' "$(pv quoted_key_with_space)" check_eq $P/line-comment '{"a":1,"b":2}' "$(pv line_comment)" check_eq $P/comment-no-space '{"a":1,"b":2}' "$(pv comment_no_space)" check_eq $P/comment-at-eof '{"a":1}' "$(pv comment_at_eof)" check_eq $P/trailing-comma-object '{"a":1,"b":2}' "$(pv trailing_comma_object)" check_eq $P/trailing-comma-array '{"a":[1,2,3]}' "$(pv trailing_comma_array)" check_eq $P/no-commas-at-all '{"a":1,"b":2}' "$(pv no_commas_at_all)" check_eq $P/crlf '{"a":1,"b":2}' "$(pv crlf)" check_eq $P/no-trailing-newline '{"a":1}' "$(pv no_trailing_newline)" # ------------------------------------------------------------ the strings ---- # Single and double quotes, and the four characters that would end a token # outside a string. A double quote inside a SINGLE-quoted string is escaped for # the author: every string leaves the tokeniser as a double-quoted JSON string, # so the backslash has to be inserted by the tokeniser. check_eq $P/double-quoted '{"a":"hello world"}' "$(pv double_quoted)" check_eq $P/single-quoted '{"a":"hello world"}' "$(pv single_quoted)" check_eq $P/empty-string '{"a":""}' "$(pv empty_string)" check_eq $P/hash-inside-string '{"a":"x # y"}' "$(pv hash_inside_string)" check_eq $P/colon-inside-string '{"a":"x: y"}' "$(pv colon_inside_string)" check_eq $P/comma-inside-string '{"a":"x, y"}' "$(pv comma_inside_string)" check_eq $P/escape-tab '{"a":"x\\ty"}' "$(pv escape_tab)" check_eq $P/escape-newline '{"a":"x\\ny"}' "$(pv escape_newline)" check_eq $P/unicode-value '{"a":"héllo"}' "$(pv unicode_value)" check_eq $P/dq-containing-sq '{"a":"he said '"'"'hi'"'"'"}' "$(pv dq_containing_sq)" check_eq $P/sq-containing-dq '{"a":"he said \\"hi\\""}' "$(pv sq_containing_dq)" # ------------------------------------------- multi-line: `|', NOT triple ------ # THE MULTI-LINE FORM IS THE `|' VERBATIM BLOCK -- one bar per line, the text # after the bar taken literally, leading spaces included, the lines joined with # a newline. BRINTOS-2204 asks for triple-quoted multi-line strings with # documented indentation stripping; that is CoffeeScript's CSON and it is NOT # implemented here. `'''' is read as an empty single-quoted string followed by a # stray quote, and the raw newlines that follow reach `json' as control # characters inside a string literal. There is no indentation stripping to # assert anywhere in this release -- `verbatim-keeps-indent' is the opposite # property, stated at the value it actually produces. check_eq $P/verbatim-two-lines '{"a":"one\\ntwo"}' "$(pv verbatim_two_lines)" check_eq $P/verbatim-three-lines '{"a":"one\\ntwo\\nthree"}' "$(pv verbatim_three_lines)" check_eq $P/verbatim-blank-line '{"a":"one\\n\\nthree"}' "$(pv verbatim_blank_line)" check_eq $P/verbatim-keeps-indent '{"a":" spaced\\nx"}' "$(pv verbatim_keeps_indent)" check_eq $P/triple-quoted '!JSONDecodeError' "$(pv triple_quoted)" # ================================================================= THE GAP ==== # INDENTATION IS NOT SIGNIFICANT, AND NOTHING IN THE PACKAGE SAYS SO. # # CSON is CoffeeScript Object Notation and its defining feature -- the reason it # exists rather than being JSON -- is that nesting is expressed by INDENTATION. # This tokeniser deletes runs of spaces (`isWS') and deletes newlines (they sit # in the same branch as `,'), so an indented block arrives at `json' as a flat # run of tokens with no brace ever inserted: # # a: # b: 1 # # becomes `{"a":"b":1}' and dies in `json' with a message about a missing `,' # delimiter at a column in a string the caller never wrote. Braces are REQUIRED, # which makes the accepted language a superset of JSON and a DISJOINT set from # the CSON CoffeeScript writes. The two cases are the same document with and # without braces, so the finding is a contrast rather than a lone failure. check_eq $P/indent-nesting '!JSONDecodeError' "$(pv indent_nesting)" check_eq $P/brace-nesting '{"a":{"b":1}}' "$(pv brace_nesting)" check_eq $P/indent-nesting-two-keys '!JSONDecodeError' "$(pv indent_nesting_two_keys)" # --------------------------------------------------------- the containers ---- check_eq $P/nested-object '{"a":{"b":1,"c":2}}' "$(pv nested_object)" check_eq $P/nested-array '{"a":[1,[2,3]]}' "$(pv nested_array)" check_eq $P/array-of-objects '{"a":[{"b":1},{"c":2}]}' "$(pv array_of_objects)" check_eq $P/deep-eight '{"a":{"b":{"c":{"d":{"e":{"f":{"g":{"h":1}}}}}}}}' "$(pv deep_eight)" check_eq $P/empty-object '{}' "$(pv empty_object)" check_eq $P/empty-array '[]' "$(pv empty_array)" check_eq $P/empty-members '{"a":[],"b":{}}' "$(pv empty_members)" check_eq $P/array-500-len 500 "$(pv array_500_len)" check_eq $P/duplicate-key '{"a":2}' "$(pv duplicate_key)" # A TOP-LEVEL DOCUMENT WITH NO BRACES IS WRAPPED, and ONE token decides which # wrapper: if the second token is `:' it becomes an object, otherwise an array. # A document of a SINGLE token is wrapped in neither, so a bare number parses # and a bare word -- which would have been a string anywhere else in the # language -- does not. check_eq $P/implicit-object '{"a":1,"b":2}' "$(pv implicit_object)" check_eq $P/implicit-array '[1,2,3]' "$(pv implicit_array)" check_eq $P/bare-number 42 "$(pv bare_number)" check_eq $P/bare-word-alone '!JSONDecodeError' "$(pv bare_word_alone)" check_eq $P/two-bare-words '!JSONDecodeError' "$(pv two_bare_words)" # ============================================== THE `@' REFERENCE IS DEAD ===== # The one feature this module has that JSON does not -- a value `@.a.b' that # resolves to another node of the same document -- RAISES ON EVERY INPUT. # # `toJSON' marks a reference by prepending `@@' to a token that already begins # with `@', so `@.a' becomes the string `"@@@.a"'. `findrefs' collects the path # of the HOLDER, `.b'. `resolveRefs' reads the holder's value, strips three # characters to get `.a', and looks it up as `'.' + '.a'' -- `..a', whose # `split('.')[1:]' is `['', 'a']'. The empty first component is looked up in the # top-level dict and raises `KeyError: '''. # # The off-by-one is in the marker: two `@' are added to a token whose first # character is already one, so `[3:]' removes one too few. Every reference in # every document takes that path, so the feature has never worked in this # release. And `csons2py' calls `resolveRefs' unconditionally, so a document # does not have to USE a reference to be affected -- it only has to contain a # string that begins with `@@@', which `ref-lookalike-string' shows. # # `isRef' itself is intact; the defect is downstream of it. check_eq $P/ref-flat '!KeyError' "$(pv ref_flat)" check_eq $P/ref-nested '!KeyError' "$(pv ref_nested)" check_eq $P/ref-dangling '!KeyError' "$(pv ref_dangling)" check_eq $P/ref-lookalike-string '!KeyError' "$(pv ref_lookalike_string)" check_eq $P/isref-at yes "$(pv isref_at)" check_eq $P/isref-plain no "$(pv isref_plain)" check_eq $P/isref-empty no "$(pv isref_empty)" # ============================ A VALUE PYTHON CALLS A NUMBER AND JSON DOES NOT = # `isNumber' is `float(s)' inside a try, so it accepts every spelling CPython's # float parser does -- and CPython accepts several JSON does not. Such a token # is left UNQUOTED as a number and then refused by `json', so the input is LOST # rather than read as the string it plainly is. A leading zero is the one a # human writes by accident, in a zero-padded version or an ISO date part. # # Quoting the same token makes it a string and it survives, which is both the # workaround and the proof that the loss is the number TEST rather than the # value. `0x10' is not a float to CPython, so it is quoted and comes back as the # string `0x10' rather than as 16 -- the case a caller expects to fail and the # one that silently succeeds with the wrong type. check_eq $P/leading-zero '!JSONDecodeError' "$(pv leading_zero)" check_eq $P/plus-sign '!JSONDecodeError' "$(pv plus_sign)" check_eq $P/leading-dot '!JSONDecodeError' "$(pv leading_dot)" check_eq $P/underscore-digits '!JSONDecodeError' "$(pv underscore_digits)" check_eq $P/nan-token '!JSONDecodeError' "$(pv nan_token)" check_eq $P/inf-token '!JSONDecodeError' "$(pv inf_token)" check_eq $P/leading-zero-quoted '{"a":"007"}' "$(pv leading_zero_quoted)" check_eq $P/isnumber-007 yes "$(pv isnumber_007)" check_eq $P/isnumber-nan yes "$(pv isnumber_nan)" check_eq $P/isnumber-underscore yes "$(pv isnumber_underscore)" check_eq $P/isnumber-word no "$(pv isnumber_word)" check_eq $P/hex-literal '{"a":"0x10"}' "$(pv hex_literal)" # ================================= MALFORMED INPUT THAT DOES *NOT* RAISE ====== # An unterminated quote is consumed to end of input and the PARTIAL string is # returned. BRINTOS-2204 asks that malformed input raise rather than return a # partial structure; these two are the counterexamples, asserted at the value # they actually produce so a release that starts refusing them turns this suite # red at the case that names the promise. check_eq $P/unterminated-dq '{"a":"abc"}' "$(pv unterminated_dq)" check_eq $P/unterminated-sq '{"a":"abc"}' "$(pv unterminated_sq)" # ------------------------------------------- malformed input that DOES raise -- # Everything else malformed is a `JSONDecodeError' out of `json' rather than a # diagnostic from `cson', so the column numbers in the message point into a # document the caller never saw. # # THE EMPTY DOCUMENT IS AN `IndexError', NOT A PARSE ERROR. `toJSON' reads # `tokens[0]' before checking that any token exists, so every input that # tokenises to nothing -- empty, blank, comment-only -- escapes as # `IndexError: list index out of range' with no mention of the document. A # caller catching `ValueError' to mean "bad CSON" -- which is what # `JSONDecodeError' is -- does not catch this one. check_eq $P/unclosed-brace '!JSONDecodeError' "$(pv unclosed_brace)" check_eq $P/stray-close '!JSONDecodeError' "$(pv stray_close)" check_eq $P/colon-no-value '!JSONDecodeError' "$(pv colon_no_value)" check_eq $P/empty-document '!IndexError' "$(pv empty_document)" check_eq $P/whitespace-only '!IndexError' "$(pv whitespace_only)" check_eq $P/comment-only '!IndexError' "$(pv comment_only)" check_eq $P/empty-is-indexerror yes "$(pv empty_is_indexerror)" # ------------------------------------------------ inputs that are not text ---- # `get_iterable' branches on the input type and only the `str' branch produces # something `tokenize' can compare against: it returns a LENGTH for a string and # a RANGE for a list, and `tokenize' then compares an int against whichever it # got. The `dict' branch references a global `cson_data' that does not exist # anywhere in the module, so a dict is a `NameError' rather than a `TypeError' # -- an unbound name that has been in the released source since 1.0.9. check_eq $P/bytes-input '!TypeError' "$(pv bytes_input)" check_eq $P/list-input '!TypeError' "$(pv list_input)" check_eq $P/dict-input '!NameError' "$(pv dict_input)" check_eq $P/none-input '!TypeError' "$(pv none_input)" check_eq $P/get-iterable-str 3 "$(pv get_iterable_str)" check_eq $P/get-iterable-list 'range(0, 2)' "$(pv get_iterable_list)" # ------------------------------------------------ JSON is accepted as CSON ---- # The compatibility claim BRINTOS-2204 makes: a plain JSON document parses # identically through `cson.loads' and `json.loads'. Asserted three ways -- the # two serialisations separately, so a red says WHICH parser moved, and the # equality itself, so the case does not rest on comparing two long lines. check_eq $P/json-doc-cson '{"a":1,"b":[1,2,{"c":"d"}],"e":null,"f":true,"g":1.5}' "$(pv json_doc_cson)" check_eq $P/json-doc-json '{"a":1,"b":[1,2,{"c":"d"}],"e":null,"f":true,"g":1.5}' "$(pv json_doc_json)" check_eq $P/json-doc-same yes "$(pv json_doc_same)" check_eq $P/json-nested-same yes "$(pv json_nested_same)" check_eq $P/json-escapes-same yes "$(pv json_escapes_same)" # ------------------------------------------------- writing, from the far side - # With no `dumps' to assert, the round-trip the ticket asks for is reached # through the two writers that do exist: `json.dumps' output is readable by # `cson.loads', and `toJSON' output is readable by BOTH parsers and is stable # across calls -- and, because its output is valid JSON and JSON is valid CSON, # idempotent under itself. check_eq $P/jsondumps-reread-cson '{"a":1,"b":[1,"x",null,true],"c":{"d":1.5}}' "$(pv jsondumps_reread_cson)" check_eq $P/jsondumps-roundtrip yes "$(pv jsondumps_roundtrip)" check_eq $P/tojson-default '{"a":1,"b":{"c":"d"},"e":[1,2]}' "$(pv tojson_default)" check_eq $P/tojson-reread-json '{"a":1,"b":{"c":"d"},"e":[1,2]}' "$(pv tojson_reread_json)" check_eq $P/tojson-reread-cson '{"a":1,"b":{"c":"d"},"e":[1,2]}' "$(pv tojson_reread_cson)" check_eq $P/tojson-stable yes "$(pv tojson_stable)" check_eq $P/tojson-idempotent yes "$(pv tojson_idempotent)" # `csons2json(s, indent)' INDENTS WITH FOUR SPACES PER LEVEL, so the integer is # a count of four-space UNITS and not a count of spaces -- `indent=1' gives four # and `indent=2' gives eight. `indent=0', the default, produces no whitespace. check_eq $P/indent-0 '{"a":{"b":1}}' "$(pv indent_0)" check_eq $P/indent-1 '{\n "a": {\n "b": 1\n }\n}' "$(pv indent_1)" check_eq $P/indent-2 '{\n "a": {\n "b": 1\n }\n}' "$(pv indent_2)" check_eq $P/indent-1-valid-json yes "$(pv indent_1_valid_json)" # ---------------------------------------------------------------- tokenize ---- # The tokeniser is public and is the layer every finding above lives in, so its # output is asserted directly: comments vanish, commas vanish, `=' is rewritten # to `:', and a verbatim block is already one JSON string literal by the time # `toJSON' sees it. check_eq $P/tok-simple '["a", ":", "1", "b", ":", "\\"x\\""]' "$(pv tok_simple)" check_eq $P/tok-verbatim '["a", ":", "\\"x\\\\ny\\""]' "$(pv tok_verbatim)" check_eq $P/tok-comment-only '[]' "$(pv tok_comment_only)" check_eq $P/tok-equals-becomes-colon '["a", ":", "1"]' "$(pv tok_equals_becomes_colon)" check_eq $P/tok-commas-dropped '["[", "1", "2", "]"]' "$(pv tok_commas_dropped)" # --------------------------------------------------------- the helpers -------- # `isName(None)' IS TRUE, which is not a bug: the tokeniser passes the character # after the last one as `None' and relies on `None' matching none of the # terminators, so the end of input reads as "still a name character". # # `isWS' IS SPACES ONLY -- a tab is not whitespace to this tokeniser -- so a # document indented with tabs tokenises differently from the same document # indented with spaces. The consequence is asserted beside the predicate. check_eq $P/string-to-literal 'a\\tb\\nc\\"d\\\\e' "$(pv string_to_literal)" check_eq $P/isname-none yes "$(pv isname_none)" check_eq $P/isname-colon no "$(pv isname_colon)" check_eq $P/isname-letter yes "$(pv isname_letter)" check_eq $P/isboolean-true yes "$(pv isboolean_true)" check_eq $P/isboolean-True no "$(pv isboolean_True)" check_eq $P/isnull-null yes "$(pv isnull_null)" check_eq $P/isnull-None no "$(pv isnull_None)" check_eq $P/iscrlf yes "$(pv iscrlf)" check_eq $P/isws-space yes "$(pv isws_space)" check_eq $P/isws-tab no "$(pv isws_tab)" check_eq $P/tab-indent-doc '!JSONDecodeError' "$(pv tab_indent_doc)" # ---------------------------------------------------------- the file API ------ # `load(path)' TAKES A FILENAME, NOT AN OPEN FILE. It is `cson2py', which calls # `open()' on what it is handed, so the `json'-shaped `load(fp)' a caller # reaches for is a `TypeError' rather than a parse. A missing file raises from # `open' before any parsing starts; an EMPTY file reaches the `IndexError' # above, so a zero-byte config is indistinguishable from a bug in the caller. check_eq $P/load-file '{"a":1,"b":"two","c":[1,2]}' "$(pv load_file)" check_eq $P/load-missing '!FileNotFoundError' "$(pv load_missing)" check_eq $P/load-fileobj '!TypeError' "$(pv load_fileobj)" check_eq $P/load-empty-file '!IndexError' "$(pv load_empty_file)" check_eq $P/load-matches-loads yes "$(pv load_matches_loads)" rm -rf "$W"