. "$BST_LIB" # BST-TIMEOUT: 1800 # THREE python3 starts, and they are most of the cost. # # Two are the module -- a bare `import', and a second whose # # STDERR is the assertion -- and the third is the probe, # # which takes ONE start for a hundred-odd measurements # # INCLUDING the whole of UAX #15 conformance. # # # # The work inside the probe is not small and is not meant # # to be: 19074 conformance records under four forms, a # # sweep over the other 1097083 code points, 11172 Hangul # # syllables, and a record-by-record cross-check against the # # interpreter's own unicodedata. MEASURED at 1.6s TOTAL on # # the x86-64 control, interpreter start included, because # # every operation is a dict lookup over a short string. # # # # 1800 is 600s per start, ten times the ~60s a python3 # # start has measured at on this arm, and it leaves room for # # the probe's compute to be two orders of magnitude slower # # here than on the control. The probe carries its own 1500s # # bound below so an overrun is reported as ONE red naming # # the last line it printed, rather than as a suite-timeout # # that costs the whole batch its round. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify. The measured elapsed replaces # # this note in the TEST PLAN rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-pyunormalize 15.1.0 (meta-openembedded/meta-python # recipes-devtools/python/python3-pyunormalize_15.1.0.bb, `inherit pypi # setuptools3', PyPI sdist) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE IS FOR, AND WHY THAT MAKES ONE CONSTANT LOAD-BEARING. # ============================================================================ # # NFC, NFD, NFKC and NFKD -- the four Unicode normalization forms -- computed # against a Unicode database THE PACKAGE CARRIES ITSELF rather than against the # one compiled into the running interpreter. Python's own `unicodedata' gives # you whatever version CPython was built with; this gives you the version it # says it gives you, and that is the entire reason to install it. # # So `UNICODE_VERSION' is not a nice-to-have version string, it is the product. # It is asserted from all three places the package states it -- the top-level # package, the generated `unicode.py', and the identity between `UCD_VERSION' # and `UNICODE_VERSION' that `__init__.py' enforces at import with a SystemExit. # `interp-unidata' beside it records the interpreter's OWN version, which on # this image is a DIFFERENT number and is supposed to be. # # ============================================================================ # THE CONFORMANCE CORPUS IS ON THE IMAGE, WHICH IS WHY THIS SUITE IS THOROUGH. # ============================================================================ # # setup.py takes `packages=find_packages()' with `include_package_data=True', # and MANIFEST.in says `global-include *.txt'. The consequence, MEASURED off the # built .deb rather than reasoned from the source, is that # `pyunormalize/test/data/NormalizationTest.txt' -- the Unicode Consortium's own # 2,625,136-byte conformance corpus -- IS INSTALLED, along with the whole `test' # subpackage. # # That is 3.5 MB of rootfs for a library, and it is worth knowing about. It is # also what lets this suite assert the WHOLE function instead of a sample: the # authoritative corpus is already there, so the probe runs all four UAX #15 # conformance conditions over all 19074 records, and then sweeps the 1097083 # code points the corpus does NOT list to assert that nothing else moved. # # Sampling `NFD("élève")' would have cost the same interpreter start. The # failure mode this package has that a hand-written library does not is a # TRUNCATED OR MIS-GENERATED TABLE -- `unicode.py' is 503 KB of generated # literals -- and a truncation is nearly invisible behaviourally: the accented # Latin cases everyone reaches for keep working while thousands of rarer # characters silently normalize to themselves. `conf-*' and `sweep-*' are what # see that; `n-*' and `qc-*' are the cheap independent statement of the same # thing, and the two fail apart. # # `part1-is-nfkd-no' is the check that costs nothing and is the strongest of # them: Part 1 of the shipped corpus is exactly the set of characters with # NFKD_Quick_Check=No, and `unicode.py's `nfkd_no' table is generated # separately. Two independently produced artefacts agreeing on a 17029-member # set is not something a truncation on either side survives. # # ============================================================================ # ONE UPSTREAM DEFECT, ASSERTED AS IT BEHAVES: U+0000 IS DELETED BY NFC/NFKC. # ============================================================================ # # `_compose' marks consumed code points by setting them to None and finishes # with `[*filter(None, elems)]'. That discards every FALSY element -- and 0 is # falsy. So any U+0000 in a string that actually takes the COMPOSITION path is # silently dropped: `NFC("a" + U+0300 + NUL)' comes back as U+00E0 alone. # # It is data loss rather than a wrong answer, it is upstream 15.1.0's and not # this platform's, and `nul-composed-ud' is what says so -- the same input # through the interpreter's own `unicodedata.normalize' keeps the NUL. # # The quick-check path returns its input untouched, so a NUL survives whenever # nothing was composed (`nul-alone', `nul-ascii', `nul-precomposed'), and NFD # and NFKD never compose at all. That asymmetry is asserted rather than assumed, # because it is the evidence for the cause. # # The `nul-composed-*' cases record what 15.1.0 DOES. A release that fixed it # turns them red, which is the correct signal -- it would mean the behaviour # changed -- and the reader is told here that it is the wanted direction. # # We file nothing upstream for this: it is a third-party public repository, and # docs/conventions/no-public-upstream-reports.md governs. # # ============================================================================ # WHY IT PORTS AT ALL: PURE PYTHON, python3-core AND NOTHING ELSE. # ============================================================================ # # This image's python3 is ONE STATIC BINARY with an EMPTY lib-dynload and no # dynamic loader (BRINTOS-1476; 99-python3 asserts both directly), so a module # carrying a compiled extension could not be imported here -- not "would be # slow", could not be imported. That is the gap BRINTOS-2046 tracks, and it does # not reach this package: setup.py declares no `ext_modules', the sources import # nothing outside the package, and the wheel is `Root-Is-Purelib: true'. # # MEASURED off the built python3-pyunormalize_15.1.0-r0_wasm32.deb: the control # stanza reads `Depends: python3-core' and nothing else, so this port waits on # no other package. `pkg-depends-exact' is what would say so if a release grew a # third-party requirement, and `pkg-no-so' is what would catch an extension at # install rather than at import. # # ============================================================================ # WHERE THE EXPECTED VALUES CAME FROM. # ============================================================================ # # Every literal below was measured on hz1 against the bytes unpacked from the # built .deb -- not against the sdist -- under CPython 3.12.3 with # `unicodedata.unidata_version' 15.0.0, which is the interpreter version AND the # UCD version this image ships. That is what makes `ud-*-diff' expectations # meaningful: they are scored against the same interpreter database the target # has. # # The on-target result is not yet measured -- the batch verify is this port's # first run on wasm32-hwjs. P=python3-pyunormalize # THE CANARY, which asserts nothing about python3-pyunormalize. # # A suite's FIRST BST result line is lost between the guest's printf and the # host's read -- per-suite, deterministic, and NOT a defect in the host tool # (BRINTOS-1149 established that and filed it as linux-in-the-browser#177). The # harness computes its pass count as total-minus-failures, so THE DROPPED LINE # IS COUNTED AS A PASS. A nothing-case therefore stands in that position and the # eaten line costs nothing. It sits ahead of the `require' guards deliberately, # because a `require' that skips would emit its skip as the first line and be # eaten too. # # Remove this case when linux-in-the-browser#177 is fixed. bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/pyunormalize DI=$SITE/pyunormalize-15.1.0.dist-info FIXTURE=$PKG/test/data/NormalizationTest.txt PKGLIST=/var/lib/dpkg/info/python3-pyunormalize.list PROBE=$BST_DATA/python3-pyunormalize-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-pyunormalize.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------- the payload --- # Where the package has to land. `import pyunormalize' further down is the # functional statement; these are the ones that say WHICH copy answered, so a # package that imported out of some other path on sys.path is distinguishable # from one that imported out of what this image installed. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" # A PACKAGE, NOT A MODULE. See the header: `find_packages()' is what drags the # `test' subpackage and its corpus onto the image, and a release that flattened # to a single file would still import and still pass every behavioural case. check $P/not-a-module test ! -f "$SITE/pyunormalize.py" check $P/init-py test -s "$PKG/__init__.py" check $P/version-py test -s "$PKG/_version.py" check $P/normalization-py test -s "$PKG/normalization.py" # THE GENERATED DATA MODULE, half a megabyte of Unicode literals. It is the # package: `normalization.py' is the algorithm and this is everything the # algorithm knows. check $P/unicode-py test -s "$PKG/unicode.py" # The precompiled bytecode, which is what `import-quiet' below turns on and what # keeps the 503 KB source out of the import path on every start. check $P/pycache-init test -s "$PKG/__pycache__/__init__.cpython-312.pyc" check $P/pycache-normalization test -s "$PKG/__pycache__/normalization.cpython-312.pyc" check $P/pycache-unicode test -s "$PKG/__pycache__/unicode.cpython-312.pyc" # THE TEST SUBPACKAGE AND ITS CORPUS, which ship because of `find_packages()' # and `global-include *.txt' rather than because anyone chose to ship them. They # are asserted because this suite DEPENDS on the corpus being here: without it # the conformance cases below have nothing to read, and a release that stopped # shipping it would turn this suite from a conformance test into a spot check # without anything else changing. check $P/test-subpackage test -d "$PKG/test" check $P/fixture-present test -s "$FIXTURE" # The exact size, so a truncated copy is caught before it is parsed -- a corpus # cut short parses cleanly and simply stops testing. check_eq $P/fixture-size "2625136" \ "$(wc -c "$FIXTURE" 2>/dev/null | awk '{print $1}')" # NO COMPILED EXTENSION ANYWHERE THE PACKAGE PUT ANYTHING. See the header: on # this image a .so could not be loaded at all, so this is the case that fails a # version bump which grew one -- at install rather than at `import'. # # The list is copied to tmpfs first and never redirected from the rootfs: a # top-level stdin redirect on the virtiofs root makes dup2(2) return 513 and ash # EXITS, ending the suite silently (brintos/linux#73). cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # EVERY REGULAR FILE THE PACKAGE SHIPS IS PYTHON, ITS METADATA, OR THE CORPUS. # The case above names the failure that is expected; this one catches the shape # nobody predicted -- a binary, a .pyd, a stray data file -- without having to # enumerate what it might be. The corpus is the single exemption and it is # exempted BY NAME, so a package that grew a second data file is reported rather # than waved through. # # A dpkg .list names DIRECTORIES too, and without a trailing slash, so a # name-only filter would report every directory in the package as an unexpected # shape; each entry is therefore tested for being a regular file first. _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in "$FIXTURE") continue ;; *.py|*.pyc) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" # ----------------------------------------------------------- the dist-info --- # The installed metadata. The package carries its own `__version__' AND its own # `UNICODE_VERSION', so the dist-info is one of several independent statements # rather than the only one; `version-agrees' below is what compares two of them. check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 15.1.0" cat "$DI/METADATA" # top_level.txt names the ONE importable this package installs. It is what an # `import pyunormalize' has to match, and a wheel that renamed it would change # this file first. check_out $P/dist-info-top-level "pyunormalize" cat "$DI/top_level.txt" # PURE PYTHON, AS THE WHEEL ITSELF DECLARES IT. `Root-Is-Purelib: true' plus the # `py3-none-any' tag is the packaging-side statement of the header's claim -- no # platform tag, no extension module, nothing that could need the absent dynamic # loader. check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag "Tag: py3-none-any" cat "$DI/WHEEL" # The upstream licence text ships beside the metadata, which is where this # recipe's LIC_FILES_CHKSUM points. check $P/dist-info-license test -s "$DI/LICENSE" # RECORD lists the corpus with its own sha256 and length, which is a statement # by the WHEEL that the file this suite reads is the file that was packaged. check_out $P/dist-info-record-fixture \ "pyunormalize/test/data/NormalizationTest.txt" cat "$DI/RECORD" # ------------------------------------------------------------- the package --- # dpkg KNOWS the package. Every case above would look identical if some other # package had installed the same files, so this is the one that says # python3-pyunormalize is a real package this image installed rather than an # IMAGE_INSTALL line that resolved to nothing. # # The stanza is read from a tmpfs copy for the same reason as the file list, and # the anchor is exact -- an unanchored match would also take the -dev or -dbg # stanza of the same family. sed -n '/^Package: python3-pyunormalize$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # THE RUNTIME DEPENDENCY THE RECIPE DECLARES, as dpkg recorded it -- and here it # is the WHOLE of it. The package imports nothing outside itself, so the built # package depends on python3-core alone; that is why this port waits on no other # package, and this is the case that would say so if a release grew a # third-party requirement. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_eq $P/pkg-depends-exact "Depends: python3-core" "$(cat "$W/dep")" # ---------------------------------------------------------- the interpreter --- # THE IMPORT, ON ITS OWN, BEFORE THE PROBE. If pyunormalize cannot be imported # then every probe case below reds at once and each names a different symptom; # this case is what says the cause is the import. It is also the cheapest # possible statement of the deliverable, and it is not free of work: the import # runs `_make_dictionaries()', which transitively closes 5857 decomposition # mappings before the first caller sees the module. check $P/import python3 -c 'import pyunormalize' # ...AND THE IMPORT SAYS NOTHING WHILE DOING IT. Two things could speak here. # `__init__.py' raises SystemExit if `unicode.py's version disagrees with its # own, so silence is a second statement of `table-unicode-version' below, # reached along a different path. And the package ships precompiled .pyc files # written with CHECKED-HASH invalidation (MEASURED off the built .deb: pyc # header flags = 3, i.e. hash-based with check_source set), so the 503 KB source # is hashed and never recompiled. A red here means the shipped .pyc was # REJECTED -- a stale hash, a python3 minor bump that changed the magic, or a # missing __pycache__ -- and unicode.py is being compiled from source on every # import, which on this arm is not a small thing. # # MEASURED SILENT on the control under CPython 3.12.3, which is this image's own # interpreter version. check_eq $P/import-quiet "" \ "$(python3 -c 'import pyunormalize' 2>&1)" # ---------------------------------------------------------------- the probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present # ONE interpreter start for everything that follows. bst_probe bounds it and # keeps BOTH streams: a probe that dies early otherwise leaves every case below # reporting got:'' and naming a symptom each, with the actual traceback nowhere. # The bound is 1500s against 1.6s measured on the control -- generous because a # python3 start on this arm is tens of seconds before any work begins, and # because an overrun reported HERE names the last line the probe printed, where # a suite-timeout would cost the whole batch its round and say nothing. bst_probe $P/probe-run "$W/probe" 1500 python3 "$PROBE" OUT="$W/probe" # Read a single key. The probe emits each key at most once, so head -n 1 is # defensive rather than load-bearing; the `^' and `=' anchors keep `conf_nfc_*' # distinguishable from `conf_nfkc_*', which share a prefix. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE EVIDENCE PLANE. If the probe emitted nothing, every case below is reading # an absent file: each would red with got:'' and the set would read exactly like # a broken library, when what actually happened is that no measurement was # taken. The word `unscorable' is PARSED -- brintos_local_verify re-measures a # suite that prints it on a fresh guest rather than trusting a shared boot's # numbers (BRINTOS-1644) -- so it goes in this structural check, not a comment. # # The cases below still print their reds: withdraw, do not hide. This case is # what says what those reds measure. if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # ...and it ran to the END. The conformance sweep is the longest single stretch # of work in this suite, so a probe that ran out of memory or was killed part # way would leave the identity cases green and everything after the corpus # reading got:'' -- which looks like a truncated library rather than a truncated # run. This is the case that tells those apart. check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "112" "$(pv emitted_count)" # ------------------------------------------------------------- the identity --- # The package answered out of the directory this image installed, not from a # stray copy on sys.path. This is what makes the payload cases above and the # behaviour cases below statements about the SAME pyunormalize. check_eq $P/module-answered "$PKG/__init__.py" "$(pv module_file)" check_eq $P/is-package "yes" "$(pv is_package)" # THE VERSION, FROM BOTH PLACES, AND THEN AGAINST EACH OTHER. The package's own # `__version__' (out of _version.py) and the installed dist-info read through # importlib.metadata are independent statements; `version-agrees' is the one # that catches a package that landed beside a dist-info describing some other # release, which either case alone would pass. check_eq $P/version-attr "15.1.0" "$(pv version_attr)" check_eq $P/dist-version "15.1.0" "$(pv dist_version)" check_eq $P/version-agrees "$(pv version_attr)" "$(pv dist_version)" # ---------------------------------------------------- THE UNICODE VERSION --- # See the header: this is the product, not a version string. Three independent # statements of it, and the identity between two of them. check_eq $P/ucd-version "15.1.0" "$(pv ucd_version)" check_eq $P/unicode-version "15.1.0" "$(pv unicode_version)" # `UNICODE_VERSION = UCD_VERSION = "15.1.0"' binds ONE object to both names. A # release that let them drift apart is caught here rather than by whichever # caller happened to read the stale one. check_eq $P/ucd-is-unicode "yes" "$(pv ucd_is_unicode)" # The generated data module's own claim, which `__init__.py' compares against # its own at import and exits on. This is the third statement and the one that # is about the TABLE rather than about the package. check_eq $P/table-unicode-version "15.1.0" "$(pv table_unicode_version)" # THE INTERPRETER'S OWN DATABASE, WHICH IS A DIFFERENT VERSION. This is not an # assertion about pyunormalize at all -- it is the fact that makes the package # worth installing, and it is the baseline the `ud-*' cross-check below is # scored against. A python3 bump that moved this number would change what that # cross-check means, and this case is where that becomes visible instead of # surfacing as an unexplained divergence. check_eq $P/interp-unidata "15.0.0" "$(pv interp_unidata)" check_eq $P/py-version "3.12" "$(pv py_version)" # The public surface, in order. `from pyunormalize import *' is what a caller # writes, and this is what that would give them. check_eq $P/all-names \ "NFC,NFD,NFKC,NFKD,normalize,UCD_VERSION,UNICODE_VERSION,__version__" \ "$(pv all_names)" check_eq $P/callable-nfc "yes" "$(pv callable_NFC)" check_eq $P/callable-nfd "yes" "$(pv callable_NFD)" check_eq $P/callable-nfkc "yes" "$(pv callable_NFKC)" check_eq $P/callable-nfkd "yes" "$(pv callable_NFKD)" check_eq $P/callable-normalize "yes" "$(pv callable_normalize)" # --------------------------------------------------------------- the tables --- # See the header. These are the cheap statement of the generated table's shape; # the conformance cases below are the expensive one, and they fail apart -- a # table short by one entry moves a count here AND reds a record there, while a # table with the RIGHT number of WRONG entries only reds there. check_eq $P/n-ccc "922" "$(pv n_ccc)" check_eq $P/n-comp-excl "81" "$(pv n_comp_excl)" check_eq $P/qc-keys "nfc_maybe,nfc_no,nfd_no,nfkc_maybe,nfkc_no,nfkd_no" \ "$(pv qc_keys)" check_eq $P/qc-nfc-maybe "111" "$(pv qc_nfc_maybe)" check_eq $P/qc-nfc-no "1120" "$(pv qc_nfc_no)" check_eq $P/qc-nfd-no "13233" "$(pv qc_nfd_no)" check_eq $P/qc-nfkc-maybe "111" "$(pv qc_nfkc_maybe)" check_eq $P/qc-nfkc-no "4928" "$(pv qc_nfkc_no)" check_eq $P/qc-nfkd-no "17029" "$(pv qc_nfkd_no)" # The three dictionaries `normalization.py' BUILDS at import out of the shipped # `_DECOMP', by transitively closing every decomposition mapping. These sizes # are a statement about that closure having RUN, not merely about the literal: # _CDECOMP and _KDECOMP start from the same mapping and end at different sizes # because compatibility decompositions are the strictly larger set. check_eq $P/n-cdecomp "2061" "$(pv n_cdecomp)" check_eq $P/n-kdecomp "5857" "$(pv n_kdecomp)" check_eq $P/n-precomp "1022" "$(pv n_precomp)" # `del _DECOMP' in normalization.py unbinds ITS OWN name, not unicode.py's. A # reader working from that line alone would conclude the source table had been # freed; it has not, and this is what the image actually carries. check_eq $P/decomp-on-data-module "yes" "$(pv decomp_on_data_module)" # --------------------------------------------------- the documented forms --- # Upstream's own docstring examples, all four forms of each. These are what a # user reads before writing any code. # # Every value is CODE POINTS, never characters: the guest runs under the C # locale and whether CPython coerces stdout to UTF-8 there is a property of the # platform rather than of this package, so a probe that printed the strings # would red on the coercion instead of on pyunormalize. check_eq $P/elev-nfc "00E9 006C 00E8 0076 0065" "$(pv elev_nfc)" check_eq $P/elev-nfd "0065 0301 006C 0065 0300 0076 0065" "$(pv elev_nfd)" check_eq $P/elev-nfkc "00E9 006C 00E8 0076 0065" "$(pv elev_nfkc)" check_eq $P/elev-nfkd "0065 0301 006C 0065 0300 0076 0065" "$(pv elev_nfkd)" # HANGUL, where composition and decomposition are ARITHMETIC rather than # table-driven -- a separate implementation inside the same package, and the # reason 11172 syllables cost no table space at all. check_eq $P/hangul-nfc "D55C AD6D" "$(pv hangul_nfc)" check_eq $P/hangul-nfd "1112 1161 11AB 1100 116E 11A8" "$(pv hangul_nfd)" check_eq $P/hangul-nfkc "D55C AD6D" "$(pv hangul_nfkc)" check_eq $P/hangul-nfkd "1112 1161 11AB 1100 116E 11A8" "$(pv hangul_nfkd)" # THE COMPATIBILITY/CANONICAL SPLIT, which is the distinction a caller is most # likely to get wrong. A ligature and a parenthesized digit are UNCHANGED under # the canonical forms and rewritten under the compatibility ones; a library that # confused the two would pass every accented-Latin case above. check_eq $P/ffi-nfc "FB03" "$(pv ffi_nfc)" check_eq $P/ffi-nfd "FB03" "$(pv ffi_nfd)" check_eq $P/ffi-nfkc "0066 0066 0069" "$(pv ffi_nfkc)" check_eq $P/ffi-nfkd "0066 0066 0069" "$(pv ffi_nfkd)" check_eq $P/paren1-nfc "2474" "$(pv paren1_nfc)" check_eq $P/paren1-nfd "2474" "$(pv paren1_nfd)" check_eq $P/paren1-nfkc "0028 0031 0029" "$(pv paren1_nfkc)" check_eq $P/paren1-nfkd "0028 0031 0029" "$(pv paren1_nfkd)" check_eq $P/sq2-nfc "00B2" "$(pv sq2_nfc)" check_eq $P/sq2-nfd "00B2" "$(pv sq2_nfd)" check_eq $P/sq2-nfkc "0032" "$(pv sq2_nfkc)" check_eq $P/sq2-nfkd "0032" "$(pv sq2_nfkd)" # THE TWO SUBJECTS WHOSE FOUR FORMS ARE ALL DIFFERENT, which is what makes them # worth more than any of the above: no pair of forms can be confused for each # other without one of these eight values moving. Upstream's own unit test picks # the same two characters for the same reason. check_eq $P/long-s-nfc "1E9B 0323" "$(pv long_s_nfc)" check_eq $P/long-s-nfd "017F 0323 0307" "$(pv long_s_nfd)" check_eq $P/long-s-nfkc "1E69" "$(pv long_s_nfkc)" check_eq $P/long-s-nfkd "0073 0323 0307" "$(pv long_s_nfkd)" check_eq $P/upsilon-nfc "03D3" "$(pv upsilon_nfc)" check_eq $P/upsilon-nfd "03D2 0301" "$(pv upsilon_nfd)" check_eq $P/upsilon-nfkc "038E" "$(pv upsilon_nfkc)" check_eq $P/upsilon-nfkd "03A5 0301" "$(pv upsilon_nfkd)" # ---------------------------------------------------------- the dispatcher --- # `normalize(form, s)' is a dict lookup on the form NAME, so it is a second, # independent entry point that could rot while the four functions stayed right. # `dispatch-agrees' runs all four forms of all seven subjects through both. check_eq $P/dispatch-agrees "yes" "$(pv dispatch_agrees)" check_eq $P/dispatch-forms "NFC,NFD,NFKC,NFKD" "$(pv dispatch_forms)" # AN INVALID FORM RAISES -- AND THE TYPE IS PART OF THE CONTRACT. The dispatch # is a bare `_normalization_forms[form]', so what a caller catches is KeyError, # where `unicodedata.normalize' raises ValueError for the same mistake. The two # libraries are NOT drop-in for each other here, and a caller swapping one for # the other is the person these two cases are written for. check_eq $P/bad-form "KeyError" "$(pv bad_form)" # ...including the lower-case spelling, which is the likeliest near miss and is # NOT accepted. check_eq $P/lower-form "KeyError" "$(pv lower_form)" # ----------------------------------------------------------- the internals --- # The three algorithm stages, driven directly with upstream's own unit-test # vectors. The public forms compose all three, so a red in exactly one of these # beside green forms would be a contradiction worth seeing; a red here beside # red forms says WHICH stage broke. check_eq $P/dec-c0 "0041 0300" "$(pv dec_c0)" check_eq $P/dec-be-k "0033 2044 0034" "$(pv dec_be_k)" check_eq $P/dec-hangul "1112 1175 11BD" "$(pv dec_hangul)" check_eq $P/reorder "017F 0323 0307" "$(pv reorder)" check_eq $P/compose "0105 0302 0301" "$(pv compose)" # THE R2 CASE FROM THE STANDARD ITSELF (Unicode 15.0, p. 140): a starter must go # on being checked against later combining marks across THREE intervening marks # that compose with nothing. An implementation that stopped at the first # non-composing mark passes every other case in this file and fails this one. check_eq $P/compose-r2 "017A 0335 0327 0324" "$(pv compose_r2)" # ================================================================ CONFORMANCE # See the header: the Unicode Consortium's own corpus, read off the image out of # this very package. Everything above is a sample; this is the whole function. check_eq $P/fixture-path "$FIXTURE" "$(pv fixture_path)" check_eq $P/fixture-bytes "2625136" "$(pv fixture_bytes)" check_eq $P/fixture-lines "19129" "$(pv fixture_lines)" # THE CORPUS AND THE CODE MUST BE THE SAME UNICODE RELEASE. The file's first # line names its version; a package shipping 15.1.0 tables beside a 14.0.0 # corpus would pass most records and quietly stop testing what it claims to. check_eq $P/fixture-version-in-head "yes" "$(pv fixture_version_in_head)" check_eq $P/records "19074" "$(pv records)" # The four @Parts have different meanings, so their tallies are asserted apart: # Part0 is hand-picked specific cases, Part1 is the character-by-character set, # Part2 is canonical ordering and Part3 is the PRI #29 sequences. check_eq $P/parts "Part0:25,Part1:17029,Part2:1844,Part3:176" "$(pv parts)" check_eq $P/part1-chars "17029" "$(pv part1_chars)" # THE STRONGEST CHECK IN THIS FILE, AND IT COSTS NOTHING. Part 1 of the corpus # is exactly the set of characters with NFKD_Quick_Check=No, and `unicode.py's # `nfkd_no' table is generated separately from it. Two independently produced # artefacts agreeing on a 17029-member set is not something a truncation on # either side survives. check_eq $P/part1-is-nfkd-no "yes" "$(pv part1_is_nfkd_no)" # THE FOUR UAX #15 CONFORMANCE CONDITIONS over all 19074 records, scored per # form. A library that got compatibility decomposition wrong reds NFKC and NFKD # and leaves NFC and NFD green, rather than reporting one undifferentiated # failure -- which is why these are four cases and not one. # # The probe also emits `conf_
_first', the line number of the first failing # record; it is not asserted because it has no expected value, and it is what a # red is investigated from -- the corpus names the character on that line. check_eq $P/conf-nfc-fail "0" "$(pv conf_nfc_fail)" check_eq $P/conf-nfd-fail "0" "$(pv conf_nfd_fail)" check_eq $P/conf-nfkc-fail "0" "$(pv conf_nfkc_fail)" check_eq $P/conf-nfkd-fail "0" "$(pv conf_nfkd_fail)" # THE CHARACTER-BY-CHARACTER SWEEP -- the half of conformance the records cannot # reach. Every code point NOT listed in Part 1 must be unchanged under all four # forms; that is what a spurious or shifted table entry breaks, and no record in # the corpus would notice. check_eq $P/sweep-checked "1097083" "$(pv sweep_checked)" check_eq $P/sweep-fail "0" "$(pv sweep_fail)" # IDEMPOTENCE, which no record in the corpus states directly: normalizing an # already-normalized string returns it unchanged, under every form. check_eq $P/idempotent-fail "0" "$(pv idempotent_fail)" # THE HANGUL ALGORITHM over all 11172 modern syllables -- arithmetic rather than # table-driven, so a separate implementation inside the same package that the # corpus exercises only part of. Every syllable decomposes to 2 or 3 jamo and # recomposes to itself. check_eq $P/hangul-syllables "11172" "$(pv hangul_syllables)" check_eq $P/hangul-fail "0" "$(pv hangul_fail)" # ------------------------------------------- against the interpreter's own --- # THE CROSS-CHECK, AND WHAT IT IS EVIDENCE FOR. # # The image's python3 carries its own Unicode database at its OWN version -- # 15.0.0, asserted above -- and the whole reason this package exists is that the # two need not agree. So a divergence here would be a VERSION DIFFERENCE, not a # failure of either side. # # Zero is nonetheless the right expectation, and the reason is specific rather # than optimistic: Unicode 15.1 added no character with a decomposition mapping # and changed no combining class. Its additions are CJK Ideograph Extension I, # which decompose to nothing. So 15.0.0 and 15.1.0 must agree on every record in # this corpus, and MEASURED on the control under exactly this interpreter # version, they do. # # A red here therefore says one of two things, and the cases above are what tell # them apart: either the image's python3 moved to a Unicode release that DOES # differ -- in which case `interp-unidata' reds too and this is expected -- or # one of the two implementations is wrong. check_eq $P/ud-ran "yes" "$(pv ud_ran)" check_eq $P/ud-nfc-diff "0" "$(pv ud_nfc_diff)" check_eq $P/ud-nfd-diff "0" "$(pv ud_nfd_diff)" check_eq $P/ud-nfkc-diff "0" "$(pv ud_nfkc_diff)" check_eq $P/ud-nfkd-diff "0" "$(pv ud_nfkd_diff)" # ------------------------------------------------------------- the corners --- check_eq $P/empty "" "$(pv empty)" # U+0000 SURVIVES WHEN NOTHING IS COMPOSED... check_eq $P/nul-alone "0000" "$(pv nul_alone)" check_eq $P/nul-ascii "0061 0000 0062" "$(pv nul_ascii)" check_eq $P/nul-precomposed "00E0 0000" "$(pv nul_precomposed)" # ...AND IS SILENTLY DELETED WHEN SOMETHING IS. See the header: `_compose' ends # with `[*filter(None, elems)]', which discards every falsy element, and 0 is # falsy. This is upstream data loss in 15.1.0, not a platform gap. # # These two cases record what 15.1.0 DOES. A release that fixed it turns them # red, which is the correct signal and the WANTED direction -- it would mean the # behaviour changed. check_eq $P/nul-composed-nfc "00E0" "$(pv nul_composed_nfc)" check_eq $P/nul-composed-nfkc "00E0" "$(pv nul_composed_nfkc)" # NFD and NFKD never compose, so they keep it. That asymmetry is the evidence # for the cause, which is why it is asserted rather than assumed. check_eq $P/nul-composed-nfd "0061 0300 0000" "$(pv nul_composed_nfd)" check_eq $P/nul-composed-nfkd "0061 0300 0000" "$(pv nul_composed_nfkd)" # ...and the same input through the interpreter's own unicodedata, which KEEPS # it. This is the case that says the expectation above is a property of # pyunormalize rather than something about U+0000 on this platform. check_eq $P/nul-composed-ud "00E0 0000" "$(pv nul_composed_ud)" rm -rf "$W"