. "$BST_LIB" # BST-TIMEOUT: 1200 # THREE python3 starts, and they are the whole cost. Every # # assertion below either reads the filesystem/dpkg with # # shell builtins or greps the single probe run's output. # # The three starts are `import', the probe, and the one # # UNICODE_VERSION case, which has to be a separate process # # because the `auto' lookup is lru_cached and cannot be # # re-read inside a running interpreter (see `env-version'). # # This package ships no script, so there is no fourth. # # # # The probe's own work is a table of ~50 width calls and one # # walk of ~6000 table ranges -- microseconds beside an # # interpreter start on this arm. # # # # 1200 is 99-python3-jstyleson's and 99-python3-jsonpointer's # # figure, for the same reason: the start dominates. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-wcwidth 0.2.13 (poky/meta recipes-devtools/python/python3-wcwidth_0.2.13.bb, # `inherit pypi setuptools3 ptest', PyPI sdist) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE MEASURES, AND WHY IT IS NOT len(). # ============================================================================ # # `wcwidth(c)' returns how many terminal COLUMNS one character occupies, and # `wcswidth(s)' the same for a string. That number is neither the byte count nor # the code-point count, and the three ways it differs are the three classes this # suite is built around: # # * a CJK ideograph occupies TWO cells -- `w-cjk', `w-hiragana', `w-hangul'; # * a combining mark occupies ZERO -- `w-comb-acute', and `s-combining', where # "e" + COMBINING ACUTE is two code points and ONE column; # * a C0 or C1 control character has an INDETERMINATE effect, for which the # POSIX signature returns -1 -- `w-soh' and the three `s-c0-*' cases. # # The -1 is the case naive width code gets wrong, and `wcswidth' propagates it: # ONE control anywhere in the string makes the whole answer -1 rather than the # width of the rest. It is asserted in three positions -- first, middle and last # -- because an implementation that checked only the leading character would pass # one of them. # # ============================================================================ # THE PACKAGE IS ITS DATA, WHICH IS WHY THIS IS NOT AN IMPORT CHECK. # ============================================================================ # # wcwidth.py is 14 KB of binary search over 467 KB of GENERATED Unicode tables -- # table_zero.py alone is 359 KB -- and each table carries a separate range list # for each of NINETEEN Unicode version levels. So the deliverable is not "does # the module import", it is "did the tables land, whole, for every level". # # Three groups of cases say that, and no one of them would do it alone: # # * `zero-ranges-*' / `wide-ranges-*' pin the table SIZES at the latest level # and at the earliest. Equal sizes at both would be the signature of a build # that shipped one table nineteen times. # * `table-monotonic' walks EVERY range of EVERY level checking the invariant # `_bisearch' relies on -- sorted, non-overlapping, start <= end. A binary # search over an unsorted table returns a WRONG ANSWER rather than an error, # which is the one failure a behavioural case sampling thirty code points # could not see. This is the part of upstream's own # tests/test_table_integrity.py that runs without pytest. # * `v410-emoji' / `v900-emoji' ask for U+1F600's width at two levels and get # 1 and 2, because GRINNING FACE became wide in Unicode 9.0.0. That is what # says the older tables are real data rather than aliases of the latest. # # ============================================================================ # THE CONTROL IS THE STDLIB, NOT A TRANSCRIBED LITERAL. # ============================================================================ # # Every literal below was written down by a person, so `ctl-eaw' and `ctl-comb' # are the two cases that would still mean something if all of them had been # transcribed wrong: the probe cross-checks wcwidth against the INTERPRETER'S OWN # Unicode database -- `unicodedata.east_asian_width' for the two-cell claim, # `unicodedata.combining' for the zero-cell claim -- computed on target over 22 # code points, and emits the disagreements. The expected value is the empty # string. # # `unicodedata' is a BUILTIN here rather than a lib-dynload .so -- this image's # python3 is one static binary with an empty lib-dynload (BRINTOS-1476), and # 99-python3's `builtin-modules' case asserts unicodedata is among the modules # linked in -- so the control costs nothing and cannot fail for want of an # extension module. # # The samples are long-stable code points on purpose: the control interpreter # carries a different UCD level from the image's, and a character whose class had # moved between them would red on version skew rather than on a defect. Nothing # sampled has moved since Unicode 4.1. `unidata-version' records the image's # level so a future skew is legible rather than mysterious. # # ============================================================================ # WHY IT PORTS AT ALL: PURE PYTHON, NO C, NOTHING BUT THE STDLIB. # ============================================================================ # # This image's python3 is ONE STATIC BINARY with an EMPTY lib-dynload and no # dynamic loader (BRINTOS-1476; 99-python3 asserts both directly). A module # carrying a compiled extension could not be imported here -- not "would be # slow", could not be imported. # # wcwidth ships NO C: no .c, .pyx or .so in the sdist, and setup.py declares # `packages' with no ext_modules at all. Its imports are `os', `sys', # `warnings' and `functools' -- all sub-packages of the ported python3 # (BRINTOS-1476), which is why the built package's control stanza reads # `Depends: python3-core' and nothing else and this port waits on no other # package. `pkg-no-so' below is a REGRESSION TRIPWIRE on a version bump rather # than a statement about today. # # It IS A PACKAGE, not a single module: setup.py declares `packages=['wcwidth']', # so what lands in site-packages is a DIRECTORY of six modules -- the opposite # shape from 99-python3-jstyleson's one file, and what lets the tables be # separate modules. `is-package' asserts it, because a release that collapsed the # shape would still import and still pass every behavioural case below. # # ============================================================================ # THE UPSTREAM ptest SUITE IS NOT RUN HERE, AND THAT IS A MEASURED DECISION. # ============================================================================ # # The recipe does `inherit ptest' and ships a run-ptest that calls # `pytest --automake', so the obvious reading of this port is "run upstream's # tests". It cannot be done on this image, for two reasons that compound: # # 1. This distro sets DISTRO_FEATURES without "ptest" # (conf/distro/brintos.conf), so # ptest.bbclass never adds ${PN}-ptest to PACKAGES at all. MEASURED: the # recipe-scope build of this port produced python3-wcwidth, # python3-wcwidth-dev and python3-wcwidth-dbg, and NO -ptest deb. # 2. Producing one would put its RDEPENDS on the image -- python3-pytest and # python3-unittest-automake-output. pytest's own RDEPENDS names sixteen # packages, of which SEVEN are not on this image and are not this ticket's: # python3-atomicwrites, python3-attrs, python3-importlib-metadata, # python3-pathlib2, python3-pluggy, python3-py and python3-setuptools. # # Landing seven unrelated packages as a side effect of a pure-Python port is the # shape this image recipe already refuses for acl-dev's RRECOMMENDS on attr. So # the cases below are the only statement about this package on this image -- and # `table-monotonic' deliberately reproduces the one part of upstream's suite that # needs no test framework, so the decision costs coverage rather than assurance. # # ============================================================================ # ONE UPSTREAM LIMITATION IS ASSERTED AS ONE. # ============================================================================ # # POSIX wcswidth() bounds its `n' argument by the string. 0.2.13 indexes the # string directly, so `wcswidth("ab", 10)' raises IndexError out of a width # calculation rather than returning 2. `n-beyond' records that, with `n-0' # through `n-4' as the controls that show `n' is otherwise honoured. It is # 0.2.13's behaviour on x86-64 and it is UPSTREAM'S, not this platform's; it is # asserted so the port ships a statement about what it does rather than a # silence, and so a release that fixed it announces itself here instead of # somewhere a caller would find it. # # ============================================================================ # WHERE THE ASSERTED VALUES COME FROM. # ============================================================================ # # Every literal below was measured on x86-64 against the sdist the recipe's # SRC_URI sha256 pins (72ea0c06...2b5, verified by digest before measuring), and # again against the bytes unpacked from the built # python3-wcwidth_0.2.13-r0_wasm32.deb -- the six shipped .py files are # byte-identical between the two, and the probe's output over them differs only # in `module_file'. The control interpreter is CPython 3.10 and the image ships # 3.12; that gap is stated here and in the test plan rather than papered over, # and it is the reason `ctl-eaw'/`ctl-comb' compare against the target's own UCD # instead of against a number from the control. The on-target result is not yet # measured -- the batch verify is this port's first run on wasm32-hwjs. P=python3-wcwidth # THE CANARY, which asserts nothing about python3-wcwidth. # # A suite's FIRST BST result line is lost between the guest's printf and the # host's read -- per-suite, deterministic, and NOT a defect in the host tool # (BRINTOS-1149 established that and filed it as linux-in-the-browser#177). The # harness computes its pass count as total-minus-failures, so THE DROPPED LINE # IS COUNTED AS A PASS. A nothing-case therefore stands in that position and the # eaten line costs nothing. It sits ahead of the `require' guards deliberately, # because a `require' that skips would emit its skip as the first line and be # eaten too. # # Remove this case when linux-in-the-browser#177 is fixed. bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKGDIR=$SITE/wcwidth DI=$SITE/wcwidth-0.2.13.dist-info PKGLIST=/var/lib/dpkg/info/python3-wcwidth.list PROBE=$BST_DATA/python3-wcwidth-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-wcwidth.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------- the payload --- # Where the package has to land. `import wcwidth' further down is the functional # statement; these are the ones that say WHICH copy answered, so a package that # imported out of some other path on sys.path is distinguishable from one that # imported out of what this image installed. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKGDIR" check $P/module-init test -f "$PKGDIR/__init__.py" check $P/module-core test -f "$PKGDIR/wcwidth.py" # THE THREE GENERATED TABLES, AS FILES. `zero-ranges-latest' and its neighbours # below assert what is INSIDE them; these assert that the files shipped at all, # so a build that dropped one is distinguishable from one that shipped an empty # table. table_zero.py is the large one -- 359 KB -- and it is the package's # actual deliverable. check $P/table-zero test -s "$PKGDIR/table_zero.py" check $P/table-wide test -s "$PKGDIR/table_wide.py" check $P/table-vs16 test -s "$PKGDIR/table_vs16.py" check $P/table-versions test -s "$PKGDIR/unicode_versions.py" # NO COMPILED EXTENSION ANYWHERE THE PACKAGE PUT ANYTHING. See the header: on # this image a .so could not be loaded at all, so this is the case that fails a # version bump which grew one -- loudly, here, instead of at `import' later. # # The list is copied to tmpfs first and never redirected from the rootfs: a # top-level stdin redirect on the virtiofs root makes dup2(2) return 513 and ash # EXITS, ending the suite silently (brintos/linux#73). cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # EVERY REGULAR FILE THE PACKAGE SHIPS IS PYTHON OR ITS METADATA. The case above # names the failure that is expected; this one catches the shape nobody predicted # -- a binary, a .pyd, a stray data file -- without having to enumerate what it # might be. This package ships no script at all, so unlike 99-python3-jsonpointer's # equivalent there is no executable to exempt: anything outside site-packages is # unexpected here by construction. # # A dpkg .list names DIRECTORIES too, and without a trailing slash, so a # name-only filter would report every directory in the package as an unexpected # shape; each entry is therefore tested for being a regular file first. _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" # ----------------------------------------------------------- the dist-info --- # The installed metadata. Unlike python3-jstyleson this module DOES carry its own # `__version__', so the dist-info is the SECOND of two independent statements of # the version rather than the only one, and `version-agrees' below is what # compares them. check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-version "Version: 0.2.13" cat "$DI/METADATA" # top_level.txt names the ONE importable this package installs. It is what an # `import wcwidth' has to match, and a wheel that renamed the package would # change it here first. check_out $P/dist-info-top-level "wcwidth" cat "$DI/top_level.txt" # ------------------------------------------------------------- the package --- # dpkg KNOWS the package. Every case above would look identical if some other # package had installed the same files, so this is the one that says # python3-wcwidth is a real package this image installed rather than an # IMAGE_INSTALL line that resolved to nothing. # # The stanza is read from a tmpfs copy for the same reason as the file list, and # the anchor is exact -- an unanchored match would also take the -dev or -dbg # stanza of the same family. sed -n '/^Package: python3-wcwidth$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" # THE RUNTIME DEPENDENCY THE RECIPE DECLARES, as dpkg recorded it -- and here it # is the WHOLE of it. wcwidth imports only os, sys, warnings and functools, so # the built package depends on python3-core and nothing else; that is why this # port waits on no other package, and this case is what would say so if a release # grew a third-party requirement. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-core "python3-core" cat "$W/dep" # NO -ptest PACKAGE IS INSTALLED, and that is asserted rather than left silent. # See the header: this distro's DISTRO_FEATURES carries no "ptest" so no -ptest # package is produced, and producing one would drag pytest's seven unported # dependencies onto the image. A future image that DID install one should have # this suite run it instead of quietly shipping an untested duplicate, and this # case is what says so. check_eq $P/no-ptest-package "absent" \ "$(if sed -n '/^Package: python3-wcwidth-ptest$/,/^$/p' /var/lib/dpkg/status 2>/dev/null | grep -q 'install ok installed'; then echo present; else echo absent; fi)" # ---------------------------------------------------------- the interpreter --- # THE IMPORT, ON ITS OWN, BEFORE THE PROBE. If wcwidth cannot be imported then # every probe case below reds at once and each names a different symptom; this # case is what says the cause is the import. It is also the cheapest possible # statement of the deliverable. check $P/import python3 -c 'import wcwidth' # ---------------------------------------------------------------- the probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present # ONE interpreter start for everything that follows. bst_probe bounds it and # keeps BOTH streams: a probe that dies early otherwise leaves every case below # reporting got:'' and naming a symptom each, with the actual traceback nowhere. # The bound is generous because a python3 start on this arm is tens of seconds # before any work begins. # # UNICODE_VERSION is cleared for this run. It is the environment variable the # `auto' default reads, so a value inherited from the guest's environment would # silently move every width answer below to a different table -- and the probe # reports what it saw in `env_unicode_version', which `probe-env-clean' asserts. bst_probe $P/probe-run "$W/probe" 900 env -u UNICODE_VERSION python3 "$PROBE" OUT="$W/probe" # Read a single key. The probe emits each key at most once, so head -n 1 is # defensive rather than load-bearing; the anchors keep `w_cjk' distinguishable # from `s_cjk', and `versions_last' from `versions_joined'. pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE EVIDENCE PLANE. If the probe emitted nothing, every case below is reading # an absent file: each would red with got:'' and the set would read exactly like # a broken library, when what actually happened is that no measurement was taken. # The word `unscorable' is PARSED -- brintos_local_verify re-measures a suite that # prints it on a fresh guest rather than trusting a shared boot's numbers # (BRINTOS-1644) -- so it goes in this structural check, not in a comment. # # The cases below still print their reds: withdraw, do not hide. This case is # what says what those reds measure. if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # ...and it ran to the END. Every key is emitted in file order, so a probe that # died two thirds of the way through would leave the earlier cases green and the # later ones reading got:'' -- which looks like a partial library rather than a # partial run. This is the case that tells those apart. check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-env-clean "" "$(pv env_unicode_version)" # The module answered out of the directory this image installed, not from a stray # copy on sys.path. This is what makes the payload cases above and the behaviour # cases below statements about the SAME wcwidth. check_eq $P/module-answered "$PKGDIR/__init__.py" "$(pv module_file)" # ...and it is a PACKAGE. See the header: a release that collapsed the six # modules into one file would still import and still pass every behavioural case. check_eq $P/is-package "yes" "$(pv is_package)" # THE VERSION, FROM BOTH PLACES, AND THEN AGAINST EACH OTHER. The module's own # `__version__' and the installed dist-info read through importlib.metadata are # independent statements; `version-agrees' is the one that catches a module that # landed beside a dist-info describing some other release, which either case # alone would pass. check_eq $P/version-attr "0.2.13" "$(pv version_attr)" check_eq $P/dist-version "0.2.13" "$(pv dist_version)" check_eq $P/version-agrees "$(pv version_attr)" "$(pv dist_version)" # THE PUBLIC API, as `__all__' declares it. This module DOES define `__all__', so # it is asserted directly rather than reconstructed from dir() -- it is the # package's own statement of what it exports, and a release that widened or # narrowed it says so here. check_eq $P/api-all "wcwidth,wcswidth,list_versions" "$(pv all_names)" for _n in wcwidth wcswidth list_versions; do check_eq $P/api-$(echo "$_n" | tr _ -) "yes" "$(pv has_$_n)" done # The lru_cache on `wcwidth' is load-bearing on this arm rather than incidental: # wcswidth calls it once per character, so a release that dropped it would be a # measurable regression in every table formatter downstream. check_eq $P/api-cache "yes" "$(pv has_cache_info)" # ------------------------------------------------------- the version levels --- # The list of Unicode levels is part of the PAYLOAD, not metadata about it: each # one names a separate shipped table. The count and the endpoints are asserted # separately from the joined list so a red says whether a level went missing or # the list drifted wholesale. check_eq $P/versions-count "19" "$(pv versions_count)" check_eq $P/versions-first "4.1.0" "$(pv versions_first)" check_eq $P/versions-last "15.1.0" "$(pv versions_last)" check_eq $P/versions-joined \ "4.1.0,5.0.0,5.1.0,5.2.0,6.0.0,6.1.0,6.2.0,6.3.0,7.0.0,8.0.0,9.0.0,10.0.0,11.0.0,12.0.0,12.1.0,13.0.0,14.0.0,15.0.0,15.1.0" \ "$(pv versions_joined)" # THE NEAREST-MATCH RULE, one case per branch of it. A caller may pass any # version string, so what happens to an unsupported one is part of the contract: # a value BETWEEN two levels rounds DOWN (4.9.9 -> 4.1.0), a short dotted form # matches on its leading pieces (8.0 -> 8.0.0), a value below every level clamps # to the earliest (1 -> 4.1.0), and `latest' resolves to the newest shipped. # These are upstream's own documented examples. check_eq $P/match-4990 "4.1.0" "$(pv match_499)" check_eq $P/match-80 "8.0.0" "$(pv match_80)" check_eq $P/match-1 "4.1.0" "$(pv match_1)" check_eq $P/match-latest "15.1.0" "$(pv match_latest)" check_eq $P/version-value "15,1,0" "$(pv version_value_1510)" # An UNPARSEABLE level warns and still answers with the latest. It does not # raise, and that is the promise worth pinning: the alternative failure mode is a # traceback out of a width calculation inside somebody else's terminal renderer. check_eq $P/match-bogus "15.1.0" "$(pv match_bogus)" check_eq $P/match-bogus-warns "yes" "$(pv match_bogus_warned)" # ------------------------------------------------------------- the tables --- # One range list per level, for each of the three tables. Nineteen keys in the # two large ones is the count the level list above demands. check_eq $P/zero-keys "19" "$(pv zero_keys)" check_eq $P/wide-keys "19" "$(pv wide_keys)" # The VS16 table is deliberately the odd one out: it exists only from 9.0.0, so # ONE key is correct and nineteen would be wrong. check_eq $P/vs16-keys "9.0.0" "$(pv vs16_keys)" # THE SIZES, at the latest level and at the earliest. Asserting only the latest # would pass on a build that shipped one table nineteen times; the pair is what # says the levels differ. The numbers are large and specific because they are # generated data -- 327 zero-width ranges and 121 wide ranges at 15.1.0, against # 145 and 34 at 4.1.0 -- and a truncated table_zero.py shows up here first. check_eq $P/zero-ranges-latest "327" "$(pv zero_ranges_latest)" check_eq $P/wide-ranges-latest "121" "$(pv wide_ranges_latest)" check_eq $P/zero-ranges-410 "145" "$(pv zero_ranges_410)" check_eq $P/wide-ranges-410 "34" "$(pv wide_ranges_410)" check_eq $P/vs16-ranges "113" "$(pv vs16_ranges)" # THE STRUCTURAL INVARIANT, over every range of every level. See the header: # `_bisearch' is a binary search, so an unsorted or overlapping table returns a # WRONG ANSWER rather than an error -- the one failure a table of thirty sampled # code points cannot see. This walks all of them and names the level that broke. check_eq $P/table-monotonic "ok" "$(pv table_monotonic)" # ------------------------------------------------- wcwidth(): ONE CHARACTER --- # ONE CELL: ordinary printable characters, including the ones that look like they # might not be. NBSP is a space and occupies a cell; a half-width katakana is # east-asian and occupies ONE, which is the case that separates "east-asian" from # "wide"; U+2588 FULL BLOCK and U+263A are drawing and symbol characters that # render narrow. `greek-alpha' is the EAST ASIAN AMBIGUOUS class, which upstream # resolves to 1 by policy -- a deliberate choice documented in wcwidth.py's # header, not an accident, and a release that changed it would be a visible # regression for every CJK terminal. for _c in space letter_a tilde nbsp eacute box greek_alpha snowman halfwidth_ka; do check_eq $P/w-$(echo "$_c" | tr _ -) "1" "$(pv w_$_c)" done # TWO CELLS: the east-asian table answered. A CJK ideograph, a hiragana, a # fullwidth latin letter, a hangul syllable, an emoji, and a plane-2 ideograph -- # the last of which is a NON-BMP code point, so a build whose tables had been # truncated at U+FFFF would red here and nowhere else. for _c in cjk hiragana fullwidth_a hangul emoji_grin cjk_extb; do check_eq $P/w-$(echo "$_c" | tr _ -) "2" "$(pv w_$_c)" done # ZERO CELLS: the zero-width table answered. A combining acute, a zero-width # space, a zero-width joiner, a variation selector, a soft hyphen and a # devanagari virama -- six different reasons for the same answer, so a table that # lost one class is distinguishable from one that lost the lot. for _c in comb_acute zwsp zwj vs16 soft_hyphen virama; do check_eq $P/w-$(echo "$_c" | tr _ -) "0" "$(pv w_$_c)" done # NOT A WIDTH AT ALL: -1, the POSIX "not printable" signal. This is the answer # naive width code gets wrong, and it is the reason the return type is signed. # Both control ranges are covered -- C0 below 0x20, DEL at 0x7F, and C1 from 0x80 # to 0x9F -- because they are three separate tests inside the function. TAB and # LF are here rather than among the printables on purpose: they are C0 characters # whose effect on the cursor is real but is not a WIDTH, and a caller who assumed # TAB was 1 or 8 is the audience for this case. for _c in soh bel tab lf esc del c1_80 c1_9f; do check_eq $P/w-$(echo "$_c" | tr _ -) "-1" "$(pv w_$_c)" done # THE TWO THAT ARE NEITHER. NUL is a control character and yet answers 0, not -1 # -- it has no printable effect, which is a different statement from having an # indeterminate one -- and the empty string is not a character at all. Both are # arguments a caller reaches by accident, both answer 0, and they reach it by # different paths inside the function. check_eq $P/w-nul "0" "$(pv w_nul)" check_eq $P/w-empty "0" "$(pv w_empty)" # --------------------------------------------------------- wcswidth(): STRINGS --- # The plain cases, where the answer is the sum: five ASCII characters are five # columns, three ideographs are six, and a mixture is four. check_eq $P/s-ascii "5" "$(pv s_ascii)" check_eq $P/s-empty "0" "$(pv s_empty)" check_eq $P/s-cjk "6" "$(pv s_cjk)" check_eq $P/s-mixed "4" "$(pv s_mixed)" # THE CASE THE PACKAGE EXISTS FOR. "e" + COMBINING ACUTE is TWO code points and # ONE column, so a caller measuring len() is already wrong -- by one column, on a # string that looks like one character. `s-wide-comb' is the same statement over # an ideograph, where the answer is 2 rather than 3. check_eq $P/s-combining "1" "$(pv s_combining)" check_eq $P/s-wide-comb "2" "$(pv s_wide_comb)" # A zero-width space between two letters adds no column either. check_eq $P/s-zwsp-mid "2" "$(pv s_zwsp_mid)" # THE POSIX -1 PROPAGATION. One control character ANYWHERE makes the whole answer # -1 rather than the width of the rest, and the three positions are asserted # separately because an implementation that checked only the leading character # would pass `s-c0-first' and fail the other two. NUL is the control: it is a # control character that does NOT trigger the rule, because its width is 0 rather # than -1, so "ab" is 2. check_eq $P/s-c0-first "-1" "$(pv s_c0_first)" check_eq $P/s-c0-mid "-1" "$(pv s_c0_mid)" check_eq $P/s-c0-last "-1" "$(pv s_c0_last)" check_eq $P/s-nul-mid "2" "$(pv s_nul_mid)" # THE TWO SEQUENCE RULES, which are what a string can do that a character cannot. # # ZWJ: the joiner AND THE CHARACTER AFTER IT are both skipped, so man+ZWJ+woman # measures as the man alone -- 2, not 4. That is upstream's rule for the emoji # sequences a terminal renders as one glyph. check_eq $P/s-zwj-emoji "2" "$(pv s_zwj_emoji)" # VS16 after a character the VS16 table names PROMOTES it from one cell to two, # so U+26A0 WARNING SIGN + VS16 is 2 while the sign alone is 1. VS16 with nothing # before it adds nothing, which is the case that says the promotion is # conditional on the preceding character rather than unconditional. check_eq $P/s-vs16-narrow "2" "$(pv s_vs16_narrow)" check_eq $P/s-vs16-alone "0" "$(pv s_vs16_alone)" # THE `n' ARGUMENT, which exists only for compatibility with the C signature. # Over "ab" the running total steps 0,1,3,4,6: the odd steps are the # ASCII characters and the even ones the ideographs. The sequence states BOTH # that `n' is honoured and that it counts CHARACTERS while the answer counts # COLUMNS -- confusing those two is the bug this argument invites, and no single # value of `n' would separate them. check_eq $P/n-0 "0" "$(pv n_0)" check_eq $P/n-1 "1" "$(pv n_1)" check_eq $P/n-2 "3" "$(pv n_2)" check_eq $P/n-3 "4" "$(pv n_3)" check_eq $P/n-4 "6" "$(pv n_4)" check_eq $P/n-negative "0" "$(pv n_neg)" # THE UPSTREAM LIMITATION. See the header: POSIX bounds `n' by the string and # 0.2.13 does not, so an `n' past the end raises IndexError out of a width # calculation instead of returning the whole width. Asserted as IndexError rather # than worked around, because the value a caller would want -- 2 -- is what a # FIXED release returns, and this case is where that release announces itself. check_eq $P/n-beyond "IndexError" "$(pv n_beyond)" # ------------------------------------ the version-scoped widths: THE PROOF --- # U+1F600 GRINNING FACE became WIDE in Unicode 9.0.0. Asking for its width at # 4.1.0 and 8.0.0 gives 1, and at 9.0.0 and later 2. That transition is what says # the nineteen shipped tables are nineteen DIFFERENT tables rather than nineteen # references to the latest -- which no single-level case, and no count of ranges, # could distinguish from a correct build. check_eq $P/v410-emoji "1" "$(pv v410_emoji)" check_eq $P/v800-emoji "1" "$(pv v800_emoji)" check_eq $P/v900-emoji "2" "$(pv v900_emoji)" check_eq $P/v1300-emoji "2" "$(pv v1300_emoji)" check_eq $P/v1510-emoji "2" "$(pv v1510_emoji)" # THE CONTROL for that transition: the CJK ideograph has been wide since before # 4.1.0, so it must answer 2 at EVERY level. Without it, a build that had made # every old table empty would pass the pair above by answering 1 for want of # data rather than by consulting a real 4.1.0 table. for _v in 410 800 900 1300 1510; do check_eq $P/v$_v-cjk "2" "$(pv v${_v}_cjk)" done # The VS16 promotion is level-scoped too -- the table exists only from 9.0.0, so # the same string measures 1 at 8.0.0 and 2 at 9.0.0. This is the string-level # reading of the same transition. check_eq $P/v410-vs16 "1" "$(pv v410_vs16_str)" check_eq $P/v800-vs16 "1" "$(pv v800_vs16_str)" check_eq $P/v900-vs16 "2" "$(pv v900_vs16_str)" check_eq $P/v1510-vs16 "2" "$(pv v1510_vs16_str)" # ------------------------------------------------ THE STDLIB AS THE CONTROL --- # Every case above compares wcwidth against a value a person wrote down. These # two compare it against the INTERPRETER'S OWN Unicode database, read on target: # `east_asian_width' for the two-cell claim over 16 code points, and `combining' # for the zero-cell claim over 6. They are the cases that would still be # meaningful if every literal above had been transcribed wrong, and their # expected value is the empty string -- the list of disagreements. check_eq $P/ctl-eaw-count "16" "$(pv ctl_eaw_count)" check_eq $P/ctl-eaw "" "$(pv ctl_eaw_bad)" check_eq $P/ctl-comb-count "6" "$(pv ctl_comb_count)" check_eq $P/ctl-comb "" "$(pv ctl_comb_bad)" # The image's own UCD level, recorded rather than asserted against a literal. # CPython ships a UCD generation of its own and wcwidth ships another, so the two # can differ legitimately; what a future reader needs when `ctl-eaw' reds is to # know WHICH levels were being compared. Asserting only that the interpreter # answered at all keeps this from going red on an ordinary python3 bump. check $P/unidata-version test -n "$(pv unidata_version)" # ---------------------------------------------- the environment-driven level --- # `auto' -- the default -- reads the UNICODE_VERSION environment variable, so a # deployment can pin the whole image's width policy without touching a caller. # This is a SEPARATE python3 start, and it has to be: `_wcmatch_version' is # lru_cached, so an interpreter that has already resolved `auto' will not see a # changed environment, and doing it inside the probe would assert the cache # rather than the feature. # # 4.1.0 is chosen because it answers 1 for U+1F600 where the default answers 2, # so the case fails on an image that ignored the variable rather than passing by # coincidence. check_eq $P/env-version "1" \ "$(UNICODE_VERSION=4.1.0 python3 -c 'from wcwidth import wcwidth; print(wcwidth("\U0001f600"))' 2>/dev/null)" rm -rf "$W"