. "$BST_LIB" # BST-TIMEOUT: 1500 # FIVE python3 starts, and they are the whole cost. # # Everything else reads the filesystem or dpkg with shell # # builtins, or greps one probe transcript. # # # # The five: a bare `import charset_normalizer'; a second # # import whose STDERR is the assertion; the probe, which # # takes ONE start for 152 measurements; and two runs of # # /usr/bin/normalizer, which is a python3 start apiece. # # # # The work inside the probe start is fifteen small file # # writes and some tens of detections over payloads under # # 1.4 kB: 1.6s wall on the x86-64 control, whole file. # # Beside an interpreter start on this arm -- tens of # # seconds before the first line of the program runs -- it # # is noise. 1500 is 300s per start. Nothing here is # # quadratic in anything. # # # # MEASURED on the batch-2270 image, 2026-09-07: # # elapsed=259s against this 1500s bound. # # python3-charset-normalizer 3.3.2 (meta-openembedded/meta-python # recipes-devtools/python/python3-charset-normalizer_3.3.2.bb, `inherit pypi # setuptools3', PyPI sdist unpatched, with a meta-brintos bbappend carrying ONE # RDEPENDS line and NO patch) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE IS, AND WHY ITS SUBJECT IS ENTIRELY BYTES IN MEMORY. # ============================================================================ # # Hand it bytes of unknown provenance and it answers which code page they are, # and hands back the decoded str. It is the modern replacement for chardet and # the detector `requests' reaches for, which is what puts it on this image. # # Ten public names, eight .py files, no compiled extension anywhere in it. So # nothing here reaches a device, a peer or a terminal, and every fixture the # suite needs the probe writes ITSELF into a temporary directory it then # removes. Nothing on the rootfs is read as input and nothing survives the run. # That is why this port closes on its suite with no interactive follow-on: # there is no interactive surface to follow up on. # # ============================================================================ # DETECTION IS A HEURISTIC, AND WHICH ANSWERS ARE DETERMINED IS THE WHOLE SHAPE # OF THIS FILE. # ============================================================================ # # Each candidate code page is scored on a "chaos" ratio -- how improbable the # resulting character sequence looks -- and on a per-language letter histogram, # and anything scoring over threshold=0.2 is dropped. Nothing checks validity. # So the suite asserts a LABEL only where the label is determined, and says so # where it is not: # # DETERMINED, asserted by label AND by exact decoded text: # ascii; German prose as utf_8, with and without a BOM; all four UTF-16 # framings; Russian as koi8_r and as cp1251. # # NOT DETERMINED, asserted as measured, which is the point of asserting it: # the same French text as latin_1 (answered cp1257, decoded to mojibake) and # an English text as cp1252 (answered ptcp154). Both are recorded as the # free run really answers, with the ambiguity stated in the case name. # # ============================================================================ # EVERY EXPECTED VALUE HERE WAS MEASURED, NOT PREDICTED -- WITH TWO EXCEPTIONS, # NAMED. # ============================================================================ # # All of them were read off an x86-64 control -- CPython 3.12.3, the image's own # minor -- running THIS BUILD'S OWN .deb: the tree unpacked out of # python3-charset-normalizer_3.3.2-r0_wasm32.deb. So a red names the exact value # the target produced against the exact value a correct interpreter produces # from the same bytes. # # The two exceptions are `unicodedata-builtin' and `multibytecodec-builtin', # which are statements about THIS ARM's interpreter and cannot be measured on an # x86-64 control at all -- there, unicodedata is builtin and _multibytecodec is # a lib-dynload .so, so the control answers yes and no. Both are expected `yes' # here because every stdlib extension is linked into the one static interpreter # on wasm32-hwjs. That is an EXPECTATION, not a measurement, until the batch # verify runs it. # # ============================================================================ # WHAT THIS PORT ACTUALLY RISKS ON THIS ARM, AND WHAT IT DOES NOT. # ============================================================================ # # NOT the dlopen gap. BRINTOS-2046 (the empty `Py*' export surface that refuses a # CPython extension at dlopen on this image) cannot reach a package with no .so # in it, and `pkg-only-python' asserts that the package really has none rather # than assuming it from the recipe. The optional mypyc accelerator -- which # WOULD carry one -- is not built, and `cli-speedup-off' asserts that from the # CLI's own version banner rather than from the recipe's silence. # # What it DOES risk is the import closure, in two layers. # # The first is the ordinary one: every module charset_normalizer imports at # module scope is a separate python3 sub-package on this distro, and a missing # one is an ImportError at `import charset_normalizer' with nothing else to see: # # logging -> python3-logging # json -> python3-json # codecs, encodings.aliases, # _multibytecodec -> python3-codecs # hashlib -> python3-crypt <- NOT declared upstream # unicodedata, importlib, re, # typing, functools, # collections, warnings -> python3-core # argparse, platform (cli only) -> python3-core # # CHECKED against poky's own python3-manifest.json rather than assumed, and the # closure itself measured as a sys.modules delta from a pristine interpreter. # python3-crypt is the gap: models.py line 2 does `from hashlib import sha256' # at module scope, and the stock recipe does not declare it. The bbappend adds # it; `pkg-depends-crypt' asserts the built package's Depends line by name, so # that case going red means the bbappend did not take and every `reachable-*' # below is passing on a neighbour's accident. # # The second layer is this arm's, and has no equivalent on any other image: # `unicodedata' and `_multibytecodec' are lib-dynload EXTENSION MODULES # upstream, and this image's lib-dynload is EMPTY BY CONSTRUCTION (BRINTOS-1476). # They resolve only because configure builds every stdlib extension statically # on wasm32 and links them into the one interpreter binary. A poky that narrowed # that case would turn them into .so files this image cannot load, and # `import charset_normalizer' would die on a line nobody would think to look at. # The two `*-builtin' cases assert the mechanism directly so that day names # itself. # # ============================================================================ # WHAT IS NOT ASSERTED HERE. # ============================================================================ # # Not the candidate ORDER. `CharsetMatches' sorts on construction and re-sorts # on every append, so best() is the first element by construction -- and that # INVARIANT is asserted. The order itself is the library's own confidence # heuristic; pinning a candidate list would assert the heuristic rather than the # port, and would go red on any upstream tuning change while the package worked. # # Not the coherence figure, for the same reason. `koi8r-chaos' is asserted # because 0.0 is a statement that the decode is clean; the coherence percentage # is a language-model output and is left alone. # # Not the language detection beyond one case. `koi8r-language' is asserted as # Russian because it is what makes the koi8_r/cp1251 separation work; the rest # of the language surface belongs to the library's data files, not to the port. # # Not performance: the interpreter start dominates the actual work by three # orders of magnitude. P=python3-charset-normalizer bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/charset_normalizer DI=$SITE/charset_normalizer-3.3.2.dist-info PKGLIST=/var/lib/dpkg/info/python3-charset-normalizer.list PROBE=$BST_DATA/python3-charset-normalizer-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-cn.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ---------------------------------------------------------------- on disk -- check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" check $P/init-file test -s "$PKG/__init__.py" check $P/api-file test -s "$PKG/api.py" check $P/models-file test -s "$PKG/models.py" check $P/md-file test -s "$PKG/md.py" check $P/cd-file test -s "$PKG/cd.py" check $P/utils-file test -s "$PKG/utils.py" check $P/legacy-file test -s "$PKG/legacy.py" check $P/constant-file test -s "$PKG/constant.py" check $P/cli-dir test -d "$PKG/cli" check $P/py-typed test -e "$PKG/py.typed" # Asserted over THIS package's own dpkg file list: $SITE/tests is a shared # namespace a sibling port can own (python3-reedsolo does). check_eq $P/no-tests-shipped "" \ "$(grep -E '/site-packages/tests(/|$)' "$PKGLIST" 2>/dev/null)" check $P/pycache test -s "$PKG/__pycache__/__init__.cpython-312.pyc" check $P/cli-pycache test -s "$PKG/cli/__pycache__/__main__.cpython-312.pyc" # The mypyc accelerator would land exactly here, as a compiled md. check $P/no-compiled-md test ! -e "$PKG/md.cpython-312-wasm32.so" cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in /usr/bin/normalizer) continue ;; *.py|*.pyc) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; */py.typed) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-name "Name: charset-normalizer" cat "$DI/METADATA" check_out $P/dist-info-version "Version: 3.3.2" cat "$DI/METADATA" check_out $P/dist-info-author "Author: Ahmed TAHRI" cat "$DI/METADATA" # top_level.txt says `charset_normalizer' with an UNDERSCORE while the # distribution is named with a HYPHEN. Both spellings are asserted, because a # caller that guesses either one from the other is wrong half the time. check_out $P/dist-info-top-level "charset_normalizer" cat "$DI/top_level.txt" check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag "Tag: py3-none-any" cat "$DI/WHEEL" check_out $P/dist-info-entrypoint \ "normalizer = charset_normalizer.cli:cli_detect" cat "$DI/entry_points.txt" check $P/dist-info-license test -s "$DI/LICENSE" check $P/cli-script-present test -x /usr/bin/normalizer check_out $P/cli-script-shebang "#!/usr/bin/env python3" cat /usr/bin/normalizer # -------------------------------------------------------------------- dpkg -- sed -n '/^Package: python3-charset-normalizer$/,/^$/p' /var/lib/dpkg/status \ > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-core "python3-core" cat "$W/dep" check_out $P/pkg-depends-codecs "python3-codecs" cat "$W/dep" check_out $P/pkg-depends-json "python3-json" cat "$W/dep" check_out $P/pkg-depends-logging "python3-logging" cat "$W/dep" # The one the stock recipe does not declare. This case going red means the # meta-brintos bbappend did not take, and every reachable-* below is then # passing on a neighbour having pulled python3-crypt in for its own sake. check_out $P/pkg-depends-crypt "python3-crypt" cat "$W/dep" check $P/pkg-no-ptest test ! -e /var/lib/dpkg/info/python3-charset-normalizer-ptest.list # ------------------------------------------------------------ import, bare -- check $P/import python3 -c 'import charset_normalizer' check_eq $P/import-quiet "" \ "$(python3 -c 'import charset_normalizer' 2>&1)" # ------------------------------------------------------------------- probe -- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" "$W/cli" OUT="$W/probe" pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/emitted-count "152" "$(pv emitted_count)" check_eq $P/py-version "3.12" "$(pv py_version)" check_eq $P/fixtures-written "yes" "$(pv fixtures_written)" check_eq $P/fixture-count "15" "$(pv fixture_count)" # ---------------------------------------------------------------- identity -- check_eq $P/import-ok "yes" "$(pv import_ok)" check_eq $P/version "3.3.2" "$(pv version)" check_eq $P/version-tuple "3,3,2" "$(pv version_tuple)" check_eq $P/module-answered "$PKG/__init__.py" "$(pv module_file)" check_eq $P/public-count "10" "$(pv public_count)" check_eq $P/public-names \ "from_fp,from_path,from_bytes,is_binary,detect,CharsetMatch,CharsetMatches,__version__,VERSION,set_logging_handler" \ "$(pv all_names)" # ------------------------------------------------------------ the closure -- # Each of these is a separate python3 sub-package on this distro, and the whole # RDEPENDS question for this port is whether each one resolved. check_eq $P/reachable-logging "yes" "$(pv reachable_logging)" check_eq $P/reachable-json "yes" "$(pv reachable_json)" check_eq $P/reachable-codecs "yes" "$(pv reachable_codecs)" check_eq $P/reachable-encodings-aliases "yes" "$(pv reachable_encodings_aliases)" check_eq $P/reachable-unicodedata "yes" "$(pv reachable_unicodedata)" check_eq $P/reachable-multibytecodec "yes" "$(pv reachable__multibytecodec)" check_eq $P/reachable-hashlib "yes" "$(pv reachable_hashlib)" check_eq $P/reachable-importlib "yes" "$(pv reachable_importlib)" check_eq $P/reachable-platform "yes" "$(pv reachable_platform)" check_eq $P/reachable-argparse "yes" "$(pv reachable_argparse)" check_eq $P/reachable-typing "yes" "$(pv reachable_typing)" check_eq $P/reachable-re "yes" "$(pv reachable_re)" check_eq $P/reachable-functools "yes" "$(pv reachable_functools)" check_eq $P/reachable-collections "yes" "$(pv reachable_collections)" check_eq $P/reachable-warnings "yes" "$(pv reachable_warnings)" # The two arm-specific ones. EXPECTED, not measured on a control -- see the # header. They say WHY the two above resolve on an image whose lib-dynload is # empty, so that a poky change turning them back into .so files is a named red. check_eq $P/unicodedata-builtin "yes" "$(pv unicodedata_builtin)" check_eq $P/multibytecodec-builtin "yes" "$(pv multibytecodec_builtin)" check_eq $P/unidata-version "15.0.0" "$(pv unidata_version)" # ------------------------------------------------- detection, determined -- check_eq $P/best-ascii "ascii" "$(pv best_ascii)" check_eq $P/best-utf8 "utf_8" "$(pv best_utf8)" check_eq $P/best-utf8-bom "utf_8" "$(pv best_utf8_bom)" check_eq $P/best-utf16 "utf_16" "$(pv best_utf16)" check_eq $P/best-utf16-be "utf_16" "$(pv best_utf16_be)" check_eq $P/best-utf16-le "utf_16_le" "$(pv best_utf16_le)" check_eq $P/best-utf16-be-nobom "utf_16_be" "$(pv best_utf16_be_nobom)" check_eq $P/best-koi8r "koi8_r" "$(pv best_koi8r)" check_eq $P/best-cp1251 "cp1251" "$(pv best_cp1251)" # The decoded str is asserted against EXACTLY what the probe encoded, so a # correct label over a wrong decode is still a red. check_eq $P/decode-ascii "yes" "$(pv decode_ascii)" check_eq $P/decode-utf8 "yes" "$(pv decode_utf8)" check_eq $P/decode-utf8-bom "yes" "$(pv decode_utf8_bom)" check_eq $P/decode-utf16 "yes" "$(pv decode_utf16)" check_eq $P/decode-utf16-be "yes" "$(pv decode_utf16_be)" check_eq $P/decode-utf16-le "yes" "$(pv decode_utf16_le)" check_eq $P/decode-utf16-be-nobom "yes" "$(pv decode_utf16_be_nobom)" check_eq $P/decode-koi8r "yes" "$(pv decode_koi8r)" check_eq $P/decode-cp1251 "yes" "$(pv decode_cp1251)" check_eq $P/count-utf8 "1" "$(pv count_utf8)" check_eq $P/count-utf16 "1" "$(pv count_utf16)" check_eq $P/count-ascii "1" "$(pv count_ascii)" # The koi8_r/cp1251 pair is the whole claim that the detector reads the content # rather than guessing a default: two different byte strings, two correct and # DIFFERENT labels, one identical decoded string. check_eq $P/cyrillic-pair-differs "yes" "$(pv cyrillic_pair_differs)" check_eq $P/cyrillic-pair-same-text "yes" "$(pv cyrillic_pair_same_text)" # -------------------------------------------------------------------- BOM -- check_eq $P/bom-ascii "no" "$(pv bom_ascii)" check_eq $P/bom-utf8 "no" "$(pv bom_utf8)" check_eq $P/bom-utf8-bom "yes" "$(pv bom_utf8_bom)" check_eq $P/bom-utf16 "yes" "$(pv bom_utf16)" check_eq $P/bom-utf16-be "yes" "$(pv bom_utf16_be)" check_eq $P/bom-utf16-le "no" "$(pv bom_utf16_le)" check_eq $P/bom-utf16-be-nobom "no" "$(pv bom_utf16_be_nobom)" check_eq $P/bom-alias "yes" "$(pv bom_is_byte_order_mark)" # "Strips it", said outright rather than by implication. check_eq $P/no-feff-utf8-bom "yes" "$(pv no_feff_utf8_bom)" check_eq $P/no-feff-utf16 "yes" "$(pv no_feff_utf16)" check_eq $P/no-feff-utf16-be "yes" "$(pv no_feff_utf16_be)" # With a BOM the codec eats the mark and the label loses the endianness, so LE # and BE are told apart ONLY by the decoded text (asserted above). Without one # the label carries it. The pair says which rule is in force where. check_eq $P/utf16-bom-labels-agree "yes" "$(pv utf16_bom_labels_agree)" check_eq $P/utf16-nobom-labels-differ "yes" "$(pv utf16_nobom_labels_differ)" # ------------------------------------------------ detection, NOT determined -- # The ticket asks for latin_1 and cp1252 to be asserted by label over "text # chosen so those encodings genuinely differ". They cannot be: the same bytes # are valid in a dozen single-byte code pages and nothing here checks validity. # Recorded as the free run really answers, wrong decode and all. check_eq $P/free-latin1-label "cp1257" "$(pv free_latin1)" check_eq $P/free-cp1252-label "ptcp154" "$(pv free_cp1252)" check_eq $P/free-latin1-not-exact "no" "$(pv free_latin1_exact)" check_eq $P/free-cp1252-not-exact "no" "$(pv free_cp1252_exact)" # And the original encoding is not even among the candidates offered. check_eq $P/free-latin1-absent "no" "$(pv free_latin1_has_latin1)" check_eq $P/free-cp1252-absent "no" "$(pv free_cp1252_has_cp1252)" # Isolating the two candidates does not rescue latin_1, and the reason is # structural rather than a tuning accident: api.py keeps latin_1 only as a # FALLBACK, reached when nothing survived chaos probing or when the payload # DECLARED that encoding. cp1252 is an ordinary candidate and does survive. check_eq $P/iso-latin1-count "0" "$(pv iso_latin1_count)" check_eq $P/iso-latin1-label "none" "$(pv iso_latin1)" check_eq $P/iso-cp1252-count "1" "$(pv iso_cp1252_count)" check_eq $P/iso-cp1252-label "cp1252" "$(pv iso_cp1252)" check_eq $P/iso-cp1252-exact "yes" "$(pv iso_cp1252_exact)" check_eq $P/excl-koi8r "koi8_t" "$(pv excl_koi8r)" # A coding declaration IS the documented route, and it promotes the latin_1 # fallback into a real, exact match. It does NOT rescue the cp1252 sample -- # the declaration is honoured and the sample still loses on chaos -- so the pair # is a statement about the declaration rather than a claim that declaring wins. check_eq $P/decl-latin1-label "latin_1" "$(pv decl_latin1)" check_eq $P/decl-latin1-exact "yes" "$(pv decl_latin1_exact)" check_eq $P/decl-latin1-count "1" "$(pv decl_latin1_count)" check_eq $P/decl-cp1252-label "ptcp154" "$(pv decl_cp1252)" check_eq $P/decl-cp1252-not-exact "no" "$(pv decl_cp1252_exact)" # THE SAME FRENCH TEXT IN UTF-8 IS REJECTED AS utf_8. Accent density puts its # mess ratio at 22.3%, over the 0.2 threshold, so utf_8 is dropped during chaos # probing while cp949 -- a Korean double-byte page in which those bytes are # valid and quiet -- passes at 0.0% and wins. Raising the threshold brings utf_8 # back, which is the evidence that chaos probing is what dropped it rather than # something else. A property of the CONTENT: the German fixture, ordinary prose, # is detected as utf_8 correctly above. The two together are what say so. check_eq $P/fr-utf8-label "cp949" "$(pv fr_utf8_best)" check_eq $P/fr-utf8-not-exact "no" "$(pv fr_utf8_exact)" check_eq $P/fr-utf8-utf8-absent "no" "$(pv fr_utf8_has_utf8)" check_eq $P/fr-utf8-threshold-label "utf_8" "$(pv fr_utf8_threshold)" check_eq $P/fr-utf8-threshold-exact "yes" "$(pv fr_utf8_threshold_exact)" # ------------------------------------------------------------ empty input -- # The ticket expects "the documented empty result rather than raising". It does # not raise -- and it does not answer empty either. check_eq $P/empty-no-raise "yes" "$(pv empty_no_raise)" check_eq $P/empty-count "1" "$(pv empty_count)" check_eq $P/empty-label "utf_8" "$(pv empty_label)" check_eq $P/empty-str "''" "$(pv empty_str)" check_eq $P/empty-bom "no" "$(pv empty_bom)" check_eq $P/empty-truthy "yes" "$(pv empty_truthy)" check_eq $P/empty-raw-len "0" "$(pv empty_raw_len)" # ---------------------------------------------------- the result container -- # The ORDER is the library's confidence heuristic and is not asserted; the # INVARIANT that best() is the head of it is, because that is the container's # contract and the thing a caller relies on. check_eq $P/koi8r-count "4" "$(pv koi8r_count)" check_eq $P/koi8r-ordered "yes" "$(pv koi8r_ordered)" check_eq $P/best-is-index0 "yes" "$(pv best_is_index0)" check_eq $P/first-is-best "yes" "$(pv first_is_best)" check_eq $P/iter-len-agrees "yes" "$(pv iter_len_agrees)" # __getitem__ takes an encoding NAME as well as an index, and resolves aliases # through iana_name -- `koi8-r' with a hyphen is not the label the match holds. check_eq $P/getitem-by-name "koi8_r" "$(pv getitem_by_name)" check_eq $P/getitem-absent-raises "KeyError" "$(pv getitem_absent_raises)" check_eq $P/empty-matches-len "0" "$(pv empty_matches_len)" check_eq $P/empty-matches-best "None" "$(pv empty_matches_best)" check_eq $P/empty-matches-first "None" "$(pv empty_matches_first)" check_eq $P/empty-matches-falsy "yes" "$(pv empty_matches_falsy)" check_eq $P/append-wrong-type-raises "ValueError" "$(pv append_wrong_type_raises)" # ------------------------------------------------------- match attributes -- check_eq $P/koi8r-language "Russian" "$(pv koi8r_language)" check_eq $P/koi8r-alphabets "Basic Latin,Cyrillic" "$(pv koi8r_alphabets)" check_eq $P/koi8r-chaos "0.0000" "$(pv koi8r_chaos)" check_eq $P/koi8r-percent-chaos "0.00" "$(pv koi8r_percent_chaos)" check_eq $P/koi8r-multi-byte "0.0000" "$(pv koi8r_multi_byte)" check_eq $P/koi8r-raw-len "369" "$(pv koi8r_raw_len)" check_eq $P/koi8r-aliases "cskoi8r" "$(pv koi8r_aliases)" check_eq $P/koi8r-could-be "koi8_r,koi8_t,koi8_u" "$(pv koi8r_could_be)" check_eq $P/koi8r-repr "yes" "$(pv koi8r_repr_has_encoding)" check_eq $P/koi8r-has-submatch "yes" "$(pv koi8r_has_submatch)" # output() re-encodes: to UTF-8 by default, or to whatever is asked for, which # round-trips back to the fixture's own bytes. check_eq $P/koi8r-output-utf8 "yes" "$(pv koi8r_output_is_utf8)" check_eq $P/koi8r-output-roundtrip "yes" "$(pv koi8r_output_roundtrip)" # The fingerprint is sha256 OF THE UTF-8 OUTPUT, not of the raw bytes. That is # why this package needs hashlib at all, and why two matches of the same text # through different code pages share one -- while comparing UNEQUAL under `==', # which compares the encoding too. check_eq $P/fingerprint-is-sha256 "yes" "$(pv koi8r_fingerprint_is_sha256)" check_eq $P/fingerprint-of-output "yes" "$(pv koi8r_fingerprint_of_output)" check_eq $P/fingerprint-across-codepages "yes" "$(pv fingerprint_matches_across_codepages)" check_eq $P/eq-across-codepages "no" "$(pv eq_across_codepages)" check_eq $P/eq-self "yes" "$(pv eq_self)" # `match == "koi8_r"' reads as the obvious thing to write and is a TypeError. check_eq $P/eq-str-raises "TypeError" "$(pv eq_str_raises)" # ------------------------------------------------------- from_path, from_fp -- check_eq $P/from-path-koi8r "koi8_r" "$(pv from_path_koi8r)" check_eq $P/from-path-utf8-bom "utf_8" "$(pv from_path_utf8_bom)" check_eq $P/from-path-utf16 "utf_16" "$(pv from_path_utf16)" check_eq $P/from-path-decodes "yes" "$(pv from_path_decodes)" check_eq $P/from-path-bom-stripped "yes" "$(pv from_path_bom_stripped)" check_eq $P/from-path-agrees "yes" "$(pv from_path_agrees_with_bytes)" check_eq $P/from-path-missing-raises "FileNotFoundError" "$(pv from_path_missing_raises)" check_eq $P/from-fp-koi8r "koi8_r" "$(pv from_fp_koi8r)" # ------------------------------------------------------------------ detect -- # The chardet-compatibility shim spells its answer the CHARDET way -- `KOI8-R' # where from_bytes answers `koi8_r' -- which is exactly what a caller migrating # off chardet depends on. Asserted as a DIFFERENCE as well as as two constants. check_eq $P/detect-keys "confidence,encoding,language" "$(pv detect_keys)" check_eq $P/detect-koi8r-encoding "KOI8-R" "$(pv detect_koi8r_encoding)" check_eq $P/detect-koi8r-language "Russian" "$(pv detect_koi8r_language)" check_eq $P/detect-koi8r-confidence "1.00" "$(pv detect_koi8r_confidence)" check_eq $P/detect-label-differs "yes" "$(pv detect_label_differs)" check_eq $P/detect-ascii "ascii" "$(pv detect_ascii_encoding)" check_eq $P/detect-utf8 "utf-8" "$(pv detect_utf8_encoding)" check_eq $P/detect-utf16 "UTF-16" "$(pv detect_utf16_encoding)" check_eq $P/detect-empty "utf-8" "$(pv detect_empty_encoding)" check_eq $P/detect-empty-language "''" "$(pv detect_empty_language)" check_eq $P/detect-confidence-float "float" "$(pv detect_confidence_is_float)" # --------------------------------------------------------------- is_binary -- check_eq $P/is-binary-png "yes" "$(pv is_binary_png)" check_eq $P/is-binary-utf8 "no" "$(pv is_binary_utf8)" check_eq $P/is-binary-koi8r "no" "$(pv is_binary_koi8r)" check_eq $P/is-binary-utf16 "no" "$(pv is_binary_utf16)" check_eq $P/is-binary-path "no" "$(pv is_binary_path)" # ------------------------------------------------------------- the CLI, live -- # /usr/bin/normalizer is run as a REAL PROCESS, over the very bytes the probe # handed across, and asserted to agree with the library. Two measurements # compared with each other, not one measurement against a constant. check_eq $P/cli-fixtures "yes" "$(pv cli_fixtures_handed_over)" if [ -f "$W/cli/cli-koi8r.bin" ] && [ -f "$W/cli/cli-utf8.bin" ]; then bst_ok $P/cli-fixtures-present _cliout=$(normalizer --minimal "$W/cli/cli-koi8r.bin" "$W/cli/cli-utf8.bin" 2>"$W/cli.err") _clirc=$? check_eq $P/cli-rc "0" "$_clirc" check_eq $P/cli-quiet "" "$(cat "$W/cli.err" 2>/dev/null)" check_eq $P/cli-agrees-koi8r "$(pv cli_subject_koi8r)" \ "$(printf '%s\n' "$_cliout" | sed -n 1p)" check_eq $P/cli-agrees-utf8 "$(pv cli_subject_utf8)" \ "$(printf '%s\n' "$_cliout" | sed -n 2p)" # The default output is a JSON report rather than a bare label. check_out $P/cli-json-encoding '"encoding": "koi8_r"' \ normalizer "$W/cli/cli-koi8r.bin" check_out $P/cli-json-language '"language": "Russian"' \ normalizer "$W/cli/cli-koi8r.bin" check_out $P/cli-json-bom '"has_sig_or_bom": false' \ normalizer "$W/cli/cli-koi8r.bin" else bst_notok $P/cli-fixtures-present "the probe did not hand over $W/cli" fi # The version banner names the mypyc accelerator's state directly, which is the # cheapest true statement that the pure-Python arm is what shipped. check_out $P/cli-version "Charset-Normalizer 3.3.2" normalizer --version check_out $P/cli-unicode "Unicode 15.0.0" normalizer --version check_out $P/cli-speedup-off "SpeedUp OFF" normalizer --version # the probe leaves nothing behind on the image check_eq $P/probe-workdir-removed "yes" "$(pv workdir_removed)" rm -rf "$W"