. "$BST_LIB" # BST-TIMEOUT: 1800 # ONE python3 start, and on this package that start is NOT # # the whole cost. babel unpickles a locale-data file per # # locale it is asked for, and the probe touches sixteen of # # them (en, en_US, en_GB, de, de_DE, fr_FR, ru, ru_RU, ja, # # ja_JP, ar, ar_EG, zh_Hans_CN and root behind all of them), # # plus the 471 kB global.dat, plus a po and a mo round trip. # # 1800 rather than the 1200 the pure-Python siblings use for # # that reason -- see 99-python3-alabaster, whose payload is # # 30 kB against this one's 37 MB. # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed replaces # # this note in the test plan rather than in this comment, # # which would cost an image build to keep the shipped suite # # byte-identical. # # python3-babel 2.14.0 (poky/meta recipes-devtools/python/python3-babel_2.14.0.bb, # `inherit pypi setuptools3', WITH a meta-brintos bbappend) on the wasm32-hwjs # glibc image -- CLDR LOCALE DATA AND GETTEXT CATALOGS. It is what # python3-sphinx names in its RDEPENDS, and it is the largest pure-Python # package on this image by a wide margin. # # ============================================================================ # THIS PACKAGE IS 97% DATA, AND THAT IS WHAT THE SUITE IS ABOUT. # ============================================================================ # # 37 MB installed, of which 36 MB is 1013 compiled CLDR files under # babel/locale-data/ plus a 471 kB babel/global.dat. The code is 26 .py files. # # So the failure this suite is built to catch is NOT `import babel' raising -- # that is one case and it is the cheapest one here. It is a locale-data tree # that landed truncated, or half-installed, or byte-swapped: babel then imports # perfectly, answers every call, and returns PLAUSIBLE WRONG STRINGS. A suite # that asserted only imports and file existence would be green for all of it. # # Every case in `the formatting' below therefore drives a public API at a named # locale and scores the string it produced. Sixteen locales across five scripts, # chosen so that a tree collapsed to English cannot pass: Cyrillic (ru), Arabic # script AND Arabic's six plural forms (ar), Japanese (ja), Han with a script # subtag (zh_Hans_CN), and the three European group/decimal conventions that # disagree with each other (de `.'/`,', fr NARROW NO-BREAK SPACE, en `,'/`.'). # # ============================================================================ # THE EXPECTATIONS ARE CLDR 44's, MEASURED, AND NOT AN OPINION ABOUT ENGLISH. # ============================================================================ # # babel 2.14.0 ships CLDR 44. Every expected string below was read off that data # by running data/python3-babel-probe.py VERBATIM on a control -- and the control # is CPython 3.12 on the build host, not the developer's 3.14, which matters and # is not pedantry: see `the interpreter this package assumes' below. # # A CLDR bump moves these strings. That is what these cases are for, and it is # why each expectation is a whole formatted string rather than a substring. # # ============================================================================ # WHY THE PORT NEEDED A bbappend, AND WHAT `pkg-depends-*' IS REALLY ASSERTING. # ============================================================================ # # The stock poky RDEPENDS is wrong in BOTH directions -- measured against the # module-scope imports of all 26 .py files rather than against the recipe text. # # It DECLARES python3-netserver (cgi, socketserver), python3-shell (cmd, glob, # shlex), python3-threading (queue, _threading_local) and python3-codecs # (_multibytecodec.so, xdrlib): babel imports not one module from any of them. # `threading', `shutil' and `codecs' -- the names that make those look right -- # are all python3-core. # # And it OMITS python3-logging, which babel/messages/frontend.py imports at # module scope and which is an rdepend of nothing the stock line names. That # module is what /usr/bin/pybabel runs, so the stock package ships a program its # own dependency line will not let start. `pkg-depends-logging' is the case that # says the bbappend is in the image, and `pybabel-*' is what says the dependency # it declares is sufficient. # # datetime, email and tempfile are imported directly by babel and arrive TODAY # only as transitive fallout of python3-netserver's own rdepends -- so the # bbappend declares them outright, which is what keeps them once netserver goes. # # The `pkg-depends-no-*' cases are the other half. A substring match would pass # just as happily with the four spurious names still on the line, and those four # names are exactly what would mean the bbappend had been lost in a merge. # # ============================================================================ # WHY THIS PORT IS POSSIBLE AT ALL. # ============================================================================ # # This image's python3 is ONE STATIC BINARY with an EMPTY lib-dynload and no # dynamic loader (BRINTOS-1476; 99-python3 asserts both directly), and its # exported symbol table carries no Py* API (BRINTOS-2046). A module carrying a # compiled extension could not be imported here. babel is pure Python and says # so itself -- `Root-Is-Purelib: true', tag `py3-none-any' -- which is why # `pkg-no-so' is a REGRESSION TRIPWIRE rather than boilerplate. # # ============================================================================ # THE INTERPRETER THIS PACKAGE ASSUMES, AND WHY THE CONTROL HAD TO MATCH IT. # ============================================================================ # # babel/messages/extract.py narrows the parsed node with `ast.Str', which # CPython deprecated in 3.8 and REMOVED in 3.14. On this image's 3.12 it is # present, warns, and works. Measured the hard way: the same probe run on a 3.14 # control returns `AttributeError: module ast has no attribute Str' for both # extractor cases, so a control one minor version too new would have recorded # this port's extractor as broken when it is not. # # `py-version' and `ast-has-str' are emitted as cases rather than left in this # comment, because they are the values that say WHY `extract-*' went red on the # day someone bumps the interpreter. # # The runner's read drops the first line a suite emits, per-suite and # deterministic -- not a defect in the host tool (BRINTOS-1149, filed as # linux-in-the-browser#177). The harness computes its pass count as # total-minus-failures, so THE DROPPED LINE IS COUNTED AS A PASS. A nothing-case # stands in that position so the eaten line costs nothing. It sits ahead of the # `require' guard deliberately, because a `require' that skips would emit its # skip as the first line and be eaten too. # # Remove this case when linux-in-the-browser#177 is fixed. P=python3-babel bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKGDIR=$SITE/babel LD=$PKGDIR/locale-data PKGLIST=/var/lib/dpkg/info/python3-babel.list PROBE=$BST_DATA/python3-babel-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-babel.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------- the payload --- # Where the package has to land. The probe's `module-answered' further down is # what says the interpreter agreed; these say what the filesystem holds, so a # package that imported out of some other path on sys.path is distinguishable # from one that imported out of what this image installed. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKGDIR" check $P/no-module-py test ! -f "$SITE/babel.py" # THE CODE, file by file. Thirteen top-level modules; each is named rather than # counted, because a wheel that dropped one still imports `babel' and fails in # somebody else's call. for m in __init__ core dates languages lists localedata numbers plural support units util; do check $P/src-$m test -s "$PKGDIR/$m.py" done # py.typed is the PEP 561 marker. Not code, and not decoration either: without # it a type checker treats every babel import in a consumer as Any. check $P/src-py-typed test -f "$PKGDIR/py.typed" # THE TWO SUBPACKAGES. babel.messages is the gettext half -- the half # /usr/bin/pybabel drives -- and babel.localtime is the timezone half. check $P/messages-dir test -d "$PKGDIR/messages" for m in __init__ catalog checkers extract frontend jslexer mofile plurals pofile setuptools_frontend; do check $P/messages-$m test -s "$PKGDIR/messages/$m.py" done check $P/localtime-dir test -d "$PKGDIR/localtime" for m in __init__ _fallback _helpers _unix _win32; do check $P/localtime-$m test -s "$PKGDIR/localtime/$m.py" done # ------------------------------------------------------------ the locale data -- # 97% OF THIS PACKAGE. global.dat carries the CLDR supplemental data -- territory # to timezone, currency to territory, plural rule sets -- and locale-data carries # one compiled file per locale. check $P/global-dat test -s "$PKGDIR/global.dat" check $P/locale-data-dir test -d "$LD" # root.dat is the fallback EVERY other locale inherits from. A tree missing it # lists a thousand identifiers and answers wrongly for all of them, which is the # exact failure this suite exists to catch. check $P/ld-root test -s "$LD/root.dat" # One per script the probe drives, so a partially-unpacked tree is caught here # rather than as a puzzling red three sections down. for l in en en_US en_GB de de_DE fr_FR ru ru_RU ja ja_JP ar ar_EG zh zh_Hans_CN; do check $P/ld-$l test -s "$LD/$l.dat" done # THE COUNT, from the package's own file list rather than from a glob: 1013 .dat # files, which is the number the wheel shipped. `locale-count' below is babel's # own view of the same tree and is 1012 -- root is data but not a locale # identifier -- so the two together say the tree is whole AND that babel agrees # it is whole. Neither number alone says that. cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" check_eq $P/ld-file-count "1013" \ "$(grep -c "^$LD/.*\.dat$" "$W/pkgf" 2>/dev/null)" # ------------------------------------------------------------- the packaging -- # NO COMPILED EXTENSION. See the header -- this is the regression tripwire. # # The list is copied to tmpfs first and never redirected from the rootfs: a # top-level stdin redirect on the virtiofs root makes dup2(2) return 513 and ash # EXITS, ending the suite silently (brintos/linux#73). check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # EXACTLY ONE PROGRAM, AND IT IS pybabel. A package that grew a second entry # point, or lost this one, is caught here rather than by a user. check_eq $P/pkg-bin-list "/usr/bin/pybabel" \ "$(grep -E '^/usr/(bin|sbin)/' "$W/pkgf" 2>/dev/null | sort | tr '\n' ' ' | sed 's/ *$//')" check $P/pybabel-present test -s /usr/bin/pybabel check $P/pybabel-executable test -x /usr/bin/pybabel # EVERY REGULAR FILE THE PACKAGE SHIPS IS PYTHON, COMPILED LOCALE DATA, OR ITS # METADATA -- without having to enumerate what an unexpected one might be. # dpkg's .list marks no directory with a trailing slash, so the entries are # sieved by asking the filesystem what each one is. _bad= while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc|*.dat) continue ;; */babel/py.typed) continue ;; /usr/bin/pybabel) continue ;; */*.dist-info/*) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-expected "" "$_bad" # ------------------------------------------------------------ the dist-info -- # THE INSTALLED DISTRIBUTION METADATA. Its DIRECTORY NAME IS CAPITALISED -- # `Babel-2.14.0.dist-info' beside a lowercase `babel/' package -- because the # recipe carries `PYPI_PACKAGE = "Babel"' and the wheel was built by # bdist_wheel 0.42.0, which does not normalise the name. A newer packaging # stack writes `babel-2.14.0.dist-info' instead, so this case is the one that # says which one built the .deb in this image, and `dist-name' below is what # importlib.metadata has to be asked with. DI=$SITE/Babel-2.14.0.dist-info check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -s "$DI/METADATA" check $P/dist-info-record test -s "$DI/RECORD" check $P/dist-info-wheel test -s "$DI/WHEEL" check $P/dist-info-license test -s "$DI/LICENSE" check $P/dist-info-entry-points test -s "$DI/entry_points.txt" check $P/dist-info-top-level test -s "$DI/top_level.txt" # PURE PYTHON, AS THE WHEEL ITSELF DECLARES IT -- the packaging-side statement of # the header's claim, and the counterpart to `pkg-no-so'. check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag "Tag: py3-none-any" cat "$DI/WHEEL" check_out $P/dist-info-name "Name: Babel" cat "$DI/METADATA" check_out $P/dist-info-version "Version: 2.14.0" cat "$DI/METADATA" # top_level.txt is the LOWERCASE import name beside the capitalised dist name. # The two spellings are the single most confusing thing about this package, so # each is asserted where it lives rather than left to a comment. check_eq $P/dist-info-top-level-name "babel" "$(cat "$DI/top_level.txt" 2>/dev/null)" # THE ENTRY POINTS, READ AS TEXT. `console_scripts' is what put /usr/bin/pybabel # on the image; `babel.extractors' and `babel.checkers' are the plugin groups # babel then reads back through pkg_resources, which is why `checker-count' # below is a statement about this file as much as about the code. check_out $P/ep-console-scripts "[console_scripts]" cat "$DI/entry_points.txt" check_out $P/ep-pybabel "pybabel = babel.messages.frontend:main" cat "$DI/entry_points.txt" check_out $P/ep-extractors "[babel.extractors]" cat "$DI/entry_points.txt" check_out $P/ep-checkers "[babel.checkers]" cat "$DI/entry_points.txt" # THE PAYLOAD COUNTED THREE WAYS, from the package's own file list. 26 .py and # 26 .pyc is the statement that EVERY module was byte-compiled -- a mismatch is # a tree that will be recompiled from source on every import, and no other case # here would notice. check_eq $P/pkg-py-count "26" "$(grep -c '\.py$' "$W/pkgf" 2>/dev/null)" check_eq $P/pkg-pyc-count "26" "$(grep -c '\.pyc$' "$W/pkgf" 2>/dev/null)" # 1014 .dat: the 1013 under locale-data, plus global.dat. check_eq $P/pkg-dat-count "1014" "$(grep -c '\.dat$' "$W/pkgf" 2>/dev/null)" # dpkg KNOWS the package. Every case above would look identical if some other # package had installed the same files. The stanza is read from a tmpfs copy for # the same reason as the file list, and the anchor is exact -- an unanchored # match would also take the -dev or -dbg stanza of the same family. sed -n '/^Package: python3-babel$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" check_out $P/pkg-version "Version: 2.14.0-r0" cat "$W/st" check_out $P/pkg-arch "Architecture: wasm32" cat "$W/st" # THE RUNTIME DEPENDENCIES, as dpkg recorded them, and this is where the port's # one substantive change is visible in both directions. See the header. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_out $P/pkg-depends-core "python3-core" cat "$W/dep" check_out $P/pkg-depends-difflib "python3-difflib" cat "$W/dep" check_out $P/pkg-depends-numbers "python3-numbers" cat "$W/dep" check_out $P/pkg-depends-pickle "python3-pickle" cat "$W/dep" check_out $P/pkg-depends-pytz "python3-pytz" cat "$W/dep" check_out $P/pkg-depends-setuptools "python3-setuptools" cat "$W/dep" # The four the bbappend ADDED. python3-logging is the one that was missing # outright; the other three were arriving only as netserver's fallout. check_out $P/pkg-depends-logging "python3-logging" cat "$W/dep" check_out $P/pkg-depends-datetime "python3-datetime" cat "$W/dep" check_out $P/pkg-depends-email "python3-email" cat "$W/dep" check_out $P/pkg-depends-io "python3-io" cat "$W/dep" # ...AND THE FOUR IT REMOVED, asserted as ABSENT. This is the half a substring # match cannot do, and the half that says the bbappend survived the merge. check_eq $P/pkg-depends-no-netserver "" "$(grep -o 'python3-netserver' "$W/dep")" check_eq $P/pkg-depends-no-shell "" "$(grep -o 'python3-shell' "$W/dep")" check_eq $P/pkg-depends-no-threading "" "$(grep -o 'python3-threading' "$W/dep")" check_eq $P/pkg-depends-no-codecs "" "$(grep -o 'python3-codecs' "$W/dep")" # python3-zoneinfo is absent BY DESIGN and not by omission: with pytz installed # babel's `except ModuleNotFoundError' branch never runs, so declaring zoneinfo # would be the same mistake as the four names above. `tz-backend' below is what # says the branch actually taken is the one this line assumes. check_eq $P/pkg-depends-no-zoneinfo "" "$(grep -o 'python3-zoneinfo' "$W/dep")" # THE IMAGE TOOK THE RUNTIME PACKAGE ALONE. The recipe BBCLASSEXTENDs to native # and nativesdk and OE splits a -dev and a -dbg from every recipe; a -dbg here # would be a second copy of 37 MB. check_eq $P/no-dev-package "" \ "$(sed -n '/^Package: python3-babel-dev$/p' /var/lib/dpkg/status 2>/dev/null)" check_eq $P/no-dbg-package "" \ "$(sed -n '/^Package: python3-babel-dbg$/p' /var/lib/dpkg/status 2>/dev/null)" # --------------------------------------------------------- the interpreter --- # THE IMPORTS, ON THEIR OWN, BEFORE THE PROBE. If babel cannot be imported then # every probe case below reds at once and each names a different symptom; these # are what say the cause is the import, and they are split by module so that a # missing dependency names itself. check $P/import python3 -c 'import babel' check $P/import-core python3 -c 'import babel.core' check $P/import-dates python3 -c 'import babel.dates' check $P/import-numbers python3 -c 'import babel.numbers' check $P/import-lists python3 -c 'import babel.lists' check $P/import-units python3 -c 'import babel.units' check $P/import-plural python3 -c 'import babel.plural' check $P/import-support python3 -c 'import babel.support' check $P/import-util python3 -c 'import babel.util' check $P/import-languages python3 -c 'import babel.languages' check $P/import-localedata python3 -c 'import babel.localedata' check $P/import-localtime python3 -c 'import babel.localtime' check $P/import-catalog python3 -c 'import babel.messages.catalog' check $P/import-pofile python3 -c 'import babel.messages.pofile' check $P/import-mofile python3 -c 'import babel.messages.mofile' check $P/import-extract python3 -c 'import babel.messages.extract' check $P/import-checkers python3 -c 'import babel.messages.checkers' # THE ONE THE STOCK DEPENDENCY LINE WOULD NOT HAVE ALLOWED. frontend.py is the # only module that imports `logging', and it is the module pybabel runs. A red # here with everything above green is the bbappend having been lost. check $P/import-frontend python3 -c 'import babel.messages.frontend' # ...and the two optional dependencies, separately, because they fail # differently from babel itself. check $P/import-pytz python3 -c 'import pytz' check $P/import-setuptools python3 -c 'import setuptools' # ...AND THE BARE IMPORT SAYS NOTHING WHILE DOING IT. This is a statement about # the shipped bytecode: OE writes .pyc with CHECKED-HASH invalidation, so each # source is hashed and never recompiled. A red here means a shipped .pyc was # REJECTED -- a stale hash, a python3 minor bump that changed the magic, or a # missing __pycache__ -- and the tree is being compiled from source on every # import. # # `import babel' only, NOT babel.messages.extract: that module's `ast.Str' emits # a DeprecationWarning on 3.12 by design, and asserting silence over it would be # asserting that CPython had not deprecated something it has. check_eq $P/import-quiet "" \ "$(python3 -c 'import babel' 2>&1)" check $P/pycache-init test -s "$PKGDIR/__pycache__/__init__.cpython-312.pyc" check $P/pycache-core test -s "$PKGDIR/__pycache__/core.cpython-312.pyc" check $P/pycache-dates test -s "$PKGDIR/__pycache__/dates.cpython-312.pyc" # ------------------------------------------------------------ /usr/bin/pybabel -- # THE PROGRAM, RUN. Every case above is about a library; this is the one thing # in the package a user invokes, and it is the path the missing python3-logging # would have broken. `--version' prints `Babel 2.14.0'. check_out $P/pybabel-version "pybabel 2.14.0" python3 /usr/bin/pybabel --version # ...AND RUN THROUGH ITS OWN SHEBANG, which is `#!/usr/bin/env python3' and is # therefore a statement about /usr/bin/env and $PATH as much as about babel. # Every other case here invokes the interpreter explicitly, so this is the only # one that would catch a script installed without its exec bit or with a shebang # naming an interpreter this image does not have. check_out $P/pybabel-shebang-run "pybabel 2.14.0" /usr/bin/pybabel --version check_out $P/pybabel-help-extract "extract" python3 /usr/bin/pybabel --help check_out $P/pybabel-help-compile "compile" python3 /usr/bin/pybabel --help check_out $P/pybabel-help-init "init" python3 /usr/bin/pybabel --help check_out $P/pybabel-help-update "update" python3 /usr/bin/pybabel --help # `--list-locales' walks the WHOLE locale-data tree and prints one line per # locale with its display name -- the cheapest end-to-end statement that the 36 # MB of data is readable, and a completely different code path from the probe's # per-locale unpickling. python3 /usr/bin/pybabel --list-locales > "$W/loc" 2>"$W/locerr" || : check $P/pybabel-list-locales-ran test -s "$W/loc" check_eq $P/pybabel-list-locales-quiet "" "$(cat "$W/locerr")" check_eq $P/pybabel-list-locales-count "1012" "$(grep -c . "$W/loc")" # ---------------------------------------------------------------- the probe --- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present # ONE interpreter start for everything that follows. bst_probe bounds it and # keeps BOTH streams: a probe that dies early otherwise leaves every case below # reading an empty file with no traceback to say why. bst_probe $P/probe-run "$W/probe" 1500 python3 "$PROBE" OUT="$W/probe" pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE EVIDENCE PLANE. If the probe emitted nothing, every case below is reading # an absent file: each would red with got:'' and the set would read exactly like # a broken package, when what actually happened is that no measurement was taken. # The word `unscorable' is PARSED -- brintos_local_verify re-measures a suite that # prints it on a fresh guest rather than trusting a shared boot's numbers # (BRINTOS-1644) -- so it goes in this structural check, not in a comment. if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # ------------------------------------------------------------- the identity --- check_eq $P/version "2.14.0" "$(pv version)" check_eq $P/dist-version "2.14.0" "$(pv dist_version)" # THE DISTRIBUTION NAME IS CAPITALISED AND THE IMPORT NAME IS NOT. `Babel' on # PyPI, `babel' on sys.path -- which is why the recipe carries PYPI_PACKAGE and # why importlib.metadata must be asked for the capitalised spelling. check_eq $P/dist-name "Babel" "$(pv dist_name)" # The package answered out of the files this image installed, not from a stray # copy on sys.path. This is what makes the payload cases above and the formatting # cases below statements about the SAME babel. check_eq $P/module-answered "$PKGDIR/__init__.py" "$(pv module_file)" # Everything babel loaded is .py or .pyc -- the interpreter's view of the claim # `pkg-no-so' makes from the packaging side. check_eq $P/all-pure-python "yes" "$(pv babel_all_py)" # THE PUBLIC SURFACE OF A BARE `import babel', which is NOT the same as after the # probe has imported the submodules. __init__.py imports babel.core and nothing # else, so `dates', `numbers' and the rest are absent here and that is correct. # A release that started eager-importing them would change this line, and would # also change what `import babel' costs on a 37 MB package. check_eq $P/api-surface-bare \ "Locale,UnknownLocaleError,core,default_locale,get_locale_identifier,localedata,negotiate_locale,parse_locale,plural" \ "$(pv api_surface_bare)" # WHICH TIMEZONE BACKEND babel BOUND AT IMPORT, and the reason the bbappend # leaves python3-pytz on the line. `pytz' means the `try' branch succeeded; # `zoneinfo' would mean the image had lost python3-pytz and babel had fallen # through to a stdlib module that needs an IANA tzdata tree this image does not # carry. See `pkg-depends-no-zoneinfo'. check_eq $P/tz-backend "pytz" "$(pv tz_backend)" # THE INTERPRETER'S OWN VERSION AND THE ATTRIBUTE babel's EXTRACTOR NEEDS. See # the header: these two are what say WHY `extract-*' went red on the day someone # bumps python3, rather than leaving it to be rediscovered. check_eq $P/py-version "3.12" "$(pv py_version)" check_eq $P/ast-has-str "yes" "$(pv ast_has_Str)" # ------------------------------------------------------------- the data tree -- # babel's OWN view of the tree, which is what every call below reads. 1012 # identifiers against 1013 files: `root' is data and is not a locale. check_eq $P/locale-count "1012" "$(pv locale_count)" check_eq $P/locale-dir-exists "yes" "$(pv locale_dirname_exists)" check_eq $P/locale-dir-in-package "yes" "$(pv locale_dirname_under_pkg)" check_eq $P/has-root "yes" "$(pv has_root)" check_eq $P/has-en "yes" "$(pv has_en)" # THE NEGATIVE. A tree that answered `yes' to everything would pass both cases # above; this is the one that says the lookup discriminates. check_eq $P/has-missing "no" "$(pv has_missing)" # global.dat, read through get_global() rather than by size. territory_zones is # the key format_datetime's zone naming reads. check_eq $P/global-nonempty "nonempty" "$(pv global_keys)" check_eq $P/global-zone-gb "Europe/London" "$(pv global_zone_gb)" # ------------------------------------------------------------- the Locale --- check_eq $P/parse-en-us "en_US" "$(pv parse_en_us)" check_eq $P/parse-dash "en_US" "$(pv parse_underscore_from_dash)" check_eq $P/locale-fields "zh|Hans|CN" "$(pv locale_fields)" # THE DISPLAY NAMES, WHICH ARE THE DATA TREE SPEAKING. A locale names itself in # its own language and names others in the asking locale's -- two different # lookups, so both are asserted. check_eq $P/display-name-de "Deutsch\s(Deutschland)" "$(pv display_name_de)" check_eq $P/display-name-en-in-de "Englisch\s(Vereinigte\sStaaten)" \ "$(pv display_name_en_in_de)" check_eq $P/language-name-ru "Russian" "$(pv language_name_ru)" check_eq $P/territory-name-ru "Russia" "$(pv territory_name_ru)" # Script direction, which is a per-locale fact and not a per-language guess. check_eq $P/text-direction-ar "rtl" "$(pv text_direction_ar)" check_eq $P/text-direction-en "ltr" "$(pv text_direction_en)" # THE REFUSAL. A locale babel does not have must RAISE, not fall back to # English -- a fallback would make every case above pass on an empty tree. check_eq $P/unknown-locale-raises "UnknownLocaleError" "$(pv unknown_locale_raises)" check_eq $P/negotiate "de_DE" "$(pv negotiate)" # parse_locale returns (language, territory, script, variant) -- territory # BEFORE script, which is not the order the identifier is written in. Both # directions are asserted so the asymmetry is recorded rather than assumed. check_eq $P/parse-locale-tuple "zh,CN,Hans,None" "$(pv parse_locale_tuple)" check_eq $P/get-locale-identifier "zh_CN_Hans" "$(pv get_locale_identifier)" # ------------------------------------------------------------ the formatting -- # THE CENTRE OF THIS SUITE. Each of these is a whole formatted string produced # from the installed locale data. A truncated or half-installed tree produces # plausible wrong answers here and passes everything above. # # The three number conventions that disagree, on the SAME input, so a tree that # collapsed to one locale cannot pass all three: en `1,234,567.891', de # `1.234.567,891', fr NARROW NO-BREAK SPACE (\nn), ru NO-BREAK SPACE (\nb). # The last two are why the probe folds those code points to their own tokens # instead of to \s -- fr and ru differ ONLY in which invisible space they use. check_eq $P/num-decimal-en "1,234,567.891" "$(pv num_decimal_en)" check_eq $P/num-decimal-de "1.234.567,891" "$(pv num_decimal_de)" check_eq $P/num-decimal-fr "1\nn234\nn567,891" "$(pv num_decimal_fr)" check_eq $P/num-decimal-ru "1\nb234\nb567,891" "$(pv num_decimal_ru)" check_eq $P/num-percent "25%" "$(pv num_percent)" check_eq $P/num-scientific "1E4" "$(pv num_scientific)" # CURRENCY, which is global.dat and locale-data together: the symbol comes from # one and the pattern from the other. JPY is the case that says the DIGITS are # read too -- zero decimals, so 1099.98 rounds to 1,100 rather than 1,099.98. check_eq $P/num-currency-usd "\$1,099.98" "$(pv num_currency_usd)" check_eq $P/num-currency-eur-de "1.099,98\nb€" "$(pv num_currency_eur_de)" check_eq $P/num-currency-jpy "￥1,100" "$(pv num_currency_jpy)" check_eq $P/num-symbol-group-de "." "$(pv num_symbol_group_de)" check_eq $P/num-symbol-decimal-de "," "$(pv num_symbol_decimal_de)" check_eq $P/num-currency-symbol-gbp "£" "$(pv num_currency_symbol_gbp)" check_eq $P/num-currency-name-usd "US\sDollar" "$(pv num_currency_name_usd)" # ar_EG USES LATIN DIGITS in CLDR 44's default numbering system, which is the # opposite of what the locale name suggests. Asserted because it is the answer # a reader would call a bug. check_eq $P/num-arabic-digits "1,234.5" "$(pv num_arabic_digits)" # PARSING, the inverse direction, so a formatter that is right and a parser that # is not are told apart. check_eq $P/num-parse-decimal-de "1234.56" "$(pv num_parse_decimal_de)" # ...and the refusal, by class. A non-number must raise, not return 0. check_eq $P/num-parse-bad "NumberFormatError" "$(pv num_parse_bad)" # DATES. `full' carries the weekday and the month name -- the two longest # strings in a locale's calendar -- so it exercises more of the tree than # `short', which is digits and separators. Both are here because they fail # differently. check_eq $P/date-full-en "Saturday,\sMarch\s9,\s2024" "$(pv date_full_en)" check_eq $P/date-full-de "Samstag,\s9.\sMärz\s2024" "$(pv date_full_de)" check_eq $P/date-short-en "3/9/24" "$(pv date_short_en)" check_eq $P/date-medium-ru "9\sмар.\s2024\nnг." "$(pv date_medium_ru)" # TIME, and the NARROW NO-BREAK SPACE before `PM'. CLDR 42 changed that from an # ordinary space, so a suite written against pre-42 data reds here -- which is # the version signal this case exists to give. check_eq $P/time-short-en "3:30\nnPM" "$(pv time_short_en)" check_eq $P/time-short-de "15:30" "$(pv time_short_de)" check_eq $P/datetime-medium-en "Mar\s9,\s2024,\s3:30:00\nnPM" "$(pv datetime_medium_en)" # A LITERAL PATTERN rather than a named skeleton, which is a different code path: # named formats are looked up in the locale, patterns are compiled. check_eq $P/date-pattern "2024-03-09" "$(pv date_pattern_yMd)" # The calendar rosters, whole. Twelve months and seven days, in order, which is # the largest single block of locale data any of these cases reads. check_eq $P/month-names-en \ "January,February,March,April,May,June,July,August,September,October,November,December" \ "$(pv month_names_en)" check_eq $P/day-names-de \ "Montag,Dienstag,Mittwoch,Donnerstag,Freitag,Samstag,Sonntag" \ "$(pv day_names_de)" # RELATIVE TIME and INTERVALS -- two more pattern families, and the Russian one # is a plural rule and a unit name resolved together (`3 дня', few). check_eq $P/timedelta-en "3\sdays" "$(pv timedelta_en)" check_eq $P/timedelta-ru "3\sдня" "$(pv timedelta_ru)" check_eq $P/interval-en "Jan\s1,\s2024\th–\thJan\s15,\s2024" "$(pv interval_en)" # LISTS. English puts a comma before `and' and German does not, which is the # whole of the difference and is exactly the sort of thing a wrong data tree # gets subtly wrong. check_eq $P/list-en "a,\sb,\sand\sc" "$(pv list_en)" check_eq $P/list-de "a,\sb\sund\sc" "$(pv list_de)" check_eq $P/list-or-en "a,\sb,\sor\sc" "$(pv list_or_en)" # UNITS, in both lengths, plus a compound one -- `per' is itself a locale # pattern rather than a literal. check_eq $P/unit-long-en "12\smeters" "$(pv unit_long_en)" check_eq $P/unit-short-en "12\sm" "$(pv unit_short_en)" check_eq $P/unit-compound-en "12\skilometers\sper\shour" "$(pv unit_compound_en)" # ------------------------------------------------------------ the plural rules -- # THE PART OF CLDR THAT IS CODE RATHER THAN STRINGS. babel compiles each locale's # rule set into a Python expression, so these cases exercise the compiler and not # just the unpickler. # # Four locales with four different form counts, which is what makes this a test: # a rule tree that had collapsed to English would answer one/other for every # locale and would pass no line here but the first. check_eq $P/plural-en "other,one,other,other,other,other" "$(pv plural_en)" check_eq $P/plural-ru "many,one,few,many,many,one,many" "$(pv plural_ru)" check_eq $P/plural-ar "zero,one,two,few,many,other" "$(pv plural_ar)" check_eq $P/plural-ja "other,other,other,other" "$(pv plural_ja)" # The TAG SETS, which are what a catalog's Plural-Forms header is generated # from. `other' is implicit and so is absent from every one of them -- stated # here rather than left to look like a missing form. check_eq $P/plural-en-tags "one" "$(pv plural_en_tags)" check_eq $P/plural-ru-tags "few,many,one" "$(pv plural_ru_tags)" check_eq $P/plural-ar-tags "few,many,one,two,zero" "$(pv plural_ar_tags)" # ------------------------------------------------------------- the catalogs --- # THE GETTEXT HALF -- the half /usr/bin/pybabel drives, and the half that needs # python3-email and python3-datetime. A catalog is built, written to .po, read # back, then compiled to .mo and read back from THAT, so both serialisers are # covered rather than only the text one. check_eq $P/cat-num-plurals "3" "$(pv cat_num_plurals)" check_eq $P/cat-plural-expr \ "(n%10==1\s&&\sn%100!=11\s?\s0\s:\sn%10>=2\s&&\sn%10<=4\s&&\s(n%100<10\s||\sn%100>=20)\s?\s1\s:\s2)" \ "$(pv cat_plural_expr)" check_eq $P/po-roundtrip-count "2" "$(pv po_roundtrip_msgids)" check_eq $P/po-roundtrip-hello "Привет" "$(pv po_roundtrip_hello)" check_eq $P/po-charset "utf-8" "$(pv po_header_charset)" # THE PLURAL MESSAGE SURVIVED THE ROUND TRIP AS A PLURAL, with all three Russian # forms and the SECOND one readable -- a serialiser that wrote only msgstr[0] # would pass a count of messages and fail here. check_eq $P/po-plural-forms "3" "$(pv po_plural_forms)" check_eq $P/po-plural-second "%d\sфайла" "$(pv po_plural_second)" check_eq $P/mo-roundtrip-count "2" "$(pv mo_roundtrip_msgids)" check_eq $P/mo-roundtrip-hello "Привет" "$(pv mo_roundtrip_hello)" check_eq $P/mo-plural-forms "3" "$(pv mo_plural_forms)" # THE EXTRACTOR, driven at real Python source rather than at a fixture path -- # `pybabel extract' is the command this package exists for. `_' takes one # argument and `ngettext' takes the pair, which is why the source uses both. check_eq $P/extract-count "2" "$(pv extract_count)" check_eq $P/extract-first "extracted" "$(pv extract_first)" check_eq $P/extract-plural "plural,plurals" "$(pv extract_plural)" check_eq $P/extract-linenos "3,4" "$(pv extract_linenos)" # The checker registry, built at import from pkg_resources entry points where # setuptools is there and from a hard-coded pair where it is not. Either way a # zero here is a package that would validate nothing. check_eq $P/checker-count "2" "$(pv checker_count)" # -------------------------------------------------------------- the support --- # babel.support wraps gettext -- the module an application actually uses, and the # only place stdlib `locale' and `gettext' are touched. check_eq $P/support-format-number "1.234,5" "$(pv support_format_number)" check_eq $P/support-lazy "lazy-value" "$(pv support_lazy)" check_eq $P/support-translations "Translations" "$(pv support_translations_null)" # The module count the probe left loaded. Not a roster -- the roster is the # `import-*' block above -- but a floor: a package that had started lazy-loading # its submodules would show a smaller number here while every import case above # stayed green. check_eq $P/module-count "21" "$(pv babel_module_count)" rm -rf "$W"