. "$BST_LIB" # BST-TIMEOUT: 2100 # FOUR python3 starts, and they are the whole cost. # # # # They are a bare `import' and THREE probes: the # # in-process one (64 measurements from one start), the # # process-back-end one, and the forced-serial one. # # # # NOTHING HERE IS COMPUTE. Measured on the build host # # against this build's own wasm32 payload under CPython # # 3.12.3: 0.31 s wall for the main probe, of which the # # start and `import joblib' are 0.12 s. The memoisation, # # the pickling and the three back ends together are a # # fraction of a second at native speed, so this suite is # # four interpreter starts and a rounding error, exactly # # like the other 99-python3-* files. # # # # 2100 is ~525 s per start, against the ~60 s a python3 # # start has measured at on this arm. The margin is for the # # start, not for the work. # # # # THE PROCESS PROBE CARRIES ITS OWN, TIGHTER BOUND -- 420 s # # inside `bst_probe' -- because it is the one call in this # # file that can hang rather than fail, and a hang there # # must cost one case rather than the suite. See below. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed # # replaces this note in the test plan rather than in this # # comment, which would cost an image build to keep the # # shipped suite byte-identical. # # python3-joblib 1.4.0 (meta-openembedded/meta-python # recipes-devtools/python/python3-joblib_1.4.0.bb, `inherit pypi setuptools3', # PyPI sdist) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE DOES. # ============================================================================ # # Two things, and this file is organised around the fact that they are # independent. `joblib.Memory' is a TRANSPARENT ON-DISK MEMOISER: a decorated # function's return value is pickled into a directory tree keyed by a hash of # its arguments, and a later call with the same arguments reads the file instead # of running the body. `joblib.Parallel' is a SMALL PARALLEL-LOOP API over a # selectable back end -- processes (`loky', the default), threads, or nothing at # all (`sequential'). `joblib.dump'/`joblib.load' are the pickling layer the # memoiser is built on, exposed as its own interface. # # ============================================================================ # MEMOISATION IS OBSERVABLE FROM OUTSIDE, AND THAT IS WHAT MAKES IT ASSERTABLE # WITHOUT TRUSTING THE LIBRARY. # ============================================================================ # # The claim `the body did not run' is not a claim about joblib's internal state: # the probe's wrapped function appends to a list the PROBE owns, so the length # of that list is the number of times the body ran, counted by the caller. Every # `mem-calls-*' case below is that count -- 1 after the first call, still 1 after # the same call repeated, 2 after a different argument, 3 after `clear()'. # # The RESULT is asserted beside each count rather than instead of it. A cache # that returned the right answer by quietly re-running would pass a value-only # assertion; one that skipped the body and returned the wrong object would pass # a count-only assertion. Neither passes both. # # ============================================================================ # PURE PYTHON, SO NO PATCH AND NO BBAPPEND. # ============================================================================ # # meta-python's recipe builds on this arm as it stands, and the wasm32 build is # a byte-compile and an install -- 1622 of 1622 tasks, zero ERROR lines. joblib # VENDORS its two dependencies (cloudpickle and loky, under # `joblib/externals/'), so the port waited on nothing: the six target RDEPENDS # the .deb declares -- python3-asyncio, python3-core, python3-json, # python3-multiprocessing, python3-pprint, python3-pydoc -- are stdlib splits of # this image's own interpreter and were already installed. # # `pkg-no-so' and `impure-modules' assert the purity from two sides, the # packaged file list and the import closure from inside the interpreter. The # 236-entry payload carries 73 `.py', 73 `.pyc' and not one shared object. # # ============================================================================ # THE PARALLEL BACK END IS WHERE THIS MACHINE MAY DIFFER, AND THE SUITE SETTLES # IT RATHER THAN ASSUMING IT. # ============================================================================ # # joblib decides at import whether a process back end is possible: it imports # `multiprocessing' and `_multiprocessing', then creates ONE POSIX NAMED # SEMAPHORE to prove locking works. If either step fails it sets `mp = None', # warns `... joblib will operate in serial mode', registers NO `loky' back end, # and makes `threading' the default. # # EITHER ANSWER IS LEGITIMATE HERE, so neither is pinned. What joblib promises, # and what `backend-linkage' pins, is that the answers AGREE: `loky' is # registered if and only if `mp' survived, and the default back end is `loky' in # that case and `threading' otherwise. A machine whose answers disagreed would # hand a caller that asked for the default a back end joblib does not believe it # has -- and that is a defect on any machine. # # THE TICKET'S REQUIREMENT FOR THE PROCESS PATH IS A DISJUNCTION, and it is # scored as one. `process-bounded' is the case that matters: the call returned, # one way or the other, inside its bound. `process-outcome' then says WHICH way # -- `ok' (workers ran, results right, in order) or `fallback' (no `loky', the # documented warning, right results on threads) -- and `process-results' asserts # the answers are the same answers the single-process back ends gave either way. # A `refused' outcome is scored as a red carrying the exception, because a # raising back end is a finding whichever side of the seam it comes from. # # THE THIRD PROBE IS WHY THAT DISJUNCTION IS NOT A DODGE. `serial-*' forces # joblib's refusal path with `JOBLIB_MULTIPROCESSING=0' -- joblib's own # supported opt-out, taking the SAME branch an absent `_multiprocessing' takes # -- and asserts, on every machine including one where processes work perfectly, # what a refusal is obliged to do: no `loky' and no `multiprocessing' in the # registry, `threading' as the default, `effective_n_jobs(-1)' of 1, and a # request for `loky' that WARNS in joblib's documented words and returns the # right answers rather than raising. So if the target reports `fallback', the # cases that say what `fallback' owes the caller have already been scored, and # scored deterministically. # # ============================================================================ # THE HANG IS THE FAILURE MODE THIS FILE IS SHAPED AROUND. # ============================================================================ # # The process probe is a SEPARATE SCRIPT run under `bst_probe' with a 420 s # bound, not a group of cases inside the main probe. `bst_probe' kills an # overrun and reports the last line the probe printed, which is why the parallel # probe emits `process_attempted' BEFORE the call: an overrun then names the # call that hung instead of reporting silence. In the main probe the same hang # would take all 64 measurements with it -- and on a batch image, the guest, and # with it every other member's suite. The main probe therefore starts no process # at all: `n_jobs=1', `threading' and `sequential' only. # # ============================================================================ # THE CACHE PATH EMBEDS THE CALLER'S OWN MODULE PATH, SO THE LAYOUT IS ASSERTED # BY SHAPE. # ============================================================================ # # joblib names a cached function's directory after its module and qualified # name, and for a function defined in a script that is the SCRIPT'S OWN PATH # with the separators rewritten. Pinning those names would pin where the probe # happens to be installed rather than joblib's layout. What is pinned instead is # what the documentation promises and what is the same everywhere: a single # `joblib' directory at the root, one `func_code.py' per cached function, one # directory per distinct argument tuple named by a 32-hex-digit hash, and # `output.pkl' + `metadata.json' inside each -- with `cache-other-files' at 0 # saying nothing else was written. # # ============================================================================ # THE ARGUMENT HASHES ARE PINNED, AND THEY ARE A STATEMENT ABOUT THE PICKLER. # ============================================================================ # # `joblib.hash' -- an md5 over a pickle -- is the function the cache keys itself # on. The pickle of a small int, of a list of small ints and of a nested dict is # byte-for-byte the same on a 32-bit wasm32 guest as on the 64-bit control, so # these three digests have an answer this build did not get to choose, and a # digest that moved would say the PICKLING moved. `hash-stable' and # `hash-discriminates' assert the two properties that make it usable as a key at # all; the literals only say which digest. # # ============================================================================ # ONE ERROR PATH HAD TO BE BUILT DIFFERENTLY BECAUSE THIS SUITE RUNS AS ROOT. # ============================================================================ # # `a cache directory that cannot be written' is naturally a mode-0500 parent, # and for root the mode is advisory -- the cache would succeed and the case # would pass while asserting nothing. The probe uses a parent that is a FILE # instead, which refuses with ENOTDIR for every uid. `err-unwritable-errno' # pins the 20, because the exception NAME alone would not distinguish it from # any other way `Memory()' might have failed to make a directory. # # AND ONE ERROR PATH IS ASSERTED FOR WHAT IT DOES NOT DO. A wrapped function # that raises propagates the exception unchanged -- and the failure is NOT # CACHED: the second call runs the body again and raises again. # `err-wrapped-call-count' at 2 is that half. A memoiser that had cached the # failure would pass both of the exception cases and fail only this one. # # ============================================================================ # UPSTREAM'S OWN TESTS ARE INSTALLED AND NOTHING RUNS THEM. # ============================================================================ # # Two separate statements. `pyproject.toml' lists `joblib.test' and # `joblib.test.data' among its packages, so 27 test modules and 96 pickle # fixtures ship into site-packages as an ordinary subpackage; and the recipe # does not `inherit ptest', so no -ptest package is built and nothing on this # image runs them. # # ============================================================================ # EXPECTED VALUES. # ============================================================================ # # Measured against THE PAYLOAD THIS BUILD PRODUCED -- the file list and the # module tree unpacked from python3-joblib_1.4.0-r0_wasm32.deb -- under CPython # 3.12.3 on the build host, byte-identical over three PYTHONHASHSEED values. # ============================================================================ P=python3-joblib bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKGDIR=$SITE/joblib DI=$SITE/joblib-1.4.0.dist-info PKGLIST=/var/lib/dpkg/info/python3-joblib.list PROBE=$BST_DATA/python3-joblib-probe.py PARPROBE=$BST_DATA/python3-joblib-parallel-probe.py SERPROBE=$BST_DATA/python3-joblib-serial-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-joblib.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ----------------------------------------------------------- what shipped --- check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKGDIR" check $P/no-module-py test ! -f "$SITE/joblib.py" check $P/init-file test -s "$PKGDIR/__init__.py" check $P/memory-py test -s "$PKGDIR/memory.py" check $P/parallel-py test -s "$PKGDIR/parallel.py" check $P/numpy-pickle-py test -s "$PKGDIR/numpy_pickle.py" check $P/hashing-py test -s "$PKGDIR/hashing.py" check $P/store-backends-py test -s "$PKGDIR/_store_backends.py" # THE TWO VENDORED DEPENDENCIES, which are why this port needed no other. check $P/externals-dir test -d "$PKGDIR/externals" check $P/cloudpickle-dir test -d "$PKGDIR/externals/cloudpickle" check $P/loky-dir test -d "$PKGDIR/externals/loky" check $P/loky-backend-dir test -d "$PKGDIR/externals/loky/backend" check $P/pycache test -s "$PKGDIR/__pycache__/__init__.cpython-312.pyc" check $P/no-cli test ! -e /usr/bin/joblib # UPSTREAM'S TESTS AND THEIR FIXTURES SHIP -- see the header on why that is not # the same statement as `pkg-no-ptest' below. check $P/tests-installed test -s "$PKGDIR/test/test_memory.py" check $P/tests-is-package test -f "$PKGDIR/test/__init__.py" check $P/test-data-dir test -d "$PKGDIR/test/data" # NOT PACKAGED, and deliberately: `_openmp_test_helper' is the one part of the # source tree carrying a `.pyx', and `pyproject.toml' lists packages explicitly # rather than discovering them, so it is left out at the source. check $P/no-openmp-helper test ! -e "$PKGDIR/test/_openmp_test_helper" cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # NOT ONE SHARED OBJECT IN THE PAYLOAD. This image's python3 is one static # binary with an empty lib-dynload, so an extension shipped here would be a # module nothing could import; joblib carries none. check_eq $P/pkg-no-so "" \ "$(grep '\.so$' "$W/pkgf" 2>/dev/null | tr '\n' ' ')" # THE LIST CARRIES THE PACKAGE'S DIRECTORIES TOO, and dpkg writes them without a # trailing separator, so they are told from files by asking the filesystem. _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc|*.json|*.txt|*.gz|*.gzip|*.bz2|*.xz|*.lzma|*.pkl|*.npy|*.z|*/METADATA|*/RECORD|*/WHEEL) continue ;; *) _bad="$_bad $_f" ;; esac done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-name "Name: joblib" cat "$DI/METADATA" check_out $P/dist-info-version "Version: 1.4.0" cat "$DI/METADATA" check_out $P/dist-info-summary \ "Summary: Lightweight pipelining with Python functions" cat "$DI/METADATA" check_out $P/dist-info-top-level "joblib" cat "$DI/top_level.txt" # THE WHEEL WAS BUILT BY setuptools' bdist_wheel, which is what `setuptools3' # reduces to here, and it is a PURE-PYTHON wheel -- the packaging-side form of # the claim `impure-modules' makes from inside the interpreter. check_out $P/dist-info-generator "Generator: bdist_wheel" cat "$DI/WHEEL" check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag "Tag: py3-none-any" cat "$DI/WHEEL" # ------------------------------------------------------------- the package --- sed -n '/^Package: python3-joblib$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" check_out $P/pkg-version "Version: 1.4.0" cat "$W/st" grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" # THE Depends LINE BY NAME. All six are stdlib splits of this image's own # interpreter, so nothing about this port waited on another. check_out $P/pkg-depends-core "python3-core" cat "$W/dep" check_out $P/pkg-depends-asyncio "python3-asyncio" cat "$W/dep" check_out $P/pkg-depends-json "python3-json" cat "$W/dep" check_out $P/pkg-depends-multiprocessing "python3-multiprocessing" cat "$W/dep" check_out $P/pkg-depends-pprint "python3-pprint" cat "$W/dep" check_out $P/pkg-depends-pydoc "python3-pydoc" cat "$W/dep" # NO -ptest PACKAGE IS BUILT -- the recipe does not `inherit ptest' at all. check_eq $P/pkg-no-ptest "" \ "$(sed -n '/^Package: python3-joblib-ptest$/p' /var/lib/dpkg/status 2>/dev/null)" # --------------------------------------------------------------- it imports --- check $P/import python3 -c 'import joblib' if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "64" "$(pv emitted_count)" # ---------------------------------------------------------------- identity --- # THE MODULE THE PROBE MEASURED IS THE PACKAGED ONE. The probe sits in # ${libdir}/brintos-tests/data and the interpreter puts a script's own directory # first on sys.path, so this is the case that would catch a probe that had # shadowed the module it measures. check_eq $P/module-file "$PKGDIR/__init__.py" "$(pv module_file)" check_eq $P/is-package "yes" "$(pv is_package)" check_eq $P/version-attr "1.4.0" "$(pv version_attr)" # HOW MUCH OF THE TREE `import joblib' PULLS IN IS DECIDED BY THE SAME SEMAPHORE # ANSWER the back-end cases turn on, so it is scored against the machine rather # than pinned: `_parallel_backends' imports the vendored cloudpickle and loky # only when `mp' survived, so a process-capable machine loads 38 `joblib.*' # modules and one in serial mode loads exactly the 17 that remain. Both numbers # are pinned -- what is not pinned is which of them this machine owes -- so a # release that made a subpackage lazy is still visible as itself rather than as # a missing attribute somewhere below. if [ "$(pv mp_available)" = "yes" ]; then check_eq $P/module-count "38" "$(pv module_count)" else check_eq $P/module-count "17" "$(pv module_count)" fi check_eq $P/all-count "18" "$(pv all_count)" # PURE PYTHON FROM INSIDE THE INTERPRETER: not one imported joblib module has a # file that is anything but a .py. check_eq $P/impure-modules "" "$(pv impure_modules)" # ------------------------------------------------------- the back-end model --- # THE ONE THING joblib PROMISES ABOUT THIS MACHINE, and the only one pinned: # the registry, the default and `mp' agree with each other. See the header on # why the VALUE of `mp_available' is not pinned and this linkage is. check_eq $P/backend-linkage "consistent" "$(pv backend_linkage)" check_eq $P/sequential-registered "yes" "$(pv sequential_registered)" check_eq $P/threading-registered "yes" "$(pv threading_registered)" # ...AND THE TWO PROCESS BACK ENDS ARE REGISTERED TOGETHER OR NOT AT ALL, which # is the same claim from the other side: both come from `mp'. check_eq $P/process-backends-agree \ "$(pv loky_registered)" "$(pv multiprocessing_registered)" # THE IMPORT IS SILENT IF AND ONLY IF THE PROCESS PATH SURVIVED. A warning here # is joblib saying it fell back to serial mode, and it lands on a caller's # stderr on every start -- so it is scored, against the machine's own answer # rather than against silence. if [ "$(pv mp_available)" = "yes" ]; then check_eq $P/import-warning-linkage "" "$(pv import_warnings)" else check_out $P/import-warning-linkage "joblib will operate in serial mode" \ echo "$(pv import_warnings)" fi # n_jobs=1 IS ONE JOB WHATEVER THE MACHINE IS. `effective_n_jobs(-1)' is the # guest's CPU count and is asserted only to be a real number of jobs. check_eq $P/eff-njobs-1 "1" "$(pv eff_njobs_1)" check_eq $P/eff-njobs-all-positive "yes" "$(pv eff_njobs_all_positive)" # ------------------------------------------------------------- memoisation --- check_eq $P/mem-store-backend "FileSystemStoreBackend" "$(pv mem_store_backend)" # THE BODY RAN ONCE... check_eq $P/mem-result-first "6" "$(pv mem_result_first)" check_eq $P/mem-calls-first "1" "$(pv mem_calls_first)" # ...AND NOT AGAIN FOR THE SAME ARGUMENT. This pair is the whole claim of the # package, counted by the caller rather than reported by the library. check_eq $P/mem-result-repeat "6" "$(pv mem_result_repeat)" check_eq $P/mem-calls-repeat "1" "$(pv mem_calls_repeat)" # ...AND AGAIN FOR A DIFFERENT ONE, which is what says the cache is keyed rather # than merely sticky. check_eq $P/mem-result-other "8" "$(pv mem_result_other)" check_eq $P/mem-calls-other "2" "$(pv mem_calls_other)" # THE DOCUMENTED LAYOUT, BY SHAPE -- see the header on why not by name. check_eq $P/cache-root-entries "joblib" "$(pv cache_root_entries)" check_eq $P/cache-func-code "1" "$(pv cache_func_code_files)" check_eq $P/cache-outputs "2" "$(pv cache_output_files)" check_eq $P/cache-metadata "2" "$(pv cache_metadata_files)" # NOTHING ELSE WAS WRITTEN. Without this the three counts above would be # satisfied by a tree that also carried whatever else it liked. check_eq $P/cache-other-files "0" "$(pv cache_other_files)" check_eq $P/cache-hashdir-count "2" "$(pv cache_hashdir_count)" # ONE DIRECTORY PER ARGUMENT TUPLE, NAMED BY A 32-HEX-DIGIT HASH. check_eq $P/cache-hashdir-shape "yes" "$(pv cache_hashdir_shape)" check_eq $P/cache-metadata-keys "duration,input_args,time" \ "$(pv cache_metadata_keys)" # CLEARING EMPTIES IT AND FORCES A RE-RUN. The root directory survives and # everything under it goes; the body count going up by one is the half that says # the cache was really gone rather than merely unreadable. check_eq $P/clear-root-entries "joblib" "$(pv clear_root_entries)" check_eq $P/clear-outputs "0" "$(pv clear_output_files)" check_eq $P/clear-metadata "0" "$(pv clear_metadata_files)" check_eq $P/clear-result "6" "$(pv clear_result)" check_eq $P/clear-calls "3" "$(pv clear_calls)" # ----------------------------------------------------------- dump and load --- check_eq $P/dump-return-type "list" "$(pv dump_return_type)" check_eq $P/dump-return-len "1" "$(pv dump_return_len)" check_eq $P/dump-file-exists "yes" "$(pv dump_file_exists)" check_eq $P/load-equal "yes" "$(pv load_equal)" # THE PLAIN FILE IS A PROTOCOL-4 PICKLE -- `\x80\x04'. Pinned because # `numpy_pickle' is joblib's own writer rather than the stdlib's, and a change # of container is exactly what a round-trip inside one release cannot see. check_eq $P/plain-magic "8004" "$(pv plain_magic)" # AND THE COMPRESSED ONE IS A GZIP MEMBER, not a pickle with a flag set. check_eq $P/gz-magic "1f8b" "$(pv gz_magic)" check_eq $P/gz-load-equal "yes" "$(pv gz_load_equal)" # IT IS SMALLER, and that took a compressible payload to mean anything: joblib's # container adds a header, so a tiny object comes out LARGER compressed than # plain and `compress=3' would have looked like a regression. See the probe. check_eq $P/gz-smaller "yes" "$(pv gz_smaller)" check_eq $P/gz-differs "yes" "$(pv gz_differs)" # THE ARGUMENT-HASH MACHINERY, MEASURED DIRECTLY. See the header: these three # digests are a statement about the PICKLER, and they have an answer this build # did not choose. check_eq $P/hash-int "56615ea01687173ebab08c915ad7e500" "$(pv hash_int)" check_eq $P/hash-list "d6728d1a1a6ba57aa70ca5a0650628a7" "$(pv hash_list)" check_eq $P/hash-dict "d076cd33a52197b02c7f179a6111975e" "$(pv hash_dict)" check_eq $P/hash-stable "yes" "$(pv hash_stable)" check_eq $P/hash-discriminates "yes" "$(pv hash_discriminates)" # ------------------------------------------ the single-process back ends --- # IN INPUT ORDER. A back end that returned the right set in the wrong order is # the failure these catch; the expected string is emitted by the probe beside # them so a divergence names which side moved. check_eq $P/par-expected "0,1,4,9,16,25,36,49" "$(pv par_expected)" check_eq $P/par-njobs1 "0,1,4,9,16,25,36,49" "$(pv par_njobs1)" check_eq $P/par-threading "0,1,4,9,16,25,36,49" "$(pv par_threading)" check_eq $P/par-sequential "0,1,4,9,16,25,36,49" "$(pv par_sequential)" # ------------------------------------------------------------ error paths --- # ENOTDIR, not EACCES -- see the header on why root made the obvious shape # useless. check_eq $P/err-unwritable "NotADirectoryError" "$(pv err_unwritable)" check_eq $P/err-unwritable-errno "20" "$(pv err_unwritable_errno)" # A TRUNCATED PICKLE RUNS OUT OF INPUT MID-OPCODE, and EOFError carries NO # MESSAGE AT ALL -- `' is the probe's token for that, and is itself the # assertion that a caller logging `str(exc)' logs nothing. check_eq $P/err-truncated "EOFError: " "$(pv err_truncated)" check_eq $P/err-empty "EOFError: " "$(pv err_empty)" check_eq $P/err-missing "FileNotFoundError" "$(pv err_missing)" check_eq $P/err-missing-errno "2" "$(pv err_missing_errno)" # THE WRAPPED FUNCTION'S EXCEPTION REACHES THE CALLER UNCHANGED... check_eq $P/err-wrapped-raises "ValueError: boom-1" "$(pv err_wrapped_raises)" check_eq $P/err-wrapped-raises-again "ValueError: boom-1" "$(pv err_wrapped_raises_again)" # ...AND WAS NOT CACHED. A memoiser that had cached the failure passes both # cases above and fails this one alone. check_eq $P/err-wrapped-call-count "2" "$(pv err_wrapped_call_count)" # ------------------------------------------------- the refusal contract --- # FORCED, NOT WAITED FOR. `JOBLIB_MULTIPROCESSING=0' takes the same branch an # absent `_multiprocessing' takes, so these cases are deterministic on every # machine -- see the header. They are what `fallback' below is obliged to mean. if [ ! -f "$SERPROBE" ]; then bst_notok $P/serial-probe-present "$SERPROBE did not ship" else bst_ok $P/serial-probe-present ( JOBLIB_MULTIPROCESSING=0 export JOBLIB_MULTIPROCESSING bst_probe $P/serial-probe-run "$W/serial" 600 python3 "$SERPROBE" ) SOUT="$W/serial" sv() { grep "^$1=" "$SOUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE ENVIRONMENT REACHED THE INTERPRETER. Without this case an environment # that did not take would make every case below pass by measuring the # ordinary arm a second time. check_eq $P/serial-env "0" "$(sv env_joblib_multiprocessing)" check_eq $P/serial-complete "yes" "$(sv probe_done)" check_eq $P/serial-count "11" "$(sv emitted_count)" check_eq $P/serial-mp "no" "$(sv mp_available)" check_eq $P/serial-backends "sequential,threading" "$(sv backends)" check_eq $P/serial-default "threading" "$(sv default_backend)" # THE OPT-OUT IS SILENT: it is a configuration, not a fault, so unlike the # semaphore failure it must put nothing on a caller's stderr. check_eq $P/serial-quiet "" "$(sv import_warnings)" # THE CALLER-VISIBLE STATEMENT THAT THERE IS NO PARALLELISM TO BE HAD. check_eq $P/serial-eff-njobs "1" "$(sv eff_njobs_all)" # ASKING FOR loky ANYWAY DOES NOT RAISE... check_eq $P/serial-outcome "fallback" "$(sv outcome)" check_eq $P/serial-exception "" "$(sv exception)" # ...IT WARNS, IN joblib's DOCUMENTED WORDS... check_eq $P/serial-warning \ "joblib backend 'loky' is not available on your system, falling back to threading." \ "$(sv warnings)" # ...AND COMPUTES THE RIGHT ANSWERS ANYWAY, in input order. This is the whole # of what `a documented refusal rather than a wedge' means. check_eq $P/serial-results "0,1,4,9,16,25,36,49" "$(sv results)" fi # --------------------------------------------------- the process back end --- # THE BOUND IS THE POINT. 420 s is seven times the ~60 s a python3 start costs # on this arm, and the work after the start is eight squarings; anything beyond # it is a hang rather than a slow machine. `bst_probe' kills it and reports the # last line printed, which is why the probe announces the call before making it. if [ ! -f "$PARPROBE" ]; then bst_notok $P/parallel-probe-present "$PARPROBE did not ship" else bst_ok $P/parallel-probe-present bst_probe $P/parallel-probe-run "$W/parallel" 420 python3 "$PARPROBE" QOUT="$W/parallel" qv() { grep "^$1=" "$QOUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } # THE TICKET'S REQUIREMENT, AND THE CASE THAT CARRIES IT: the call RETURNED, # one way or the other, inside the bound. `never to hang' is this line. check_eq $P/process-bounded "yes" "$(qv process_bounded)" check_eq $P/process-complete "yes" "$(qv probe_done)" check_eq $P/process-count "9" "$(qv emitted_count)" # THE TWO PROBES AGREE ABOUT THE MACHINE. They are separate interpreter # starts, so a disagreement would say the answer is not a property of the # image. check_eq $P/process-mp-agrees "$(pv mp_available)" "$(qv mp_available)" # WHICH SIDE OF THE DISJUNCTION THIS MACHINE IS ON. `ok' means the workers # ran; `fallback' means joblib refused in the documented way the `serial-*' # cases above pin. `refused' -- the back end raising -- is neither, and is the # red this case exists to produce. case "$(qv process_outcome)" in ok|fallback) bst_ok $P/process-outcome ;; *) bst_notok $P/process-outcome \ "want ok|fallback, got '$(qv process_outcome)': $(qv process_exception)" ;; esac # AND THE ANSWERS ARE THE SAME ANSWERS THE SINGLE-PROCESS BACK ENDS GAVE, # whichever side it landed on. Boundedness without this would be satisfied by # a back end that returned promptly and wrongly. check_eq $P/process-results "0,1,4,9,16,25,36,49" "$(qv process_results)" check_eq $P/process-expected "0,1,4,9,16,25,36,49" "$(qv expected)" fi rm -rf "$W"