. "$BST_LIB" # BST-TIMEOUT: 1200 # THREE python3 starts: a bare `import', a second one whose # # STDERR is the assertion (`import-quiet'), and the probe, # # which takes ONE start for 124 measurements. # # # # 400s per start is the per-start budget the neighbouring # # python3 suites use, and this suite is a PURE-PYTHON one: # # no fork, no execve, no peer process. Everything the probe # # does after the interpreter is up is stat(2), read(2) and # # write(2) against a handful of tiny files under $TMPDIR, # # which is why 3 x 400 needs no execution allowance added # # on top the way 99-python3-editor's does. # # # # The one deliberate sleep is `waitget''s own polling # # schedule, which owes 0.2s and is bounded at 30s. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed # # replaces this note in the test plan rather than in this # # comment, which would cost an image build to keep the # # shipped suite byte-identical. # # python3-pickleshare 0.7.5 (meta-openembedded meta-python/recipes-devtools/ # python, python3-pickleshare_0.7.5.bb, `inherit setuptools3 pypi', PyPI sdist, # no patches and no bbappend) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE ACTUALLY IS, AND THEREFORE WHAT HAS TO BE TESTED. # ============================================================================ # # `PickleShareDB' IS A MutableMapping WHOSE BACKING STORE IS A DIRECTORY OF # PICKLE FILES -- one file per key, each written with # `pickle.dump(..., protocol=2)'. One 300-line flat module, `pickleshare.py'. # # That layout is not an implementation detail, it IS the product. `shelve' keeps # one dbm file and therefore needs a lock; pickleshare spends a file per key so # that two processes can write different keys with no lock at all, and reads a # stale value only until the file's mtime changes. The package exists for that # trade and for nothing else. # # So every claim here has TWO halves that must agree: what the MAPPING API # returns, and what LANDS ON DISK. A suite that asserted only the first would # pass against an implementation that kept the whole store in memory -- which # would be a different package, and a broken one for its only real user. That # user is IPython, which keeps its history and its configuration in one of # these directories. # # ============================================================================ # WHY THIS PORT IS CHEAP, AND WHERE THE ONE REAL RISK IS. # ============================================================================ # # Pure Python, no compiled extension, no shared object, no script in # ${bindir} -- the wheel is `Root-Is-Purelib: true' and installs exactly one # module plus a dist-info. So none of BRINTOS-1476's lib-dynload problems are # in scope and there is nothing to cross-compile. # # The recipe's `RDEPENDS:${PN} += "python3-pickle"' IS LOAD-BEARING and it is # the only interesting line in it. `pickleshare.py' imports `pathlib', # `collections.abc', `os', `stat', `time', `errno' and `sys' -- poky's # python3-manifest.json puts every one of those in python3-core, which # setuptools3-base already pulls -- and then `pickle', which is the ONE import # that lands in its own subpackage. Without that RDEPENDS the module does not # import on this image at all. `pkg-depends-exact' asserts the built .deb's # line verbatim, so a manifest reshuffle that moved `pathlib' out of core reds # beside the recipe that stopped being complete. # # The upstream `try: import cPickle' / `except ImportError: import pickle' and # `try: from pathlib import Path' / `except ImportError: import pathlib2' # fallbacks are Python-2 backports. On 3.12 both fall to the stdlib arm, so # neither cPickle nor pathlib2 is a dependency of this port and neither is # packaged. # # ============================================================================ # SEVEN BEHAVIOURS A CALLER WILL GET WRONG. THESE ARE THE POINT OF THE SUITE. # ============================================================================ # # This class is `dict'-shaped and is NOT a dict, and every one of the following # is a place where code written against a dict quietly does something else. # Each is measured, not inferred. # # 1. `del db['missing']' DOES NOT RAISE. `__delitem__' swallows OSError by # design -- "we lost, the other process wins the conflict" -- so the # KeyError a dict would give you is gone. `del-missing'. # # 2. `clear()' LEAVES THE DIRECTORIES. It is MutableMapping's popitem() loop, # and popitem only ever unlinks files, so a store that held `paths/are/ok/key' # still has `paths/are/ok' on disk afterwards with no keys in it. # `clear-leaves-dirs'. # # 3. A VALUE THAT CANNOT BE PICKLED LEAVES A FILE BEHIND. `__setitem__' does # `with fil.open('wb') as f: pickle.dump(...)' -- the open TRUNCATES before # pickle is ever asked, so when pickle refuses, a zero-length file is already # there. The key is then LISTED by keys() and UNREADABLE through # __getitem__: a state the mapping API cannot describe. # `unpicklable-left-file', `unpicklable-key-listed'. # # THIS IS WHERE THE TICKET'S OWN DESCRIPTION IS WRONG. BRINTOS-2287 says the # write "raises at write time rather than leaving a truncated file". A # truncated file is precisely what it leaves. The suite records the measured # behaviour; the ticket's sentence was a reasonable guess at what a careful # library would do, and this one is not that careful. # # 4. A CORRUPT FILE IS A KeyError, NOT AN UNPICKLING ERROR. `__getitem__' wraps # the load in a BARE `except:', so every failure mode collapses into "no such # key". A caller cannot tell absent from corrupt, and `'k' in db' answers # False for a file that is plainly on disk. `corrupt-read', # `corrupt-not-in-contains'. # # 5. ...AND A CORRUPT FILE STOPS `clear()' DEAD. clear() is popitem() in a loop; # popitem() READS the value before deleting it; and the KeyError that read # raises is the very signal clear() uses to mean "the store is empty". So the # loop exits at the first unreadable file and the corrupt entry -- the one a # caller most wants gone -- is the one clear() cannot remove. # `clear-with-corrupt-left'. # # 6. AN ABSOLUTE KEY ESCAPES THE STORE. The lookup is `self.root / key' and # pathlib's `/' REPLACES the left side when the right side is absolute. So # `db['/tmp/x'] = v' writes to /tmp/x, outside the directory that is supposed # to govern every file in it, and keys() -- which only rglobs under the root # -- never shows it. `absolute-key-escapes', `absolute-key-invisible'. # # 7. `keys()' IS A LIST, NOT A VIEW, while `items()' and `values()' ARE views. # The class overrides keys() and inherits the other two, so the three # disagree about their own types. `hdict' depends on the list (it calls # .sort() on the result). `keys-type', `items-type'. # # ============================================================================ # THE CONCURRENCY CLAIM IS MEASURED, NOT ASSUMED -- AND NEEDS NO SECOND PROCESS. # ============================================================================ # # What the package advertises is that many processes may share a store. What # makes that work is not a lock and not a notification: it is one file per key, # plus an mtime check that invalidates a cached value. BOTH are reachable from a # single interpreter using two PickleShareDB OBJECTS over one directory, which # is what a second process has minus the process. So the claim is exercised # here without a peer, a device or a terminal -- which is what lets this port # close on its suite. # # Three cases carry it: `peer-sees-write' (a write through one object is # visible through the other), `external-rewrite-reread' (a file rewritten # behind the library's back, with its mtime pushed forward, is re-read), and # `unchanged-mtime-serves-cache' -- the converse, which states the stale window # the design accepts instead of leaving it to be discovered. # # THE MTIME COMPARISON IS BETWEEN AN int AND A float, which is the one thing # here that depends on the target's filesystem. `__setitem__' caches # `fil.stat().st_mtime' (float); `__getitem__' compares # `fil.stat()[stat.ST_MTIME]' (int). They are equal only when the mtime has no # fractional part, so on a sub-second filesystem the cache MISSES on the value # just written and the object comes back re-unpickled rather than identical. # # Neither answer is a defect and the suite does not assert either one: the # probe emits `mtime_is_integral' and `write_then_read_identical' for the # reader and the SUITE asserts only that THE TWO AGREE # (`cache-identity-tracks-mtime') and that the value is equal regardless # (`write-then-read-equal'). Those hold on any filesystem; a literal would have # been a red the day the target's timestamp granularity changed. # # ============================================================================ # WHAT IS DELIBERATELY NOT ASSERTED, AND WHY. # ============================================================================ # # `gethashfile()' is `hash()' of a str, which CPython RANDOMISES PER PROCESS # unless PYTHONHASHSEED is set. So the NAME of the file an `hset' lands in is # not reproducible across interpreter starts. Nothing here asserts it -- only # its SHAPE (`hashfile-shape': two lowercase hex digits) and the round trip # through it, both true whatever the seed. # # For the same reason `hget' is never called for a MISSING key without an # explicit default: whether that raises KeyError or returns the module's own # private `_sentinel' OBJECT depends on whether the missing key happens to # collide into an existing bucket, which is seed-dependent. `hget-missing- # default' takes the deterministic path through both branches. # # The probe was checked for exactly this: three separate interpreter starts on # a host CPython 3.12 produced byte-identical output. # # ============================================================================ P=python3-pickleshare bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages MOD=$SITE/pickleshare.py DI=$SITE/pickleshare-0.7.5.dist-info PKGLIST=/var/lib/dpkg/info/python3-pickleshare.list PROBE=$BST_DATA/python3-pickleshare-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-pickleshare-suite.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # ------------------------------------------------------------- on the rootfs -- # ONE MODULE, NOT A PACKAGE. The wheel is purelib and this is the whole payload. check $P/site-packages-dir test -d "$SITE" check $P/module-file test -s "$MOD" check $P/not-a-package test ! -d "$SITE/pickleshare" check $P/pycache-mod test -s "$SITE/__pycache__/pickleshare.cpython-312.pyc" # The sdist ships a test module and a README beside the library; neither is a # module the wheel installs, and finding either on the rootfs would mean the # recipe packaged the source tree rather than the built wheel. check $P/no-test-py test ! -e "$SITE/test_pickleshare.py" check $P/no-readme test ! -e "$SITE/README.md" # ----------------------------------------------------------- the .deb payload -- cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # PURE PYTHON: no shared object and no program. Asserted about THIS package's # own file list -- python3-pickle next door does ship a `_pickle' .so, and that # is its business, not this one's. check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" check_eq $P/pkg-no-bin "" \ "$(grep -E '^/usr/(s?bin)/' "$W/pkgf" 2>/dev/null)" _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" sed -n '/^Package: python3-pickleshare$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" check_out $P/pkg-version "Version: 0.7.5-r0" cat "$W/st" # THE ONE LINE THAT MAKES THE PORT WORK, ASSERTED EXACTLY. Everything # pickleshare imports is in python3-core except `pickle', which is the one that # lands in its own subpackage -- so this is the whole of the recipe's RDEPENDS # and it is not decorative. A lean image that dropped python3-pickle would say # so here once, rather than present as a library that will not import. grep '^Depends:' "$W/st" > "$W/dep" 2>/dev/null || : > "$W/dep" check $P/pkg-depends-line test -s "$W/dep" check_eq $P/pkg-depends-exact \ "Depends: python3-core, python3-pickle" \ "$(cat "$W/dep")" # ------------------------------------------------------------- the dist-info -- check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-name "Name: pickleshare" cat "$DI/METADATA" check_out $P/dist-info-version "Version: 0.7.5" cat "$DI/METADATA" check $P/dist-info-record test -s "$DI/RECORD" check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" # ------------------------------------------------------------------ import -- # The port turns on this line. Everything below is unscorable without it. check $P/import python3 -c 'import pickleshare' # AND IT IMPORTS SILENTLY. A DeprecationWarning from `pathlib' or a shim # announcing itself on stderr would be invisible to the case above, and is # exactly the kind of thing that turns up in a caller's log as noise. check_eq $P/import-quiet "" \ "$(python3 -c 'import pickleshare' 2>&1)" # ------------------------------------------------------------------- probe -- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # The probe's own two bookends. `probe-complete' says it reached its last line # and `probe-count' says it emitted every key it meant to: a probe that died # two-thirds of the way through would otherwise leave the missing cases scoring # against an empty string, which `check_eq' would read as a wrong answer rather # than as an absent one. check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "124" "$(pv emitted_count)" # ------------------------------------------------------------------ identity -- # WHICH COPY ANSWERED. Every behavioural case below is a statement about the # module named here. check_eq $P/module-answered "$MOD" "$(pv module_file)" check_eq $P/is-package "no" "$(pv is_package)" check_eq $P/version-attr "0.7.5" "$(pv version_attr)" # NO `__all__' -- so `from pickleshare import *', which is what this module's # own README tells you to write, leaks the stdlib modules it imported at the # top. Two of them are named here because a caller who shadows `os' or `pickle' # that way has a bug that is very hard to see. check_eq $P/no-dunder-all "no" "$(pv has_dunder_all)" check_eq $P/star-leaks-os "yes" "$(pv star_leaks_os)" check_eq $P/star-leaks-pickle "yes" "$(pv star_leaks_pickle)" # THE CLASS IS A MutableMapping, which is where clear/items/values/get/pop/ # popitem/update/setdefault come from -- every one built out of the five # methods listed here. That inheritance is the reason several of the surprises # below exist at all. check_eq $P/is-mutablemapping "yes" "$(pv is_mutablemapping)" check_eq $P/own-methods \ "__delitem__,__getitem__,__iter__,__len__,__setitem__,getlink,hcompress,hdict,hget,hset,keys,uncache,waitget" \ "$(pv own_methods)" # ---------------------------------------------------------------- the root -- check_eq $P/root-created "yes" "$(pv root_created)" check_eq $P/root-is-path "PosixPath" "$(pv root_is_path)" check_eq $P/root-value '$W/store' "$(pv root_value)" check_eq $P/repr-db "PickleShareDB('\$W/store')" "$(pv repr_db)" check_eq $P/cache-empty-at-start "yes" "$(pv cache_empty_at_start)" # `__init__' NORMALISES THE ROOT three ways, so a caller's cwd, a `~', or a # pathlib.Path cannot move a store out from under it. check_eq $P/relative-root-absolute "yes" "$(pv relative_root_absolute)" check_eq $P/tilde-expanded '$W/tildestore' "$(pv tilde_expanded)" check_eq $P/pathlib-root-ok '$W/pstore' "$(pv pathlib_root_ok)" # Opening a store that already exists does not disturb it: __init__ only mkdirs. check_eq $P/mkdir-of-existing-ok "yes" "$(pv mkdir_of_existing_ok)" # ------------------------------------------------------------- round trip -- # THE STORE IS THE DIRECTORY, so there is no close() to call: "reopening" is a # second PickleShareDB over the same path -- exactly what a second process has. # Nothing is flushed on the way out, and these say so. check_eq $P/rt-int "yes" "$(pv rt_an_int)" check_eq $P/rt-str "yes" "$(pv rt_a_str)" check_eq $P/rt-list "yes" "$(pv rt_a_list)" check_eq $P/rt-dict "yes" "$(pv rt_a_dict)" check_eq $P/rt-nested "yes" "$(pv rt_a_nested)" check_eq $P/rt-object "yes" "$(pv rt_an_obj)" check_eq $P/rt-none "yes" "$(pv rt_a_none)" check_eq $P/rt-bool "yes" "$(pv rt_a_bool)" check_eq $P/rt-float "yes" "$(pv rt_a_float)" check_eq $P/rt-bytes "yes" "$(pv rt_a_bytes)" check_eq $P/rt-tuple "yes" "$(pv rt_a_tuple)" # A CUSTOM OBJECT COMES BACK AS ITS CLASS, not as a dict of its attributes -- # which is the whole difference between this and a JSON-backed store. check_eq $P/rt-object-type "Thing" "$(pv rt_obj_type)" check_eq $P/rt-object-attr "1" "$(pv rt_obj_attr)" # ...and as a DISTINCT object: unpickling reconstructs, it does not share. check_eq $P/rt-object-not-identical "yes" "$(pv rt_obj_not_identical)" # A tuple stays a tuple and bytes stay bytes. Protocol 2 is not JSON. check_eq $P/rt-tuple-type "tuple" "$(pv rt_tuple_type)" check_eq $P/rt-bytes-type "bytes" "$(pv rt_bytes_type)" # ------------------------------------------------------------ the file layout -- # A KEY IS A FILE. This is the assertion that separates this package from any # mapping that merely behaves like it. check_eq $P/file-per-key "yes" "$(pv file_per_key)" check_eq $P/file-count "11" "$(pv file_count)" check_eq $P/len-matches-files "yes" "$(pv len_matches_files)" # THE FILE IS A PICKLE AT PROTOCOL 2, which the module hardcodes so that a # store stays readable by a Python 2. Protocol 2 opens with the PROTO opcode # \x80\x02, and asserting those two bytes is the most direct statement of that # promise there is: a release that quietly moved to protocol 5 reds here. check_eq $P/pickle-proto-magic "8002" "$(pv pickle_proto_magic)" check_eq $P/file-is-loadable-pickle "yes" "$(pv file_is_loadable_pickle)" # A HIERARCHICAL KEY BECOMES A SUBDIRECTORY, and reads back through the same # slash-joined name -- `_normalized' rewrites the separator on the way out. check_eq $P/nested-is-subdir "yes" "$(pv nested_is_subdir)" check_eq $P/nested-is-file "yes" "$(pv nested_is_file)" check_eq $P/nested-roundtrip "yes" "$(pv nested_roundtrip)" check_eq $P/nested-key-in-keys "yes" "$(pv nested_key_in_keys)" # THE THREE ACCESSORS DISAGREE ABOUT THEIR OWN TYPES: keys() is overridden and # returns a LIST, items() and values() are inherited and return VIEWS. check_eq $P/keys-type "list" "$(pv keys_type)" check_eq $P/items-type "ItemsView" "$(pv items_type)" check_eq $P/values-type "ValuesView" "$(pv values_type)" check_eq $P/keys-matches-disk "yes" "$(pv keys_matches_disk)" check_eq $P/items-len "12" "$(pv items_len)" check_eq $P/iter-matches-keys "yes" "$(pv iter_matches_keys)" check_eq $P/contains-hit "yes" "$(pv contains_hit)" check_eq $P/contains-miss "no" "$(pv contains_miss)" # DIRECTORIES ARE NOT KEYS: keys() filters on is_file(), so the intermediate # directories are on disk and absent from the mapping. check_eq $P/dir-not-a-key "yes" "$(pv dir_not_a_key)" check_eq $P/dirs-on-disk "paths,paths/are,paths/are/ok" "$(pv dirs_on_disk)" # THE GLOB FORMS. With no argument keys() is rglob('*') and is RECURSIVE; with # an argument it is glob(pat) and is NOT, unless the pattern says so. That # asymmetry between `db.keys()' and `db.keys("*")' is the trap. check_eq $P/glob-nested-star "paths/are/ok/key" "$(pv glob_nested_star)" check_eq $P/glob-top-star-excludes-nested "yes" "$(pv glob_top_star_excludes_nested)" check_eq $P/glob-recursive-star "yes" "$(pv glob_recursive_star)" check_eq $P/glob-prefix \ "a_bool,a_bytes,a_dict,a_float,a_list,a_nested,a_none,a_str,a_tuple" \ "$(pv glob_prefix)" check_eq $P/glob-no-match "" "$(pv glob_no_match)" # ---------------------------------------------------------------- deletion -- check_eq $P/del-file-existed "yes" "$(pv del_file_existed)" check_eq $P/del-removes-file "yes" "$(pv del_removes_file)" check_eq $P/del-removes-key "yes" "$(pv del_removes_key)" # SURPRISE 1: DELETING A MISSING KEY DOES NOT RAISE. __delitem__ swallows # OSError on purpose, so this mapping diverges from dict exactly where a caller # is least likely to look. check_eq $P/del-missing "yes" "$(pv del_missing)" # `pop' DOES raise -- it is MutableMapping's, and goes through __getitem__. check_eq $P/pop-missing "!KeyError" "$(pv pop_missing)" check_eq $P/pop-missing-default "dflt" "$(pv pop_missing_default)" # SURPRISE 2: `clear()' EMPTIES THE KEYS AND LEAVES THE DIRECTORIES. check_eq $P/clear-keys-empty "yes" "$(pv clear_keys_empty)" check_eq $P/clear-len-zero "0" "$(pv clear_len_zero)" check_eq $P/clear-files-gone "yes" "$(pv clear_files_gone)" check_eq $P/clear-leaves-dirs "paths,paths/are,paths/are/ok" "$(pv clear_leaves_dirs)" check_eq $P/clear-root-still-there "yes" "$(pv clear_root_still_there)" # ------------------------------------------------------------- concurrency -- # TWO OBJECTS, ONE DIRECTORY, NO LOCK -- what a second process has, minus the # process. This is the claim the package exists for. check_eq $P/peer-sees-write "from-a" "$(pv peer_sees_write)" check_eq $P/origin-sees-peer-write "from-b" "$(pv origin_sees_peer_write)" check_eq $P/peer-sees-new-key-in-keys "yes" "$(pv peer_sees_new_key_in_keys)" # A FILE REWRITTEN BEHIND THE LIBRARY'S BACK IS RE-READ, and the mechanism is # the mtime -- not a lock and not a notification. The probe pushes the mtime # forward explicitly rather than writing fast and hoping, because on a coarse # filesystem a rewrite inside one tick keeps the old mtime. check_eq $P/cached-before-external "from-b" "$(pv cached_before_external)" check_eq $P/external-rewrite-reread "from-outside" "$(pv external_rewrite_reread)" # THE CONVERSE, which is the cache working: mtime held still, the rewritten # file is NOT re-read. This is the stale window the design accepts, stated # rather than left to be discovered. check_eq $P/unchanged-mtime-serves-cache "from-outside" \ "$(pv unchanged_mtime_serves_cache)" # `uncache()' is the escape hatch for exactly that window. check_eq $P/uncache-empties-cache "yes" "$(pv uncache_empties_cache)" check_eq $P/uncache-forces-reread "third-value" "$(pv uncache_forces_reread)" check_eq $P/uncache-refills-cache "1" "$(pv uncache_refills_cache)" # THE int/float MTIME COMPARISON. `mtime_is_integral' and # `write_then_read_identical' are emitted for the reader and are DELIBERATELY # NOT ASSERTED as literals: both follow from the target filesystem's timestamp # granularity, not from this package. What is asserted is that they AGREE -- # identity holds exactly when the mtime is integral -- and that the value is # equal either way. Those two are true on any filesystem. check_eq $P/write-then-read-equal "yes" "$(pv write_then_read_equal)" check_eq $P/cache-identity-tracks-mtime "yes" "$(pv cache_identity_tracks_mtime)" check_eq $P/cache-size-after-write "1" "$(pv cache_size_after_write)" check_eq $P/cache-key-type "PosixPath" "$(pv cache_key_type)" # ------------------------------------------------------------- error paths -- # A missing key is a KeyError carrying the KEY, not the path. check_eq $P/missing-key "!KeyError" "$(pv missing_key)" check_eq $P/missing-key-arg "no-such-key" "$(pv missing_key_arg)" # SURPRISE 3: AN UNPICKLABLE VALUE RAISES **AND LEAVES A FILE BEHIND**. The # open('wb') truncates before pickle is ever asked, so a zero-length file is # already there when pickle refuses -- listed by keys(), unreadable through # __getitem__, and invisible to `in'. THE TICKET'S DESCRIPTION SAYS THE # OPPOSITE; this is the measurement. check_eq $P/unpicklable-raises "!PicklingError" "$(pv unpicklable_raises)" check_eq $P/unpicklable-left-file "yes" "$(pv unpicklable_left_file)" check_eq $P/unpicklable-file-size "0" "$(pv unpicklable_file_size)" check_eq $P/unpicklable-key-listed "yes" "$(pv unpicklable_key_listed)" check_eq $P/unpicklable-key-unreadable "!KeyError" "$(pv unpicklable_key_unreadable)" check_eq $P/unpicklable-not-in-contains "yes" "$(pv unpicklable_not_in_contains)" # SURPRISE 4: A CORRUPT FILE IS REPORTED AS KeyError, because the load is # wrapped in a bare `except:'. Absent and corrupt are indistinguishable to a # caller, and `in' says False about a file that is plainly there. check_eq $P/corrupt-file-exists "yes" "$(pv corrupt_file_exists)" check_eq $P/corrupt-read "!KeyError" "$(pv corrupt_read)" check_eq $P/corrupt-in-keys "yes" "$(pv corrupt_in_keys)" check_eq $P/corrupt-not-in-contains "yes" "$(pv corrupt_not_in_contains)" check_eq $P/corrupt-len-counts-it "yes" "$(pv corrupt_len_counts_it)" # SURPRISE 5: A CORRUPT FILE STOPS `clear()' DEAD -- popitem() reads before it # deletes, and the KeyError that read raises is the same one clear() reads as # "the store is empty". The entry a caller most wants gone is the one clear() # cannot remove. Here BOTH files in the store are unreadable, so clear() # removes nothing at all and `shrank' is `no'. check_eq $P/clear-with-corrupt-raises "yes" "$(pv clear_with_corrupt_raises)" check_eq $P/clear-with-corrupt-left "yes" "$(pv clear_with_corrupt_left)" check_eq $P/clear-with-corrupt-shrank "no" "$(pv clear_with_corrupt_shrank)" # SURPRISE 6: AN ABSOLUTE KEY ESCAPES THE STORE. `self.root / key' is pathlib's # `/', which REPLACES the left side when the right is absolute -- so the write # lands outside the directory that is supposed to govern every file in it, and # keys() never shows it. check_eq $P/absolute-key-escapes "yes" "$(pv absolute_key_escapes)" check_eq $P/absolute-key-invisible "yes" "$(pv absolute_key_invisible)" check_eq $P/absolute-key-reads-back "escaped" "$(pv absolute_key_reads_back)" # ------------------------------------------------------------------ waitget -- # Polls for a key to appear and gives up with KeyError. Only a LOWER bound is # asserted on the sleep: this runs inside a batch verify where load can only # make elapsed times longer, so an upper bound would be a red waiting for a # busy afternoon. `bounded' is loose enough that only a hang trips it. check_eq $P/waitget-present "here" "$(pv waitget_present)" check_eq $P/waitget-missing "!KeyError" "$(pv waitget_missing)" check_eq $P/waitget-missing-slept "yes" "$(pv waitget_missing_slept)" check_eq $P/waitget-missing-bounded "yes" "$(pv waitget_missing_bounded)" # --------------------------------------------------------------- hashed set -- # `hset'/`hget' bucket many keys into few files, for a store that would # otherwise grow one file per key. The bucket NAME is hash-randomised and is # not asserted -- only its shape and the round trip through it. check_eq $P/hset-roundtrip-k1 "v1" "$(pv hset_roundtrip_k1)" check_eq $P/hset-roundtrip-k2 "v2" "$(pv hset_roundtrip_k2)" check_eq $P/hget-missing-default "dflt" "$(pv hget_missing_default)" check_eq $P/hashfile-shape "yes" "$(pv hashfile_shape)" check_eq $P/hashfile-stable-in-process "yes" "$(pv hashfile_stable_in_process)" check_eq $P/hset-made-files "yes" "$(pv hset_made_files)" check_eq $P/hdict-has-both "k1,k2" "$(pv hdict_has_both)" # `hcompress' folds a category into one `xx' file, after which `hget' in its # default fast_only mode can no longer find a key -- which the docstring says # outright. The slow path still can, and hdict is unaffected. check_eq $P/hcompress-single-file "cat/xx" "$(pv hcompress_single_file)" check_eq $P/hcompress-slow-get "v1" "$(pv hcompress_slow_get)" check_eq $P/hcompress-hdict-intact "k1,k2" "$(pv hcompress_hdict_intact)" # ------------------------------------------------------------------ getlink -- # An attribute facade over a subdirectory: `lnk.foo = 2' is `db["objs/foo"] = 2'. check_eq $P/link-type "PickleShareLink" "$(pv link_type)" check_eq $P/link-set-reaches-db "2" "$(pv link_set_reaches_db)" check_eq $P/link-get "2" "$(pv link_get)" check_eq $P/link-missing "!KeyError" "$(pv link_missing)" # SURPRISE 7: `repr()' OF A LINK IS BROKEN IN THIS RELEASE, and broken by a # stdlib rename -- it calls `Path(k).basename()' and pathlib.Path has no # `basename'; that is os.path's spelling. The line has never worked on ANY # Python 3. Asserted rather than skipped, because a later release may fix it # silently and the suite should be what notices. check_eq $P/link-repr "!AttributeError" "$(pv link_repr)" # `__init__' does `self.__dict__.update(locals())', which stores `self' in the # instance's own __dict__ -- a reference cycle, and one more name than the # class documents. check_eq $P/link-dict-keys "db,keydir,self" "$(pv link_dict_keys)" # ------------------------------------------------------------------ cleanup -- check_eq $P/probe-workdir-gone "yes" "$(pv workdir_gone)" check_eq $P/probe-cwd-restored "yes" "$(pv cwd_restored)" rm -rf "$W" exit 0