. "$BST_LIB" # BST-TIMEOUT: 1600 # FOUR python3 starts and nothing else. # # # # They are: a bare `import'; `import mmap' on its own, # # because an image whose interpreter lacks that sub-package # # fails this port at IMPORT rather than at the first read # # and the two are worth telling apart; a third whose STDERR # # is the assertion (`import-quiet'); and the probe, which # # takes ONE start for 264 measurements. # # # # 400s per start is the budget the neighbouring python3 # # suites use on this arm, so four starts is 1600 and the # # number is the start count times that budget rather than a # # figure carried over from a suite with fewer starts. The probe writes a 200000-byte # # file, an empty one and a one-byte one into ${TMPDIR:-/tmp} # # -- tmpfs -- and maps them; there is no fork, no exec and # # no device, so the interpreter start is the bulk of the # # cost. The heaviest single measurement is `walk_bytes_ok', # # which steps the whole 200000 bytes in 4096-byte reads and # # checks every byte against a computed value. # # # # A suite-timeout red is a batch-wide cost, so if the # # starts prove slower than that on target the RIGHT fix is # # this number, not fewer cases. # # # # NOT MEASURED ON TARGET YET: this port's first on-target # # run is the batch verify, and the measured elapsed # # replaces this note in the test plan rather than in this # # comment, which would cost an image build to keep the # # shipped suite byte-identical. # # python3-smmap 6.0.0 (poky meta/recipes-devtools/python/python3-smmap_6.0.0.bb, # `inherit pypi setuptools3', PyPI sdist) on the wasm32-hwjs glibc image. # # ============================================================================ # WHAT THE PACKAGE ACTUALLY IS, AND THEREFORE WHAT HAS TO BE TESTED. # ============================================================================ # # `smmap' is a sliding-window memory-map manager. It lets a program read a file # larger than the address space it wants to spend, by handing out windowed # `mmap' regions through `SlidingWindowMapManager' and `SlidingWindowMapBuffer' # and evicting least-recently-used regions once a configured budget is reached. # It is the mapping layer GitPython reads pack files through. # # It is PURE PYTHON -- the whole package is four modules and no extension -- so # there is nothing arch-specific in the SOURCE. What it does have is a single # system dependency, and that dependency is the point: EVERY read below goes # through `mmap(2)'. This suite is therefore the round's most direct probe of # `mmap' on this arm, and a red in the read cases is read as a PLATFORM result # first and a port defect second. # # ============================================================================ # THE EXPECTED VALUES ARE MEASURED, NOT PREDICTED -- BUT NOT ON TARGET. # ============================================================================ # # Every literal below was taken from running data/python3-smmap-probe.py under # CPython 3.12.3 against THIS PORT'S OWN BUILT PACKAGE -- the # python3-smmap_6.0.0-r0_wasm32.deb this recipe produced, unpacked and put on # PYTHONPATH -- rather than from reading the source and guessing. # # So they are facts about the shipped bytes. What is NOT yet measured is this # arm: the first on-target run of this suite is the batch verify. # # ============================================================================ # HOW THE READS ARE ASSERTED, AND WHY THAT IS NOT CIRCULAR. # ============================================================================ # # The probe writes its own backing files, with the byte at offset i set to # pat(i) = (i * 37 + 11) & 0xFF. The value at every offset is a function of the # offset alone, so a window read back at [i, j) is checked against a value # COMPUTED from i and j. There is no stored copy to drift, and a 200000-byte # window is as cheap to assert as one byte. # # That makes most read emissions `yes'/`no', which on its own would be a suite # that cannot fail -- a broken comparator would emit `yes' for everything. Three # things stop that: # # `pattern-at-*' pin the generator's own output at nine fixed offsets against # literals here, so the thing computing the expectations is # itself checked against constants. # `control-*' compare a CORRECT read against the WRONG bytes and require a # mismatch. A comparator that cannot say `no' fails these. # `*-hex' pin the actual bytes read at four places, so at least some of # the evidence is data rather than a verdict about data. # # ============================================================================ # WHAT IS ARM-DEPENDENT HERE, AND WHY NONE OF IT IS ASSERTED AS A NUMBER. # ============================================================================ # # `mmap.ALLOCATIONGRANULARITY' and `sys.maxsize' are facts about the target, and # smmap derives BOTH of its defaults from the second: `SlidingWindowMapManager' # with no arguments picks a 1024 MB window and an 8192 MB budget on a 64-bit # host and 64 MB / 1024 MB otherwise, off `is_64_bit()'. # # This suite asserts NO literal for any of them. Guessing wasm32's page size or # word size here would be a prediction dressed as a measurement, and it would # fail on the port rather than on the guess. What is asserted instead is # INTERNAL CONSISTENCY, which holds on any arm: # # `arch-*-consistent' the defaults are the arch coefficient times 1 MB, for # whichever arch `is_64_bit()' reports. # `is-64-bit-consistent' `is_64_bit()' agrees with `sys.maxsize'. # `maxsize-one-of-two' `sys.maxsize' is one of the two values a CPython build # can report -- not which one. # `granularity-*' the granularity is a power of two and divides 65536. # # The probe's files are a FIXED 200000 bytes and its managers take a FIXED # 65536-byte window rather than sizes derived from the granularity, so every # offset, length and region count below is a literal on any arm whose # granularity divides 65536. `granularity-divides-65536' is the case that says # so, and `region-size-window-or-granularity' is the one that stays green and # names the cause if it does not. # # THE ONE THING THIS FILE PREDICTS, and it is recorded in the test plan rather # than asserted here: that wasm32 reports a 32-bit `sys.maxsize' and therefore # the 64 MB / 1024 MB defaults. Nothing above depends on it. # # ============================================================================ # SIX THINGS A CALLER WILL GET WRONG. THESE ARE THE POINT OF THE SUITE. # ============================================================================ # # `buf[i:j]' HAS TWO RETURN TYPES. `__getslice__' has a fast path -- the # slice lies wholly inside the region the cursor is already # on -- which hands back a MEMORYVIEW slice of the mmap, and a # slow path which joins pieces and hands back BYTES. Which one # you get depends on where the PREVIOUS read left the cursor, # so `buf[i:j] + b"x"' is a TypeError some of the time and # fine the rest. `read-fastpath-type', `read-slowpath-type', # `read-fastpath-concat' and `read-slowpath-concat' are the # four halves of that, each measured on a buffer positioned on # purpose. # # AN OVER-LONG SLICE RAISES, AND DOES NOT COME BACK SHORT. `__getslice__' # loops `while l', consuming what each region gives it; past # EOF the next `use_region' comes back invalid and the loop's # own `assert c.is_valid()' is what stops it. The bounds check # on this interface is a debugging aid -- and under # `python3 -O' it is not compiled in at all. # `read-past-end-overlong' is that fact. # # AN OVERRUN POISONS THE BUFFER. The failed `use_region' leaves the cursor # with no region and nothing puts one back, so every LATER # read on that buffer dies on the ENTRY assert -- including # reads that worked a moment earlier. A caller who catches the # first AssertionError and carries on gets a second one from a # correct read. `poison-read-before', `poison-read-after' and # `poison-fresh-buffer' are the before, the after, and the # fact that the damage is per-cursor rather than per-file. # # AN INT INDEX AND A SLICE FAIL DIFFERENTLY. `buf[FSIZE]' is an # AttributeError, not the AssertionError `buf[FSIZE:FSIZE+10]' # gives: `__getitem__' asserts validity BEFORE calling # `use_region', so the assert passes, the call invalidates the # cursor, and `c.buffer()' then dereferences None. # `int-index-past-end' is that fact. # # THE BUFFER INDEXES ABSOLUTELY, THOUGH ITS DOCSTRING SAYS RELATIVELY. "if you # map an offset, index 0 will map to the first byte at the # offset you used" -- but the code does `c.buffer()[i - # c.ofs_begin()]' against absolute offsets. For a buffer built # at offset 0 the two agree, which is why it survives. # `int-index-offset-buffer-*' are the same read on a buffer # built at offset 1000, where they do not. # # util's NAMES ARE NOT AT THE PACKAGE ROOT. `smmap/__init__.py' does # `from .mman import *' and `from .buf import *', and those # two `__all__' lists name four classes between them. So # `smmap.align_to_mmap', `smmap.is_64_bit', `smmap.MapWindow' # and `smmap.ALLOCATIONGRANULARITY' are all AttributeErrors, # although mman imported every one of them by name. # `root-has-*' / `util-has-*' are that pair of facts. # # ============================================================================ # WHAT SHIPPED THAT A READER WOULD NOT EXPECT. # ============================================================================ # # `setup.py' calls `find_packages()', which finds `smmap' AND `smmap.test'. So # the upstream unit tests are installed on the image, at # $SITE/smmap/test/. It is asserted rather than removed: the recipe is left as # poky wrote it, and unlike a TOP-LEVEL `tests' package this one is namespaced # under `smmap', so it cannot shadow anything. `test-subpackage-*' and # `pkg-test-*' pin it so a change either way is visible. P=python3-smmap # --------------------------------------------------------------------------- bst_ok $P/suite-canary require python3 $P/python3-present SITE=/usr/lib/python3.12/site-packages PKG=$SITE/smmap DI=$SITE/smmap-6.0.0.dist-info PKGLIST=/var/lib/dpkg/info/python3-smmap.list PROBE=$BST_DATA/python3-smmap-probe.py tmp=${TMPDIR:-/tmp} W=$tmp/bst-smmap-suite.$$ mkdir -p "$W" || { bst_notok $P/workdir "cannot create $W"; exit 0; } # --- the package, as installed --------------------------------------------- # find_packages() and py_modules are not the same thing: what lands here is a # PACKAGE DIRECTORY with four modules, not a flat module. check $P/site-packages-dir test -d "$SITE" check $P/package-dir test -d "$PKG" check $P/is-a-package test -s "$PKG/__init__.py" check $P/not-a-flat-module test ! -e "$SITE/smmap.py" check $P/module-mman test -s "$PKG/mman.py" check $P/module-buf test -s "$PKG/buf.py" check $P/module-util test -s "$PKG/util.py" check $P/pycache-init test -s "$PKG/__pycache__/__init__.cpython-312.pyc" check $P/pycache-mman test -s "$PKG/__pycache__/mman.cpython-312.pyc" check $P/pycache-buf test -s "$PKG/__pycache__/buf.cpython-312.pyc" check $P/pycache-util test -s "$PKG/__pycache__/util.cpython-312.pyc" # The upstream unit tests, installed. See the header block. check $P/test-subdir-shipped test -d "$PKG/test" check $P/test-has-init test -f "$PKG/test/__init__.py" check $P/test-lib test -s "$PKG/test/lib.py" check $P/test-test-mman test -s "$PKG/test/test_mman.py" check $P/test-test-buf test -s "$PKG/test/test_buf.py" check $P/test-test-util test -s "$PKG/test/test_util.py" check $P/test-test-tutorial test -s "$PKG/test/test_tutorial.py" # Namespaced under smmap, NOT at the top level -- which is what makes it # harmless rather than a shadowing hazard. # # Asserted against smmap's OWN dpkg file list, not against the directory. The # question is whether THIS package claims a top-level `test'/`tests', and on a # shared image the path existing is not evidence either way: a sibling may ship # one legitimately, as python3-reedsolo does with $SITE/tests. Testing the # filesystem here asserted an image-wide property this port does not control. grep -E "^$SITE/test(/|\$)" "$PKGLIST" > "$W/tl-test" 2>/dev/null || : > "$W/tl-test" grep -E "^$SITE/tests(/|\$)" "$PKGLIST" > "$W/tl-tests" 2>/dev/null || : > "$W/tl-tests" check $P/no-toplevel-test test ! -s "$W/tl-test" check $P/no-toplevel-tests test ! -s "$W/tl-tests" # --- the package, as dpkg has it ------------------------------------------- cat "$PKGLIST" > "$W/pkgf" 2>/dev/null || : > "$W/pkgf" check $P/pkg-list-present test -s "$W/pkgf" # This image's python3 has no dynamic loader at all (BRINTOS-1476), so a .so in # this list would be a file that could never be imported. smmap has no # extension module upstream either, so this is two reasons for the same zero. check_eq $P/pkg-no-so "" \ "$(grep -E '\.so($|\.)' "$W/pkgf" 2>/dev/null)" # setup.py declares no entry_points and no scripts, so nothing lands in a bindir. check_eq $P/pkg-no-bin "" \ "$(grep -E '^/usr/(s?bin)/' "$W/pkgf" 2>/dev/null)" # Everything dpkg owns for this package is Python or wheel metadata. This is the # machine-readable form of "pure Python", and it is checked rather than assumed # because the whole cheapness of this port rests on it. _bad="" while IFS= read -r _f; do [ -f "$_f" ] || continue case "$_f" in *.py|*.pyc) continue ;; */__pycache__/*) continue ;; */*.dist-info/*) continue ;; esac _bad="$_bad $_f" done < "$W/pkgf" check_eq $P/pkg-only-python "" "$_bad" check_out $P/pkg-test-listed "$PKG/test/lib.py" cat "$W/pkgf" check_out $P/pkg-util-listed "$PKG/util.py" cat "$W/pkgf" sed -n '/^Package: python3-smmap$/,/^$/p' /var/lib/dpkg/status > "$W/st" 2>/dev/null check $P/pkg-known test -s "$W/st" check_out $P/pkg-installed "Status: install ok installed" cat "$W/st" check_out $P/pkg-version "Version: 6.0.0-r0" cat "$W/st" # The recipe's own RDEPENDS, which are the two interpreter sub-packages this # port actually needs. python3-mmap is the one every read below goes through. check_out $P/rdepends-mmap "python3-mmap" cat "$W/st" check_out $P/rdepends-codecs "python3-codecs" cat "$W/st" # --- the wheel metadata ---------------------------------------------------- check $P/dist-info-dir test -d "$DI" check $P/dist-info-metadata test -f "$DI/METADATA" check_out $P/dist-info-name "Name: smmap" cat "$DI/METADATA" check_out $P/dist-info-version "Version: 6.0.0" cat "$DI/METADATA" check $P/dist-info-record test -s "$DI/RECORD" check $P/dist-info-license test -s "$DI/LICENSE" # Purelib and a py3-none-any tag are the machine-readable form of "there is # nothing arch-specific in here", which is the whole reason this port is one # EMCRAFT_TOOLS line with no bbappend, no patch and no site answer. check_out $P/dist-info-purelib "Root-Is-Purelib: true" cat "$DI/WHEEL" check_out $P/dist-info-tag-py3 "Tag: py3-none-any" cat "$DI/WHEEL" check_eq $P/dist-info-no-py2-tag "" \ "$(grep '^Tag: py2' "$DI/WHEEL" 2>/dev/null)" # top_level.txt names smmap and nothing else -- smmap.test is a SUBpackage, so # its absence here is correct and is what says the shipped tests cannot shadow # a top-level name. check_out $P/dist-info-top-level "smmap" cat "$DI/top_level.txt" check_eq $P/dist-info-top-level-only "smmap" \ "$(tr -d '\r' < "$DI/top_level.txt" | sed '/^$/d' | tr '\n' ' ' | sed 's/ *$//')" check $P/dist-info-zip-safe test -f "$DI/zip-safe" # --- import ---------------------------------------------------------------- check $P/import python3 -c 'import smmap' # NOT a start each for smmap.util, smmap.mman and smmap.buf: the probe imports # all three in its own start and emits their __file__, so `util-answered', # `mman-answered' and `buf-answered' below already say they imported. Three # more interpreter starts to learn the same thing is three more chances at the # suite timeout, which is a batch-wide cost. # smmap imports `mmap' at util.py's top level, so an image without the # interpreter's mmap sub-package fails at IMPORT rather than at the first read. check $P/import-mmap-module python3 -c 'import mmap' # An import that WORKS but complains is a port that is not finished. Asserted # on its own start so that the noise, if any, is the whole of the evidence. check_eq $P/import-quiet "" \ "$(python3 -c 'import smmap' 2>&1)" # --- the probe ------------------------------------------------------------- if [ ! -f "$PROBE" ]; then bst_notok $P/probe-present "$PROBE did not ship" rm -rf "$W" exit 0 fi bst_ok $P/probe-present bst_probe $P/probe-run "$W/probe" 900 python3 "$PROBE" OUT="$W/probe" pv() { grep "^$1=" "$OUT" 2>/dev/null | head -n 1 | cut -d= -f2-; } if [ -s "$OUT" ]; then bst_ok $P/probe-output else bst_notok $P/probe-output "the probe emitted nothing -- python3 did not start, or died before its first line; every CASE below is unscorable" fi # The probe writes nothing to stderr on a healthy run. A ResourceWarning about # an unclosed mmap would land here and nowhere else. check_eq $P/probe-stderr-quiet "" "$(cat "$OUT.err" 2>/dev/null)" check_eq $P/probe-complete "yes" "$(pv probe_done)" check_eq $P/probe-count "264" "$(pv emitted_count)" # --- the module, as the interpreter sees it -------------------------------- check_eq $P/module-answered "$PKG/__init__.py" "$(pv module_file)" check_eq $P/util-answered "$PKG/util.py" "$(pv submodule_util_file)" check_eq $P/mman-answered "$PKG/mman.py" "$(pv submodule_mman_file)" check_eq $P/buf-answered "$PKG/buf.py" "$(pv submodule_buf_file)" check_eq $P/is-package "yes" "$(pv is_package)" check_eq $P/version "6.0.0" "$(pv version)" check_eq $P/version-info "6.0.0" "$(pv version_info)" check_eq $P/version-info-tuple "yes" "$(pv version_info_is_tuple)" check_eq $P/author "Sebastian Thiel" "$(pv author)" check_eq $P/contact "byronimo@gmail.com" "$(pv contact)" check_eq $P/homepage "https://github.com/gitpython-developers/smmap" "$(pv homepage)" # --- the public names ------------------------------------------------------ check_eq $P/root-has-static-manager "yes" "$(pv root_has_staticwindowmapmanager)" check_eq $P/root-has-sliding-manager "yes" "$(pv root_has_slidingwindowmapmanager)" check_eq $P/root-has-cursor "yes" "$(pv root_has_windowcursor)" check_eq $P/root-has-buffer "yes" "$(pv root_has_slidingwindowmapbuffer)" # THE TRAP -- see the header. Each name is asserted ABSENT at the root and # PRESENT in util, so the pair is one fact about the export list rather than two # guesses about where a name lives. check_eq $P/root-has-align-to-mmap "no" "$(pv root_has_align_to_mmap)" check_eq $P/util-has-align-to-mmap "yes" "$(pv util_has_align_to_mmap)" check_eq $P/root-has-is-64-bit "no" "$(pv root_has_is_64_bit)" check_eq $P/util-has-is-64-bit "yes" "$(pv util_has_is_64_bit)" check_eq $P/root-has-mapwindow "no" "$(pv root_has_mapwindow)" check_eq $P/util-has-mapwindow "yes" "$(pv util_has_mapwindow)" check_eq $P/root-has-mapregion "no" "$(pv root_has_mapregion)" check_eq $P/util-has-mapregion "yes" "$(pv util_has_mapregion)" check_eq $P/root-has-mapregionlist "no" "$(pv root_has_mapregionlist)" check_eq $P/util-has-mapregionlist "yes" "$(pv util_has_mapregionlist)" check_eq $P/root-has-granularity "no" "$(pv root_has_allocationgranularity)" check_eq $P/util-has-granularity "yes" "$(pv util_has_allocationgranularity)" # mman DOES hold those names as module globals: it imported them by name, and # only the star-import export list omits them. That is what makes the six cases # above an export decision rather than an absence. check_eq $P/mman-has-is-64-bit "yes" "$(pv mman_has_is_64_bit)" check_eq $P/mman-has-mapwindow "yes" "$(pv mman_has_mapwindow)" check_eq $P/root-has-no-all-attr "no" "$(pv root_has_all_attr)" check_eq $P/mman-all "StaticWindowMapManager,SlidingWindowMapManager,WindowCursor" \ "$(pv mman_all)" check_eq $P/buf-all "SlidingWindowMapBuffer" "$(pv buf_all)" check_eq $P/util-all \ "align_to_mmap,is_64_bit,MapWindow,MapRegion,MapRegionList,ALLOCATIONGRANULARITY" \ "$(pv util_all)" check_eq $P/test-subpackage-importable "yes" "$(pv test_subpackage_importable)" check_eq $P/test-subpackage-is-package "yes" "$(pv test_subpackage_is_package)" check_eq $P/sliding-is-static-subclass "yes" "$(pv sliding_is_static_subclass)" check_eq $P/regionlist-is-list "yes" "$(pv regionlist_is_list_subclass)" check_eq $P/window-has-slots "yes" "$(pv window_has_slots)" check_eq $P/window-no-dict "yes" "$(pv window_no_dict)" check_eq $P/region-has-slots "yes" "$(pv region_has_slots)" check_eq $P/cursor-has-slots "yes" "$(pv cursor_has_slots)" check_eq $P/buffer-has-slots "yes" "$(pv buffer_has_slots)" # --- the arm --------------------------------------------------------------- # NO LITERAL for the granularity, sys.maxsize, is_64_bit() or either default # size -- see the header. What is asserted is that they agree with each other. check_eq $P/granularity-power-of-two "yes" "$(pv granularity_is_power_of_two)" check_eq $P/granularity-divides-65536 "yes" "$(pv granularity_divides_65536)" check_eq $P/granularity-is-mmap-value "yes" "$(pv util_granularity_is_mmap_granularity)" check_eq $P/maxsize-one-of-two "yes" "$(pv maxsize_is_31_or_63_bit)" check_eq $P/is-64-bit-consistent "yes" "$(pv is_64_bit_consistent)" check_eq $P/arch-sliding-window-consistent "yes" "$(pv sliding_default_window_matches_arch)" check_eq $P/arch-sliding-budget-consistent "yes" "$(pv sliding_default_max_memory_matches_arch)" check_eq $P/arch-static-budget-consistent "yes" "$(pv static_default_max_memory_matches_arch)" check_eq $P/arch-static-window-consistent "yes" "$(pv static_explicit_minus_one_window_matches_arch)" # The one default that is NOT arch-derived: StaticWindowMapManager's window_size # defaults to 0, meaning "map the whole file", where SlidingWindowMapManager's # defaults to -1, meaning "ask the arch". Same keyword, two defaults. check_eq $P/static-default-window-zero "0" "$(pv static_default_window_size)" check_eq $P/static-default-window-is-zero "yes" "$(pv static_default_window_is_zero)" check_eq $P/default-max-handles-maxsize "yes" "$(pv default_max_handles_is_maxsize)" check_eq $P/sys-platform "linux" "$(pv sys_platform)" check_eq $P/force-removal-win-is-none "yes" "$(pv force_removal_win_is_none)" # asserts are what the over-long-slice cases below are actually measuring, so a # suite run under `python3 -O' would be measuring something else. This says # which of the two ran. check_eq $P/asserts-enabled "yes" "$(pv asserts_enabled)" check_eq $P/fresh-manager-memory "0" "$(pv fresh_manager_memory_size)" check_eq $P/fresh-manager-handles "0" "$(pv fresh_manager_handles)" check_eq $P/fresh-manager-open-files "0" "$(pv fresh_manager_open_files)" # --- align_to_mmap --------------------------------------------------------- # Every answer expressed as a multiple of the arm's own granularity, so these # are exact literals without a page size having been guessed. check_eq $P/align-0-down "0" "$(pv align_0_down_ratio)" check_eq $P/align-0-up "0" "$(pv align_0_up_ratio)" check_eq $P/align-1-down "0" "$(pv align_1_down_ratio)" check_eq $P/align-1-up "1" "$(pv align_1_up_ratio)" check_eq $P/align-g-minus-1-down "0" "$(pv align_g_minus_1_down_ratio)" check_eq $P/align-g-minus-1-up "1" "$(pv align_g_minus_1_up_ratio)" check_eq $P/align-g-down "1" "$(pv align_g_down_ratio)" check_eq $P/align-g-up "1" "$(pv align_g_up_ratio)" check_eq $P/align-g-plus-1-down "1" "$(pv align_g_plus_1_down_ratio)" check_eq $P/align-g-plus-1-up "2" "$(pv align_g_plus_1_up_ratio)" check_eq $P/align-3g-down "3" "$(pv align_3g_down_ratio)" check_eq $P/align-3g-up "3" "$(pv align_3g_up_ratio)" check_eq $P/align-3g-plus-7-up "4" "$(pv align_3g_plus_7_up_ratio)" check_eq $P/align-all-ok "yes" "$(pv align_all_ok)" # round_up is tested for TRUTH, not compared to True, so any truthy value rounds # up. Worth pinning: a caller passing a flags-style integer gets rounding. check_eq $P/align-round-up-truthy "yes" "$(pv align_round_up_is_truthy_test)" # --- MapWindow ------------------------------------------------------------- check_eq $P/window-ofs "10" "$(pv window_ofs)" check_eq $P/window-size "20" "$(pv window_size)" check_eq $P/window-ofs-end "30" "$(pv window_ofs_end)" check_eq $P/window-repr "MapWindow(10, 20)" "$(pv window_repr)" check_eq $P/window-align-ofs-ratio "1" "$(pv window_align_ofs_ratio)" check_eq $P/window-align-size-ratio "1" "$(pv window_align_size_ratio)" # align()'s contract in one case: whatever it does to ofs and size, the area the # window covered before must still be covered after. check_eq $P/window-align-still-covers "yes" "$(pv window_align_still_covers)" check_eq $P/window-extend-left-ofs "200" "$(pv window_extend_left_ofs)" check_eq $P/window-extend-left-size "900" "$(pv window_extend_left_size)" check_eq $P/window-extend-left-capped-ofs "800" "$(pv window_extend_left_capped_ofs)" check_eq $P/window-extend-left-capped-size "300" "$(pv window_extend_left_capped_size)" check_eq $P/window-extend-right-size "4000" "$(pv window_extend_right_size)" check_eq $P/window-extend-right-capped-size "200" "$(pv window_extend_right_capped_size)" # --- the pattern generator itself ------------------------------------------ # The constants that stop everything below being a comparator agreeing with # itself. pat(i) = (i * 37 + 11) & 0xFF. check_eq $P/pattern-at-0 "11" "$(pv pattern_at_0)" check_eq $P/pattern-at-1 "48" "$(pv pattern_at_1)" check_eq $P/pattern-at-2 "85" "$(pv pattern_at_2)" check_eq $P/pattern-at-100 "127" "$(pv pattern_at_100)" check_eq $P/pattern-at-255 "230" "$(pv pattern_at_255)" check_eq $P/pattern-at-256 "11" "$(pv pattern_at_256)" check_eq $P/pattern-at-4095 "230" "$(pv pattern_at_4095)" check_eq $P/pattern-at-65535 "230" "$(pv pattern_at_65535)" check_eq $P/pattern-at-199999 "38" "$(pv pattern_at_199999)" check_eq $P/pattern-first16 "0b30557a9fc4e90e33587da2c7ec1136" "$(pv pattern_first16_hex)" # Period 256, so the same sixteen bytes recur at 65536 -- which is why the # window-boundary reads below cannot be passed by a comparator that has simply # memorised the file's first bytes. check_eq $P/pattern-period "256" "$(pv pattern_period)" check_eq $P/pattern-at-65536 "0b30557a9fc4e90e33587da2c7ec1136" "$(pv pattern_at_65536_16_hex)" # --- the files and the manager the reads run against ----------------------- check_eq $P/file-size "200000" "$(pv file_size)" check_eq $P/file-size-expected "yes" "$(pv file_size_expected)" check_eq $P/empty-file-size "0" "$(pv empty_file_size)" check_eq $P/one-byte-file-size "1" "$(pv one_byte_file_size)" check_eq $P/configured-window "65536" "$(pv configured_window)" check_eq $P/configured-budget "131072" "$(pv configured_budget)" check_eq $P/window-divides-file "yes" "$(pv window_divides_file)" check_eq $P/cursor-associated "yes" "$(pv cursor_associated)" # A fresh cursor is NOT valid: it is associated with a file and points at no # region until the first use_region. check_eq $P/cursor-invalid-before-use "no" "$(pv cursor_valid_before_use)" check_eq $P/cursor-file-size "200000" "$(pv cursor_file_size)" check_eq $P/cursor-path "yes" "$(pv cursor_path)" check_eq $P/cursor-path-or-fd-is-str "yes" "$(pv cursor_path_or_fd_is_str)" # fd() on a path cursor and path() on an fd cursor each refuse rather than # guessing. The pair is the whole of the path/fd distinction. check_eq $P/cursor-fd-on-path-refuses "!ValueError" "$(pv cursor_fd_on_path_cursor)" check_eq $P/buffer-len "200000" "$(pv buffer_len)" check_eq $P/buffer-len-is-file-size "yes" "$(pv buffer_len_is_file_size)" check_eq $P/buffer-cursor-is-ours "yes" "$(pv buffer_cursor_is_ours)" # --- the reads ------------------------------------------------------------- # 1. wholly inside one window. check_eq $P/read-inside-len "1000" "$(pv read_inside_len)" check_eq $P/read-inside-ok "yes" "$(pv read_inside_ok)" check_eq $P/read-inside-first8 "7fa4c9ee13385d82" "$(pv read_inside_first8_hex)" # 2. across the 65536 window boundary. The seam bytes are pinned as DATA, so # this case says the two halves were joined in the right order and not merely # that something 1000 bytes long came back. check_eq $P/read-span-len "1000" "$(pv read_span_len)" check_eq $P/read-span-ok "yes" "$(pv read_span_ok)" check_eq $P/read-span-seam "779cc1e60b30557a" "$(pv read_span_seam_hex)" # 3. across three regions at once. check_eq $P/read-three-regions-len "131092" "$(pv read_three_regions_len)" check_eq $P/read-three-regions-ok "yes" "$(pv read_three_regions_ok)" # 4. the exact start of the file. check_eq $P/read-at-start-len "64" "$(pv read_at_start_len)" check_eq $P/read-at-start-ok "yes" "$(pv read_at_start_ok)" check_eq $P/read-at-start-hex "0b30557a9fc4e90e33587da2c7ec1136" "$(pv read_at_start_hex)" # 5. the exact end of the file. check_eq $P/read-at-end-len "64" "$(pv read_at_end_len)" check_eq $P/read-at-end-ok "yes" "$(pv read_at_end_ok)" check_eq $P/read-at-end-last8 "23486d92b7dc0126" "$(pv read_at_end_last8_hex)" # 6. the whole file in one slice -- every byte, across all four regions. check_eq $P/read-whole-len "200000" "$(pv read_whole_len)" check_eq $P/read-whole-ok "yes" "$(pv read_whole_ok)" # 7. the two return types. See the header. check_eq $P/read-fastpath-type "memoryview" "$(pv read_fastpath_type)" check_eq $P/read-fastpath-ok "yes" "$(pv read_fastpath_ok)" check_eq $P/read-slowpath-type "bytes" "$(pv read_slowpath_type)" check_eq $P/read-slowpath-ok "yes" "$(pv read_slowpath_ok)" check_eq $P/read-paths-differ "yes" "$(pv read_paths_differ_in_type)" check_eq $P/read-fastpath-concat "!TypeError" "$(pv read_fastpath_concat)" check_eq $P/read-slowpath-concat "ok" "$(pv read_slowpath_concat)" # 8. zero length, on both paths. check_eq $P/read-zero-len "0" "$(pv read_zero_len)" check_eq $P/read-zero-is-empty "yes" "$(pv read_zero_is_empty)" check_eq $P/read-zero-type "memoryview" "$(pv read_zero_type)" check_eq $P/read-zero-slowpath-type "bytes" "$(pv read_zero_slowpath_type)" check_eq $P/read-zero-slowpath-empty "yes" "$(pv read_zero_slowpath_is_empty)" # 9. the exact-fit read at EOF works -- established BEFORE the overrun cases, so # that what the overrun does to a buffer cannot be mistaken for EOF being # unreadable. check_eq $P/read-last-10-len "10" "$(pv read_last_10_len)" check_eq $P/read-last-10-ok "yes" "$(pv read_last_10_ok)" # 10. past the end. All four RAISE; none comes back short. See the header for # why the int index fails differently from the slice. check_eq $P/read-past-end-overlong "!AssertionError" "$(pv read_past_end_overlong)" check_eq $P/read-slice-at-eof "!AssertionError" "$(pv read_slice_at_eof)" check_eq $P/read-slice-past-eof "!AssertionError" "$(pv read_slice_past_eof)" check_eq $P/int-index-past-end "!AttributeError" "$(pv int_index_past_end)" # 11. the overrun poisons the buffer. check_eq $P/poison-read-before "yes" "$(pv poison_read_before_ok)" check_eq $P/poison-overrun-raised "AssertionError" "$(pv poison_overrun_raised)" check_eq $P/poison-cursor-invalid "yes" "$(pv poison_cursor_invalid_after)" check_eq $P/poison-read-after "!AssertionError" "$(pv poison_read_after)" check_eq $P/poison-fresh-buffer "yes" "$(pv poison_fresh_buffer_ok)" # 12. negative indices, relative to the buffer's length. check_eq $P/read-negative-len "16" "$(pv read_negative_len)" check_eq $P/read-negative-ok "yes" "$(pv read_negative_ok)" # 13. integer indexing. check_eq $P/int-index-0 "11" "$(pv int_index_0)" check_eq $P/int-index-0-ok "yes" "$(pv int_index_0_ok)" check_eq $P/int-index-1000 "yes" "$(pv int_index_1000_ok)" check_eq $P/int-index-across-window "yes" "$(pv int_index_across_window_ok)" check_eq $P/int-index-type "int" "$(pv int_index_type)" check_eq $P/int-index-negative "yes" "$(pv int_index_negative_ok)" # the negative controls -- see the header. check_eq $P/control-mismatch-detected "yes" "$(pv control_mismatch_detected)" check_eq $P/control-length-mismatch-detected "yes" "$(pv control_length_mismatch_detected)" check_eq $P/control-pattern-not-constant "yes" "$(pv control_pattern_not_constant)" # end_access() zeroes the reported length. A buffer is not reusable afterwards # without a begin_access, and __len__ is what says so. check_eq $P/buffer-len-after-end-access "0" "$(pv buffer_len_after_end_access)" # 14. THE DOCSTRING/CODE DISAGREEMENT, on a buffer built at offset 1000. check_eq $P/offset-buffer-len "5000" "$(pv offset_buffer_len)" check_eq $P/offset-buffer-absolute-slice "yes" "$(pv offset_buffer_absolute_slice_ok)" check_eq $P/int-index-offset-buffer-absolute "yes" "$(pv int_index_offset_buffer_is_absolute)" check_eq $P/int-index-offset-buffer-not-relative "yes" "$(pv int_index_offset_buffer_is_not_relative)" check_eq $P/int-index-offset-buffer-0-is-file-0 "yes" "$(pv int_index_offset_buffer_0_is_file_offset_0)" # --- the manager's own accounting ------------------------------------------ # Asserted, not assumed: the whole value of this package over a bare mmap is # that it bounds what it holds open, so the bound is what has to be measured. check_eq $P/acct-handles-before "0" "$(pv acct_handles_before)" check_eq $P/acct-memory-before "0" "$(pv acct_memory_before)" check_eq $P/acct-open-files-before "0" "$(pv acct_open_files_before)" check_eq $P/acct-valid-after-first-use "yes" "$(pv acct_valid_after_first_use)" check_eq $P/acct-handles-after-1 "1" "$(pv acct_handles_after_1)" check_eq $P/acct-region-size-1 "65536" "$(pv acct_region_size_1)" check_eq $P/acct-region-size-is-window "yes" "$(pv acct_region_size_is_window)" # The same fact restated so it survives a granularity larger than the configured # window: if the case above ever goes red on an arm, this one stays green and # says the cause was the granularity rather than the manager. check_eq $P/acct-region-size-window-or-granularity "yes" \ "$(pv acct_region_size_is_window_or_granularity)" check_eq $P/acct-memory-after-1 "65536" "$(pv acct_memory_after_1)" check_eq $P/acct-memory-is-window "yes" "$(pv acct_memory_after_1_is_window)" check_eq $P/acct-region-ofs-begin-1 "0" "$(pv acct_region_ofs_begin_1)" check_eq $P/acct-region-includes-0 "yes" "$(pv acct_region_includes_0)" check_eq $P/acct-region-excludes-window "yes" "$(pv acct_region_excludes_window)" check_eq $P/acct-cursor-ofs-begin "0" "$(pv acct_cursor_ofs_begin)" check_eq $P/acct-cursor-size "16" "$(pv acct_cursor_size)" check_eq $P/acct-cursor-includes-0 "yes" "$(pv acct_cursor_includes_0)" # Two: the manager's own reference, and the cursor's. check_eq $P/acct-region-client-count "2" "$(pv acct_region_client_count_1)" check_eq $P/acct-open-files-after-1 "1" "$(pv acct_open_files_after_1)" check_eq $P/acct-handles-after-2 "2" "$(pv acct_handles_after_2)" check_eq $P/acct-memory-2-windows "yes" "$(pv acct_memory_after_2_is_2_windows)" check_eq $P/acct-region-ofs-begin-2 "65536" "$(pv acct_region_ofs_begin_2)" # The budget is two windows, so the THIRD distinct region has to evict one. The # handle count not climbing to 3 is the eviction, measured. check_eq $P/acct-handles-after-3 "2" "$(pv acct_handles_after_3)" check_eq $P/acct-handles-bounded "yes" "$(pv acct_handles_bounded_by_budget)" check_eq $P/acct-memory-within-budget "yes" "$(pv acct_memory_within_budget)" check_eq $P/acct-region-ofs-begin-3 "131072" "$(pv acct_region_ofs_begin_3)" # Re-reading a region that WAS evicted gives the same bytes back. This is the # case the whole eviction machinery exists to keep true. check_eq $P/acct-reread-evicted-len "32" "$(pv acct_reread_evicted_len)" check_eq $P/acct-reread-evicted-ok "yes" "$(pv acct_reread_evicted_ok)" check_eq $P/acct-reread-evicted-hex "0b30557a9fc4e90e33587da2c7ec1136" \ "$(pv acct_reread_evicted_hex)" check_eq $P/acct-invalid-after-unuse "no" "$(pv acct_valid_after_unuse)" check_eq $P/acct-collect-freed-some "yes" "$(pv acct_collect_freed_some)" check_eq $P/acct-handles-after-collect "0" "$(pv acct_handles_after_collect)" check_eq $P/acct-memory-after-collect "0" "$(pv acct_memory_after_collect)" check_eq $P/acct-handles-return-to-zero "yes" "$(pv acct_handles_return_to_zero)" check_eq $P/acct-memory-returns-to-zero "yes" "$(pv acct_memory_returns_to_zero)" check_eq $P/acct-collect-twice-zero "yes" "$(pv acct_collect_twice_is_zero)" # A sequential walk of all 200000 bytes in 4096-byte reads through a manager # budgeted at two windows: every byte correct, and the peak never above the # budget. This is the package's headline claim, run end to end. check_eq $P/walk-bytes-ok "yes" "$(pv walk_bytes_ok)" check_eq $P/walk-end-pos "200000" "$(pv walk_end_pos)" check_eq $P/walk-covered-file "yes" "$(pv walk_covered_whole_file)" check_eq $P/walk-peak-memory "131072" "$(pv walk_peak_memory)" check_eq $P/walk-peak-in-budget "yes" "$(pv walk_peak_within_budget)" check_eq $P/walk-handles-freed "0" "$(pv walk_handles_after_collect)" # max_open_handles is a limit in its own right, independent of the memory # budget, and a read still works with only one handle allowed. check_eq $P/acct-max-handles-1 "1" "$(pv acct_max_handles_1)" check_eq $P/acct-handles-capped "yes" "$(pv acct_handles_capped_at_1)" check_eq $P/acct-capped-read-ok "yes" "$(pv acct_capped_read_ok)" # --- the static manager: one region, the whole file ------------------------ check_eq $P/static-region-size "200000" "$(pv static_region_size)" check_eq $P/static-region-whole-file "yes" "$(pv static_region_is_whole_file)" check_eq $P/static-handles "1" "$(pv static_handles)" check_eq $P/static-read-at-end-ok "yes" "$(pv static_read_at_end_ok)" check_eq $P/static-handles-after-2 "1" "$(pv static_handles_after_second_use)" check_eq $P/static-never-grows "yes" "$(pv static_never_grows)" check_eq $P/static-handles-collected "0" "$(pv static_handles_after_collect)" # --- a cursor built on a file descriptor ----------------------------------- check_eq $P/fd-cursor-valid "yes" "$(pv fd_cursor_valid)" check_eq $P/fd-cursor-read-ok "yes" "$(pv fd_cursor_read_ok)" check_eq $P/fd-cursor-file-size "200000" "$(pv fd_cursor_file_size)" check_eq $P/fd-cursor-fd-matches "yes" "$(pv fd_cursor_fd_matches)" check_eq $P/fd-cursor-path-refuses "!ValueError" "$(pv fd_cursor_path_raises)" check_eq $P/fd-cursor-path-or-fd-is-int "yes" "$(pv fd_cursor_path_or_fd_is_int)" # --- error paths ----------------------------------------------------------- # A path that does not exist. make_cursor does NOT touch the filesystem, so # nothing raises until the first call that needs the file's size -- which is a # real trap: the cursor for a missing file looks fine. check_eq $P/missing-make-cursor-ok "yes" "$(pv missing_make_cursor_ok)" check_eq $P/missing-associated "yes" "$(pv missing_associated)" check_eq $P/missing-use-region "!FileNotFoundError" "$(pv missing_use_region)" check_eq $P/missing-file-size "!FileNotFoundError" "$(pv missing_file_size)" check_eq $P/missing-buffer "!FileNotFoundError" "$(pv missing_buffer)" # A zero-length file. use_region does NOT raise: offset 0 is already >= the # size, so the cursor comes back INVALID and it is the buffer that refuses. A # caller who tests only for an exception sees nothing wrong here. check_eq $P/empty-file-size-via-cursor "0" "$(pv empty_file_size_via_cursor)" check_eq $P/empty-use-region-no-raise "yes" "$(pv empty_use_region_no_raise)" check_eq $P/empty-cursor-invalid "yes" "$(pv empty_cursor_invalid)" check_eq $P/empty-no-handle "0" "$(pv empty_handles)" check_eq $P/empty-buffer-refuses "!ValueError" "$(pv empty_buffer)" # One byte -- the smallest file that maps at all, and the case where the region # is smaller than both the window and the granularity. check_eq $P/one-byte-cursor-valid "yes" "$(pv one_byte_cursor_valid)" check_eq $P/one-byte-cursor-size "1" "$(pv one_byte_cursor_size)" check_eq $P/one-byte-region-size "1" "$(pv one_byte_region_size)" check_eq $P/one-byte-read-ok "yes" "$(pv one_byte_read_ok)" check_eq $P/one-byte-buffer-len "1" "$(pv one_byte_buffer_len)" check_eq $P/one-byte-buffer-read-ok "yes" "$(pv one_byte_buffer_read_ok)" # An offset at or past EOF on a real file: invalid cursor, and no handle taken. check_eq $P/offset-at-eof-invalid "yes" "$(pv offset_at_eof_invalid)" check_eq $P/offset-past-eof-invalid "yes" "$(pv offset_past_eof_invalid)" check_eq $P/offset-past-eof-no-handle "yes" "$(pv offset_past_eof_no_handle)" check_eq $P/offset-last-byte-valid "yes" "$(pv offset_last_byte_valid)" check_eq $P/offset-last-byte-size "1" "$(pv offset_last_byte_size)" check_eq $P/buffer-offset-past-eof "!ValueError" "$(pv buffer_offset_past_eof)" # A directory. os.open succeeds and mmap is what refuses, so the error arrives # as an OSError from the kernel rather than from smmap -- which makes this one # of the few cases here whose ANSWER comes from the platform. A different # errno-bearing exception type on this arm is a platform result, not a port # defect. check_eq $P/directory-refuses "!OSError" "$(pv directory_use_region)" # --- cursor copy semantics ------------------------------------------------- check_eq $P/copy-valid "yes" "$(pv copy_valid)" check_eq $P/copy-read-ok "yes" "$(pv copy_read_ok)" check_eq $P/copy-is-distinct "yes" "$(pv copy_is_distinct_object)" # Three now: the manager, the original cursor, and the copy. A copy that did not # take its own reference would be a use-after-free waiting for the first evict. check_eq $P/copy-client-count "3" "$(pv copy_client_count)" check_eq $P/probe-tmpdir-removed "yes" "$(pv tmpdir_removed)" rm -rf "$W"