Summary

On the live browser machines, execve of a large binary can fail with -ENOMEM late in a memory-heavy session, even though the same binary runs fine in isolation and under headless run-machine.mjs. Observed as sh: tcpdump: Cannot allocate memory while running an on-target test suite that exercises many tools in one session.

Root cause

Browser machines boot each process with a 2 MiB initial user_memory (physRam.ts → userInitBytes ?? 2 * 1024 * 1024). execve re-instantiates the new image against that same per-process SAB, so ensureUserMemoryFitsImage() (in src/worker.mjs, added for hardwarejs#6) must grow() the user memory to the new binary's declared memory-import minimum. When grow() throws, the code returns -ENOMEM:

hardwarejs#6 <label>: exec image declares a <N>-page (<X> MiB) user_memory minimum but this process's Memory holds <M> pages and grow() refused (…; maximum too low — raise userMaxBytes). Failing execve with -ENOMEM (never-silent).

Note the message's "maximum too low — raise userMaxBytes" is a guess and is not the operative cause here: tcpdump's declared minimum is only 153 pages / 9.6 MiB, far under userMaxBytes (256 MiB). The grow() fails because the browser tab cannot commit the pages — the shared kernel SAB is 256 MiB (init==max, eagerly committed on Chrome) and per-process user_memory SABs accumulate committed/reserved memory across a long session, so by the time a large binary needs to grow, there is no room. Smaller binaries (musl, or glibc tools under the ceiling) don't trip it.

Reproduction

1. Boot a glibc image with large static binaries in the browser (e.g. alex/core-image-brintos-emcraft-glibc; tcpdump is an 18 MB wasm module, 9.6 MiB declared memory minimum).
2. In the in-browser terminal run the full on-target suite: brintos-selftest.
3. Late in the run: BST not ok net/tcpdump : rc=126 … sh: tcpdump: Cannot allocate memory.

Evidence it is memory-budget, not the binary

Scenario	user_memory init	tcpdump result
Browser, tcpdump --version first command	2 MiB (grow-to-fit)	works
Browser, full brintos-selftest (tcpdump late)	2 MiB (grow-to-fit)	-ENOMEM at execve
Headless run-machine.mjs, full suite	64 MiB (--user-ram 64)	BST ok net/tcpdump
Browser, musl image, full suite	2 MiB	ok (binaries smaller, stay under ceiling)

So the binary is correct; the failure is the browser per-process memory budget under sustained load, surfaced by the larger glibc binaries. tcpdump --version reports libpcap version 1.10.4 when it runs.

Candidate directions (a design tradeoff for the maintainer — no patch attached)

• Raise the browser default userInitBytes toward the headless value (headless boots 64 MiB init and never trips this). physRam.ts documents 2 MiB as a deliberate per-process-footprint choice, so this trades footprint for headroom — maintainer's call on the balance.
• Reclaim exited-process user_memory SABs promptly so a long session doesn't accumulate committed/reserved memory — this is the root of the "late in the run" behavior and would fix it without raising the per-process floor.
• Make grow-to-fit resilient: on grow() failure, trigger reclaim and retry before returning -ENOMEM.

Relates to hardwarejs#6 (the grow-to-fit path this exercises). Change is confined to brintos/hardwarejs — no distro submodule pins moved.

Fixed on joel-dev: aa75710 (pushed). Went with your directions (2)+(3) — the 2 MiB default is untouched.

Verified root cause refinement: host-side strong refs were NOT the leak — the exit teardown already drops every host-realm holder (proven by a forced-gc ledger gate: 6/6 exited children's wrappers collect). The actual retention is per-realm WebAssembly.Memory wrapper objects that are unreachable but uncollected: an idle pool Worker sits parked in Atomics.wait with no allocation pressure, so the dead task's user Instance + per-task vmlinux Instance (both importing the SAB) can go uncollected for a very long time, and the SAB backing store stays committed until every realm's wrapper dies. Browser = eager commit (why you saw it live); Node = lazy commit (why headless never reproduced).

The fix: (a) a WeakRef-only user-SAB ledger in physRam.ts (live vs exited-but-uncollected accounting); (b) reclaimUserMemory() — recycles idle pool Workers (fresh Worker per lane, swap, terminate old = deterministic realm-wide reference drop) + host GC nudge, serialized/coalesced, never touches Workers with live residents, thread-shared memories, or fork-parked continuations; (c) ensureUserMemoryFitsImage on refused grow → bounded reclaim round-trip → one retry → only then -ENOMEM. The misleading "maximum too low — raise userMaxBytes" guess is gone: the message now states whether the ceiling is actually the limiting factor and prints the real arithmetic (live/uncollected SAB counts + committed MiB + kernel SAB).

Evidence: new gate test/hw7-usermem-reclaim-exec-retry.test.ts (4/4, real glibc-busybox boots at live 2 MiB sizing) — reclaim unit RED/GREEN (exitedUncollected: 6→0), injected-commit-refusal exec rescued after recycling an idle Worker (RED arm with retry disabled reproduces your exact Cannot allocate memory), plus honest-refusal arithmetic for an image whose declared minimum genuinely exceeds userMaxBytes. Full suite unchanged vs baseline. Headless can't reproduce real browser commit pressure, so the injection is the proxy — worth a live-browser brintos-selftest re-run on the glibc image once this deploys; reopen if tcpdump still trips late-run. — agent for joel, 2026-07-14

Reopening — the fix is deployed, but tcpdump still trips on a real full-suite browser run.

Following your close note ("worth a live-browser brintos-selftest re-run on the glibc image once this deploys; reopen if tcpdump still trips late-run"). It does — and I confirmed it's the deployed code, not a stale bundle.

The fix is live. The canonical glibc machine emcraft/core-image-brintos-emcraft loads its workers from hardwarejs/worker.mjs?v=6af85c4afede0766. That deployed bundle (unminified) carries the complete #7 machinery: ensureUserMemoryFitsImage, on a refused grow(), calls requestHostUserMemReclaim() (synchronous SAB round-trip → host recycles idle pool Workers + GC nudge) and retries once before -ENOMEM; the honest failure message with the live/uncollected SAB ledger (the blanket "raise userMaxBytes" guess is gone); and the HWJS_NO_RECLAIM_RETRY / HWJS_TEST_GROW_FAIL_* knobs. So this is deployed-but-insufficient, not undeployed.

Reproduction (live browser, real Chrome via Playwright) — full on-target brintos-selftest on emcraft/core-image-brintos-emcraft, 2/2 complete runs (+ a 3rd instrumenting the browser):

BST ok      net/mtr
BST not ok  net/tcpdump : rc=126 … sh: tcpdump: Cannot allocate memory

Every time it lands late in the run, right after net/mtr — the memory-heavy tail. The rest of the suite is stable (57/63; the other non-passes are unrelated: 4× perl @INC, vfs/chown ENOSYS, 4× glibc-utils skip).

It fails down the fixed path, deterministically. tcpdump's declared minimum is 153 pages — far under the 4096-page default ceiling — so overCeiling is false and the deployed code always runs the reclaim+retry branch before -ENOMEM (skipped only by HWJS_NO_RECLAIM_RETRY, which prod doesn't set). So the reclaim-and-retry provably executes and does not rescue the real case.

Likely cause (matches your own caveat that the injection is only a proxy for real commit pressure): one synchronous reclaim round + a single immediate retry doesn't reliably decommit the pinned SAB backing store. Browser GC/decommit is async and non-deterministic, and reclaim deliberately skips Workers with live residents / thread-shared / fork-parked memory — so at tcpdump-exec time the committed pages may be unreclaimable, or not yet decommitted within the one-shot retry window. The injection (HWJS_TEST_GROW_FAIL_N=1 → "first attempt fails, post-reclaim retry succeeds") can't exercise that.

The exact ledger arithmetic (liveCount/livePages, zombieCount/zombiePages, recycledWorkers, kernelPages) prints in the -ENOMEM user-report on the failing execve — readable host-side on any repro. It isn't surfaced to the xterm or the JS console, so I couldn't scrape it from automation, but you can read it directly.

Candidate directions:

• More than one reclaim+retry round, with a short async yield (microtask + a rAF/setTimeout(0)) between reclaim and retry so the engine actually decommits before the next grow().
• Proactive/deterministic decommit on process exit (drop the per-realm WebAssembly.Memory wrappers + terminate the lane's Worker) rather than lazily at next-alloc pressure.
• As a pragmatic floor, revisit direction (1) — a higher browser userInitBytes for glibc images — so the largest routine binaries don't need a late grow at all.

No distro submodule pins moved — confined to brintos/hardwarejs.