Repro

1. Log out of brintos.io (or open an incognito window).
2. Open a public machine, e.g. dmitry/core-image-minimal-glibc, click Power on, wait for the shell prompt.
3. Type any command, e.g. cat /proc/version.

Observed

The command hangs forever. Browser console:

/api/repos/dmitry/core-image-minimal-glibc/fs/blobs/presign: 403
virtio_fs: backend error -> errno 13: fs write refused (HTTP 403 —
no write permission on this repo): Request failed with 403.

Cause chain

busybox ash saves command history to /.ash_history after every input line (FEATURE_EDITING_SAVEHISTORY without SAVE_ON_EXIT); the write triggers a blob presign, which is 403 for an anonymous session. The backend logs "fs write refused" errno 13 — but the guest process stays blocked: the error apparently never completes the virtio request back to the guest. The same image + commands under run-machine.mjs against a writable local directory work; and if the guest had actually received EACCES, ash would continue (it ignores history-save failures).

Impact

Every public machine hangs for logged-out visitors on their first shell command — the public demo path.

Expected

Complete the refused request with -EACCES so the guest write fails cleanly and the shell keeps working (history save silently skipped).

Secondary suggestion

An ephemeral copy-on-write overlay for anonymous sessions would make public machines fully usable without repo writes.

Note

A hang with the same symptom (first spawned command never returns, console clean, no 403) is also observed for the repo owner logged in; under separate investigation — possibly the same non-completion path with a different trigger.

Fixed in db40e58 (joel-dev), deployed to production, and verified end-to-end against this exact machine. Closing with the evidence below.

Root cause — not the virtio-fs backend

The hypothesis in this ticket (the 403/EACCES refusal never completes the guest request) is disproven with evidence: the FUSE dispatch awaits every mutator, so sync AND async (rejected-promise-after-presign) refusals already complete the request with the attributed errno. Proven at every layer: unit gates over the wire protocol, node boots of the musl and glibc rigs, and a boot of a copy of this machine's actual /boot/vmlinux.wasm + rootfs — in all of them the refused write surfaces as Permission denied and ash/bash continues. The errno 13 console line was a coincident symptom — which the ticket's own Note anticipated (the logged-in owner hangs identically with a clean console).

The real defect is in the spawn pool placement (still hardwarejs, different subsystem):

1. This machine's spec is cpu: 2 → 2 pool Workers (lanes cpu=1, cpu=2). busybox init (pid 1, lane cpu=1) polls waitpid(-1, NULL, WNOHANG) in a tight loop — traced live: an unbroken stream of wait4 → 0, no park, no sleep between polls.
2. A pool lane drives its residents' wasm synchronously; the JS event loop only turns when a resident parks. A resident that never parks starves that Worker's event loop indefinitely — onmessage never fires.
3. Your first command (cat /proc/version) forks; the child's add-task placement avoided the parent's lane (cpu=2) and went to init's lane (cpu=1) — the only other candidate at cpu=2. The message was never processed, the fork parent stayed parked on its ctrl SAB, and the shell hung forever. Deterministic for owner and anonymous alike; the pool-size default (min(cores, 8)) is why nobody saw it on dev boxes — children land on empty lanes there.

Placement trace of the hang (this machine's kernel+rootfs, cpu=2): place tid=0 ppid=28 -> pool cpu=1 routeKey=3 … and no addTask routeKey=3 ever — starved.

Fix (hardwarejs db40e58)

Claim handshake + re-placement, all mechanisms genuine:

• Every pool add-task carries a claim word (i32 SAB): 0 posted, 1 canceled-by-host, 2 claimed-by-lane. The lane CASes 0→2 before any side effect; observing 1 declines the stale placement loudly with no side effects.
• The spawn handler waits 250 ms for the claim; on timeout it CASes 0→1 (exactly one side wins), marks the lane suspect (30 s cooldown so later spawns skip the stall), and re-places on the next candidate. The parent's lane is a legitimate fallback (requestForkChild waits with Atomics.waitAsync, so its event loop is live during the fork — the old avoid-parent rationale predated that).
• If every lane is starved it keeps cycling with a loud per-cycle dmesg — never a silent hang.

RED → GREEN

• RED: copy of dmitry/core-image-minimal-glibc (its own old kernel AND the current kernel), cpu=2, live anonymous cmdline: cat /proc/version hangs forever (watchdog), across async-403 / sync-403 / no-refusal — refusals exonerated, spawn starvation confirmed.
• GREEN (same rig, fixed runtime): all variants CLEAN, incl. a multi-command line with a refused history write — sh: write error: Permission denied, HW3-RC=0, shell keeps working. Trace shows re-place tid=0 routeKey=3 cpu=1 -> cpu=2 (lane starved) and the follow-up spawns skipping the stall via suspect memory.
• New gates (both RED arms proven flipping): test/a4c-addtask-claim.test.ts (claim/decline contract — pre-fix the claim word stays 0 and a canceled spec double-instantiates) and test/virtio-fs-errno-async.test.ts (this ticket's stated contract pinned: async-rejecting mutators must complete with the attributed errno — fails if FuseServer.handle ever stops awaiting dispatch). Full hardwarejs vitest suite: no regressions (the 15 pre-existing failures on joel-dev are unchanged).
• Distro harnesses committed at superproject 329c381: distros/linux-6.12-{musl-bash-coreutils,glibc-bash-coreutils}/tools/repro-hw3-*.mjs (async/sync refusal completion, end to end).

Live production confirmation

Deployed (make web && make web-sync, sst deploy --stage production). Headless-Chrome E2E as a logged-out visitor on dmitry/core-image-minimal-glibc: machine boots, cat /proc/version prints Linux version 6.12.0-wasm32-v0.1-g12fce771e201 …, echo HW3-RC=$? → HW3-RC=0, prompt returns. The page's diag log shows the fix engaging on the live path: hwjs#3: add-task tid=0 routeKey=3 claim timed out on starved pool lane cpu=1 (resident spinning without a yield); re-placing on cpu=2, and the two history-save 403s complete as EACCES with the shell continuing.

Notes / follow-ups (not this ticket)

• Why busybox init spins WNOHANG without sleeping (rather than blocking in wait4) is guest-side behavior worth its own look; the host now survives any such spinner.
• The copy-on-write overlay for anonymous sessions remains #58.
• Observed while building the glibc harness: after a shell redirects fd 1, onUserOutput low-fd routing swaps which host channel carries fd-1 bytes (pre-existing, unrelated to this fix; the harness works around it).