brintos

brintos / hardwarejs public Read only

0
0

browser fork path: spawning a second user Worker for a 15 MB glibc busybox image never completes — fork alone ((echo x) subshell) hangs; same image forks fine under run-machine.mjs #4

Closed dmitry opened this issue · 1 comment
D dmitry commented

Repro

  1. Machine: dmitry/core-image-minimal-glibc (Yocto core-image-minimal, glibc/wasm32-hwjs, busybox 15.2 MB merged module; kernel g12fce771e201 — the same build alex/core-image-minimal-musl runs).
  2. Open it logged-in on brintos.io, Power on — boots to the shell prompt.
  3. Type (echo subshell-ok) — a subshell fork with a builtin, no exec.

Observed

The fork never returns — the shell hangs. Same for any fork+exec (cat /proc/version, uname). Browser console is clean — no errors, no 403s. Boot-time spawns of the very same 15.2 MB module (/init, -/bin/sh) complete fine; builtins and virtio-fs reads work.

Isolation matrix (all on the same site, same browser session)

  • alex/core-image-minimal-musl (10.6 MB musl busybox, same kernel): fork OK
  • dmitry/core-image-minimal-glibc (15.2 MB glibc busybox): fork HANGS
  • the same glibc rootfs + kernel under hardwarejs run-machine.mjs on a build host (worker.mjs byte-identical to the site's, only the ?v= cache-buster differs): fork OK
  • glibc bash machines (~10.3 MB mains, e.g. linux-6.12-glibc- bash-coreutils): fork OK in browser

Ruled out empirically: kernel build (A/B'd to g12fce771e201), fs writes (HISTFILE unset and verified live; also hangs for anonymous/403 sessions — see #3), fs-tree shape (entry fields identical to the musl machine, raw encoding both), site worker.mjs version (byte-identical to the passing local one).

Where this points

The browser-side D1-FORK spawn of a second user Worker (parent-memory copy

  • per-Worker module instantiation) for this larger glibc module. The failure is a silent non-completion — no error event, no dmesg — which makes it look like a promise in the spawn path that never resolves for this image size/shape (.tdata=116@8426297, data_segments=32, size=15228924).

Impact

The Yocto glibc base image (BRINTOS-27) boots but cannot run a single command in the browser — blocks browser verification of glibc Yocto images while the same bytes pass on the host harness.

No fix authored — needs someone with the D1-FORK internals; happy to run any diagnostic build on the deployed machine.

jdbrinton jdbrinton commented

Resolved — root cause was hardwarejs#3's pool-lane starvation; the fix is already live. Closing on real-browser evidence + a new committed end-to-end gate for this image class.

Current-HEAD status

Production brintos.io already serves current joel-dev HEAD (served worker.mjs/kernelWorker.mjs byte-identical to a fresh make web && make web-sync of this tree, sha256 f9177cc9…/1fe5349c…; live boot trace pins kernelWorker-browser.mjs?v=49505b1f412ceb95 == the local build id; the app chunk carries the #3/#65/#67 markers). Driving the real site in headless Chrome (crossOriginIsolated === true, anonymous session — the exact repro from this ticket):

machine (echo subshell-ok) (echo SUB$((6*7))OK) cat /proc/version uname prompt alive
dmitry/core-image-minimal-glibc PASS 519 ms PASS 510 ms PASS 1.0 s PASS 1.0 s PASS
alex/core-image-minimal-musl (control) PASS 503 ms PASS 508 ms PASS 1.0 s PASS 1.0 s PASS
brintos/linux-6.12-glibc-bash-coreutils (control) PASS 504 ms PASS 514 ms PASS 1.0 s completes instantly with RC=1 (uname: cannot get system name: Function not implemented — that machine's older kernel g9ceff07e lacks the syscall; not a hang) PASS

hw#4 no longer reproduces anywhere in the matrix.

Root cause — and why "browser-only, size-only" was a mirage

It is hardwarejs#3 (pool-lane event-loop starvation vs fork placement, fixed at db40e58, deployed 2026-07-10 — after this ticket was filed). The hidden variable was pool topology, not the engine and not the 15.2 MB module:

  • glibc busybox init polls waitpid(-1, WNOHANG) in a loop that never parks, starving its pool Worker's onmessage forever.
  • Your machine's spec is cpu=2 ⇒ 2 pool lanes. The pre-#3 avoid-parent placement left init's starved lane as the only candidate for the shell's fork child; its add-task was never processed; the fork parent parked forever. Deterministic, silent, console clean — exactly what you reported.
  • run-machine.mjs defaults maxCpuId=7. "Node doesn't reproduce" was really "8 lanes don't reproduce".

Proof, with this machine's exact rootfs + /boot/vmlinux.wasm (mounted read-only via brintos-fs) under Node — real /init → inittab respawn shell, the live LaunchMachine cmdline (prestrike.rootfs_ro init=/init, no wasm_user_pin), num_vcpu=3:

hardwarejs maxCpuId=2 (live browser shape) maxCpuId=7 (run-machine shape)
a313136 (= db40e58^, pre-#3) HANG — hw#4 reproduced verbatim (prompt reached, (echo subshell-ok) echoed, output never arrives) GREEN
joel-dev HEAD GREEN, with the #3 machinery visibly doing the work at exactly the subshell fork: hwjs#3: add-task tid=0 routeKey=3 claim timed out on starved pool lane cpu=1 (resident spinning without a yield); re-placing on cpu=2 GREEN

The musl control never hung because musl busybox init parks (releasing its lane's event loop); the glibc-bash machines never hung because bash-as-init parks. The size/shape numbers (.tdata, data_segments, 15.2 MB) were coincidental — the Yocto glibc-busybox image was simply the only machine whose pid 1 spins. Your other rule-outs hold: boot-time spawns never route through a starved lane, and the anonymous-session 403 write refusals completed with attributed EACCES throughout my runs (the #3 errno half).

Gate (committed, non-vacuous)

hardwarejs/test/hw4-glibc-busybox-init-fork.test.ts (hardwarejs a960eb8 + e12d13f; superproject gitlinks 98957b6, 0937614; branch joel-dev): boots this machine's real rootfs + kernel at the live topology (maxCpuId=2, num_vcpu=3, live cmdline), requires the inittab respawn shell to complete the ticket's literal (echo subshell-ok) and a fork+exec, and asserts the starved-lane re-placement dmesg witness so the gate cannot go vacuous if placement ever stops brushing init's lane. Prereq is a copy of this machine's rootfs (loud throw with a mount+rsync one-liner if absent — never a silent skip).

RED arm proven flipping (2026-07-10): the identical file against pre-#3 hardwarejs a313136 fails exactly like this ticket (47 s timeout, subshell output never arrives); GREEN on HEAD in ~3 s.

Regression sweep on HEAD: a4c-addtask-claim, hw65-early-signal-window, virtio-fs-errno{,-async}, d1-fork-accept-hwjs, d1-fork-exec-accept-hwjs, d1-fork-pipe-accept, hwjs-boot-probe, glibc-dualcon-gate (cpu=1/2 + both RED arms) — all GREEN. The musl dualcon-gate is unstable at HEAD for a pre-existing, unrelated kernel-side reason (a K3-era arch_send_call_function_single_ipi BUG() stub now reachable, likely via the #66 stop_machine work) — filed with full stacks and triage as brintos/linux-in-the-browser#74.

SHAs

  • Fix (already landed + deployed before this investigation): hardwarejs db40e58 (hardwarejs#3), with c33d1a3 (#65) and 56dde7a (#67) also live.
  • This ticket's additions (gate + evidence): hardwarejs a960eb8, e12d13f; superproject 98957b6, 0937614.
  • No product-code change was needed — the deployed runtime already contained the fix; this ticket adds end-to-end coverage for the glibc-busybox image class.

Closing as completed on the live-browser evidence above. If you can still reproduce a hang on the current site, please reopen with the boot trace's ?v= build id.

jdbrinton closed this as completed