brintos

brintos / hardwarejs public Read only

0
0

Served host worker: spawned-child pty I/O hangs in the browser (works on fra2 @ 63a34b9) #2

Closed vlad opened this issue · 4 comments
vlad vlad commented

In the browser on brintos.io/emcraft/linux-6.12-musl-bash-coreutils, a spawned child's pty I/O does not flow: expect's first spawn hangs indefinitely — not even Expect's own timeout fires. The Tcl self-test passes in the same browser session (ALL-TCL-OK), so kernel, musl, Tcl_Init, encodings, and access(2) all work — the problem is specifically the pty master↔slave I/O for a spawned child.

Reproduction (browser)

tclsh /root/selftest.tcl                → ALL-TCL-OK
expect /root/selftest_expect.exp        → prints "E-E1ver OK", then hangs at the first spawn

Minimal, with an explicit timeout that should fire but doesn't:

expect -c 'set timeout 5; spawn /bin/cat; send "HI\r"; expect { "HI" {puts OK} timeout {puts TIMEOUT} }'
# browser: hangs, no output (not even TIMEOUT).   fra2: prints OK.

Evidence it is host-side, not the image

The identical rootfs + kernel bytes pass an Expect self-test 5/5 fresh boots on fra2 via run-machine.mjs against hardwarejs 63a34b9. Our client-side Expect fixes (dup2 for the pty-slave stdout/stderr wiring, openpty, ptsname) are compiled into the deployed /bin/expect — the same binary that ran Tcl fine in the browser. The only variable between the passing fra2 run and the hanging browser run is the host worker / its runtime.

The hang with no timeout (rather than an EBADF or a TIMEOUT) points at the pty read/write/poll for the spawned child never being serviced — the master↔slave coupling between the expect process and its child.

Not the same as the two already filed

  • hardwarejs#1 (fcntl(F_DUPFD) returns ENOSYS on non-console kernel fds) — we worked that around client-side with dup2, so it is not the browser blocker.
  • linux-in-the-browser#47 (a spawned child's coroutine drive traps after ~2 blocking-read exchanges) — the browser hangs at the first spawn, before that limit applies.

This is the actual blocker for Expect browser-green.

Two hypotheses to disambiguate

You have the served build + the browser runtime, so you can tell which:

  • (a) the served worker lags fra2's 63a34b9 for the spawn/pts read/write/poll routing (the K12-A / D1-FD / blocking-tty-read paths), or
  • (b) a browser-specific cross-Worker pts coupling issue (real Web Workers + SharedArrayBuffer) that does not manifest under fra2's Node run-machine runtime even at the same 63a34b9.

Happy to run any probe you want on either side (fra2 or a deployed browser machine).

jdbrinton jdbrinton commented

(agent for joel)

Root-caused and fixed. Answering your (a)-vs-(b) first: neither — it is not a served-bundle lag and not a browser-specific Worker coupling. The hang reproduces headlessly under Node with your exact image bytes, at hardwarejs HEAD and at your "good" 63a34b9-era tree. It's a kernel (arch/wasm32) bug: a user task's poll/select timeout could never fire on a process Worker. Your fra2 5/5 passed because the matching arm is data-driven (pty bytes wake the poller — no timer needed); anything that ever had to fall to the timeout arm parked forever, and in the browser your selftest's first spawn interaction evidently took that path. Your instinct that "hang with no timeout points at the pty poll never being serviced" was exactly right — the fds were serviced, the deadline never was.

Root cause (three stacked, all in arch/wasm32)

poll/select/ppoll/pselect6 are kernel-routed and park via poll_schedule_timeout → schedule_hrtimeout_range, which arms a per-CPU hrtimer. On this arch:

  1. The static percpu image was never copied into the per-CPU slots. percpu.c's setup_per_cpu_areas claimed pcpu_setup_first_chunk does the memcpy — it doesn't (the copy lives in the upstream first-chunk helpers this arch bypasses). Every statically-initialized percpu var read as zeros through per_cpu_ptr: measured, hrtimer_bases' clock_base[].get_time was NULL on every CPU (this is also the long-standing init.c "hrtimer_start null function" landmine, now closed).
  2. Pool CPUs (1..N) never ran hrtimers_prepare_cpu — there is no cpuhp bring-up on this arch (__cpu_up = -ENODEV), so a process Worker's hrtimer base had NULL cpu_base back-pointers; the sleeper enqueue wrote through address 0 (the intermittent whole-machine wedge you saw is this low-memory corruption).
  3. Nothing ever ran a pool CPU's hrtimer queues — the host tick (wasm-tickd) services CPU 0 only, and hrtimers are strictly per-CPU.

(nanosleep always worked because it's host-side in kernelSpawnHandler.ts — which is why sleep 2 passed while expect's set timeout, Tcl after, and bash read -t all hung.)

Fix

brintos/linux@48715b28efec (joel-dev): copy the static percpu image to every possible CPU's slot; prepare all pool-CPU hrtimer bases at wasm_tick_init; and self-service the owning CPU's hrtimer queues at every process-Worker blocking-wait resume (expiry lateness bounded by the ~15ms re-drive cadence — same service level as host-injected IRQs). RED seam: boot with wasm_userhrt_red to restore the old behavior without a rebuild.

Gate + verification (all headless, your image bytes)

  • New gate distros/linux-6.12-musl-bash-coreutils/tools/run-poll-timeout-gate.mjs (superproject d036f64): GREEN 3/3 (read -t 2 fires on a quiet pipe and on the console pts, console survives), RED 2/2 (under wasm_userhrt_red the timeout genuinely never fires while the machine stays alive — non-vacuous flip).
  • A staged C probe: poll(NULL,0,2000), poll on a quiet pipe fd, pure-timeout select, select on a quiet fd — all four now return 0 at ~2000ms (all four hung before).
  • Your minimal repro: spawn /bin/cat round-trip prints OK; a deliberately non-matching pattern prints TIMEOUT in ~the configured 5s; the console stays alive after both.
  • Your selftest_expect.exp: ALL-EXPECT-OK, E-SUMMARY pass=4/4.
  • Regressions green: run-tty-gate, run-a2-jobcontrol, ls-freeze gate, hardwarejs timer-tick-gate.

No hardwarejs change was needed (the served worker is fine). Ships to the browser with the next kernel deploy/nightly. Leaving this open until the fixed vmlinux.wasm is actually deployed to emcraft/linux-6.12-musl-bash-coreutils and you can confirm in-browser; the fix itself is verified headlessly including the timeout arm and console survival.

jdbrinton jdbrinton commented

(agent for joel — correction + completing the ship note)

One correction to the comment above: "No hardwarejs change was needed" is wrong. The kernel hrtimer fix (brintos/linux@48715b28) is the piece that makes deadlines fire, but two real host-worker bugs were also root-caused from your exact image bytes and are fixed in hardwarejs@5d3f9fb (joel-dev):

  1. vfork/posix_spawn-shaped clone mis-dispatched as a thread. Your /bin/expect's spawn child issues a raw clone(CLONE_VM|CLONE_VFORK|SIGCHLD, stack) (Tcl's helper riding musl posix_spawn). The worker's SYS_clone arm keyed thread-mode on CLONE_VM alone, so it returned a fabricated tid to the caller and spawned a rogue shared-memory Worker carrying the caller's 14MB expect image. Seconds later (when its compile landed — after the child had already exec'd /bin/cat into that same memory) the rogue instantiation wrote expect's data segments over cat's live coroutine arena: measured headlessly as expect rodata strings appearing inside the 420KB cat image's memory, followed by a drive trap in cat and (interacting with the hrtimer bug) the machine-wide wedge. Now refused loudly with -ENOSYS unless CLONE_THREAD (musl's wrappers surface a clean errno; your spawn already tolerates the helper failing — wait4 returned ECHILD even on fra2). fork() rides the _Fork park protocol and is untouched.
  2. Spawned-child pty output leaked to the host console. Your spawn child does close(0/1/2); open("/dev/pts/N") → fd 0; dup2(0,1); dup2(0,2); exec. A kernel-routed open/F_DUPFD landing on a freed low fd was never re-added to the worker's kernel-routing set, so the dup2 bookkeeping propagated "not kernel-routed" onto fds 1/2 and the exec'd child's writes went to the host console instead of the pts slave — expect only ever saw the line-discipline echo of its own send (which is why your minimal repro could "work" on fra2: the echo satisfied the "HI" match while cat's actual output silently leaked).

Both are pinned by a new gate with genuinely flipping RED arms (distros/linux-6.12-musl-bash-coreutils/tools/run-hw2-expect-shape-gate.mjs, superproject c7ee012): a spawnclone probe asserting the vfork-shape clone refuses (HWJS_RED_CLONE_LAX=1 flips it — fabricated tid again), and a ptyspawn probe doing your exact spawn-child fd dance with echo off, so only the child's bytes can satisfy the round-trip (HWJS_RED_LOWFD_OPEN=1 flips it — the round-trip starves at 0 bytes), plus the quiet-master poll-timeout leg.

Ship note (corrected): the browser needs BOTH the kernel vmlinux.wasm redeploy AND a served-worker update (hwjs:sync + web deploy of hardwarejs@5d3f9fb) — both ride the next nightly. Verified headlessly post-fix with your image bytes: minimal repro OK 5/5, timeout arm TIMEOUT at ~5s with the console surviving, after 2000 stable 6/6; regression gates green (tty, jobcontrol, ls-freeze, pty-vfork, fork/exec + dualcon + timer-tick vitest). Leaving open until you can confirm on the deployed machine.

jdbrinton jdbrinton commented

(Agent report — Cursor agent working joel's bug queue.)

All three fixes are live on the platform as of tonight's deploy, and the timer half is verified in-browser.

Live witness just now on Chrome at brintos/linux-6.12-musl-bash-coreutils/launch (serial console):

a2$ time read -t 2 x; echo RC-$?
real    0m2.030s
RC-142

A user-task timer firing at exactly its deadline — pre-fix that parked forever and wedged the machine. The worker-bundle halves (the vfork/posix_spawn clone-shape refusal and the low-fd pty routing) shipped in the same deploy (bundle id in the page source; both are RED/GREEN-gated in run-hw2-expect-shape-gate.mjs).

One note for your emcraft/linux-6.12-musl-bash-coreutils machine, vlad: the timer fix is kernel-side (brintos/linux@48715b28efec, now on wasm32 head), and your fork boots its own /boot/vmlinux.wasm — your last rebuild was at 12fce771, which predates it. Pull the new vmlinux.wasm from brintos/linux-6.12-musl-bash-coreutils into your fork (or rebuild at ≥48715b28), hard-reload, and your minimal repro should print OK — and the timeout arm TIMEOUT in 5s instead of hanging. The worker halves come to you automatically from the site.

Leaving this open for your in-browser confirmation per the earlier comment — please close it (or report back) once your fork's kernel is updated.

jdbrinton jdbrinton commented

validated by vlad

jdbrinton closed this as completed