Summary

Running a standalone (non-busybox) binary in the browser hangs at exec(): no output, no returning prompt. The binary's content blob is present in the store, and the identical rootfs+kernel runs it cleanly under the fra2 hardwarejs run-machine.mjs harness. This blocks running any distinct tool in the browser.

Repro

Machine: dmitry/bzip2-glibc (Yocto core-image-minimal-brintos, glibc/wasm32-hwjs; busybox + bzip2 + brintos-selftests).

1. Open https://brintos.io/dmitry/bzip2-glibc, Power on — boots to the shell.
2. Run: bzip2 --help (or: echo hi > /tmp/x; bzip2 -c /tmp/x > /tmp/xb)

Observed: the command hangs — no output, no next prompt. The browser devtools console shows Failed to load resource: the server responded with a status of 404. Every bzip2 invocation hangs the same way.

Isolation

• busybox applets (cat, wc, ls, sha256sum, find, xargs) all run fine — those are forks of the already-running busybox module (same image).
• bzip2 is the ONLY non-busybox binary on the machine: a distinct 2.6 MB wasm module (/usr/bin/bzip2 -> bzip2.bzip2), fork+exec'd as a NEW user Worker. That exec is what hangs.
• NOT a missing-blob/deploy problem: the fs-tree entry for /usr/bin/bzip2.bzip2 has blobOid a2d2a1135d44c03309b0c66e79d7b10fd7169ba796a1c047876039d51377148f, and GET on that blob's presigned R2 URL returns HTTP 200, 2 649 147 bytes, magic 00 61 73 6d (\0asm). The binary content is fully uploaded and readable.
• Same bytes PASS on fra2 hardwarejs run-machine.mjs (reads files from a local dir, not the cloud FUSE bridge): bzip2 -c in > out; bzip2 -t OK; bunzip2 -c / bzcat recover; cmp byte-identical.

Where this points

The browser-side fork+exec of a distinct wasm module over the cloud FUSE bridge: a 404 on some per-exec resource fetch that never resolves, so the second Worker's module load never completes and the spawn hangs. Parallels the closed hardwarejs#4 (second-Worker fork hang), but for exec of a NEW module rather than a fork of the same one.

Impact

Blocks browser verification of any standalone tool (bzip2 here; the target dpkg chain next). No fix authored — happy to run any diagnostic build on the deployed machine; the exact 404 request URL from a devtools Network capture would pin the missing fetch.

Correction + follow-up after the hardwarejs#4 resolution (root cause hardwarejs#3 pool-lane starvation, fixed at db40e58, deployed 2026-07-10).

The console 404 in the original report is a red herring — disregard it. It is GET /dmitry/manifest.webmanifest -> 404, the PWA web-app manifest. The site's <link rel="manifest" href="manifest.webmanifest"> is relative, so on any /<owner>/... page the browser resolves it to /<owner>/manifest.webmanifest, which 404s (the real manifest is at /manifest.webmanifest, HTTP 200). It 404s on every owner's pages (/alex/manifest.webmanifest too), is unrelated to bzip2, and is not the hang's cause. (Minor separate site bug: make that <link rel="manifest"> href root-absolute.)

So the exec hang is a spawn/scheduling stall, not a missing fetch — and it looks like a #3/#4 sibling for the distinct-module exec path. Tested 2026-07-12 (post-db40e58), anonymous headless Chrome on dmitry/bzip2-glibc:

• Boot + busybox fork/exec are healthy, matching the #4 re-test: busybox applets (cat, wc, ls, sha256sum, find, xargs) and (echo x) subshells all run.
• Only the exec of a DISTINCT (non-busybox) module hangs: bzip2 --help and bzip2 -c in > out hang with no output and no returning prompt. bzip2 is the only non-busybox binary on the image (a 2.6 MB wasm module, fork+exec'd as a new Worker). Its content blob is present (R2 GET -> HTTP 200), and the identical rootfs+kernel runs bzip2 cleanly under run-machine.mjs on fra2.

#4's re-test could not exercise this: core-image-minimal-glibc has no non-busybox binary, so its cat/uname are busybox applets (same module), never a second-module load. The #3 root cause (glibc-busybox pid 1 spinning in waitpid(WNOHANG) starves its pool lane; with cpu=2 the fork child is stranded) plausibly still bites the exec-a-NEW-module path: that spawn instantiates an additional Worker/module and may re-strand on the same starved lane — fixed by db40e58 for fork-of-same-module, but perhaps not for exec-of-distinct-module. Happy to run any diagnostic build on the deployed machine.

Root-caused, fixed, gated, and deployed. bzip2 (and any distinct non-init binary) now execs, runs, and returns to the prompt in a real browser on the production build.

Root cause — not a fetch, not scheduling: an exec-in-place LinkError with no kernel-side exit

Reproduced in headless Chrome on the live dmitry/bzip2-glibc with full console + network + __hwjsLog capture. The per-exec FUSE fetch was healthy — the kernel staged all 2 649 147 bytes of /usr/bin/bzip2:

[proc32] wasm32_user_exec_load: staged /usr/bin/bzip2 (2649147 bytes) at kmem offset 0x1800000 for in-place exec (pid=32)
hardwarejs: user[32] ERROR: WebAssembly.instantiate(): Import #0 "env" "user_memory":
            memory import has 201 pages which is smaller than the declared initial of 256
hardwarejs: user[32] exited with code -1        ← and the shell hangs forever

The exec-in-place path re-instantiates the NEW image against the process's existing env.user_memory, which was sized for the OLD image: the A-5.a small-initial (2 MiB) + grow-to-fit runs at spawn time only (kernelSpawnHandler.initUserMemoryPartition). The busybox shell's Memory was 201 pages by exec time; bzip2 declares a 256-page (16 MiB) initial → WebAssembly.instantiate threw a LinkError. That rejection killed the Worker host-side with no kernel-side task exit, so the parent's wait4 never completed — the silent hang, on every invocation.

• Why only distinct binaries: fork is a delta=0 copy of the same module (memory sized to the parent's — always fits); busybox→busybox execs re-instantiate an image with the same declared minimum. Only a distinct image with a larger declared initial trips it. bzip2 was the machine's first.
• Why node "didn't reproduce": run-machine.mjs/the older gates allocate 64–128 MiB initial user SABs, which already satisfy any image's minimum. At the live LaunchMachine sizing (userInitBytes: 2 MiB) node reproduces the hang byte-for-byte (the new gate proves it).
• The 404s: the console 404 was indeed the PWA-manifest red herring you identified (now also fixed — the <link rel="manifest"> is root-absolute, brintos-web 99fe3c2); the blob store and /fs/download keying were never at fault.

Fix (hardwarejs 42d3ff3, joel-dev)

1. ensureUserMemoryFitsImage (src/worker.mjs, + parseImportedMemoryMinPages in src/dynLoader.mjs): before the exec's point of no return — on both the HWJS-B-2 inline (glibc) and park-serviced (musl) legs — parse the staged image's declared env.user_memory minimum and grow the Memory to fit. An un-satisfiable minimum (past userMaxBytes) fails the execve with an attributed -ENOMEM: the shell prints Cannot allocate memory and lives.
2. imageLoop belt (src/worker.mjs): any residual user-image COMPILE/INSTANTIATE failure is now loud and reaped — a re-exec failure posts the attributed diagnostic and exits 127 through the normal tail (task_exit_cleanup + zombie + wait4 wake); a spawn failure keeps the legacy kind:'error' contract but drives the kernel child-exit first. Never a silent hang.
3. Partition re-init on exec (src/kernelSpawnHandler.ts exec-image-installed): the F30 brk/mmap partition is re-derived from the NEW image's __heap_base (monotonic bump), so a larger exec'd image's mmaps can't be served from inside its own .data/.bss/arena span.

RED → GREEN

Committed gate test/hw6-distinct-exec-grow.test.ts: boots the real 15 MB glibc-busybox rootfs at the live sizing (2 MiB initial, cpu=2) and fork+execs a byte-patched busybox whose declared minimum is raised to 512 pages (distinct module, larger initial — the exact #6 shape).

• RED proven flipping on pre-fix eb6f186: memory import has 201 pages which is smaller than the declared initial of 512, no output, no reap — the ticket's hang, verbatim, under node.
• GREEN on 42d3ff3: exec completes (HW6EXEC42END), prompt returns, the grow dmesg witness is asserted (its absence fails the gate — non-vacuous), and the never-silent arm (an 8192-page minimum past the 256 MiB ceiling) refuses -ENOMEM with a returning prompt.
• Full hardwarejs suite: no regressions (the only failures are pre-existing at eb6f186, identical pre/post).

Live (authoritative): deployed to production (make web && make web-sync + pnpm sst deploy --stage production), then headless Chrome on https://brintos.io/dmitry/bzip2-glibc:

• bzip2 --help prints the full help text; prompt returns.
• echo hw6-roundtrip-payload > /tmp/x; bzip2 -c /tmp/x > /tmp/xb; bunzip2 -c /tmp/xb → payload recovered byte-identical; prompt returns.
• Grow witness live: hwjs exec pid=32 exec-load (inline): grew user_memory 201 -> 256 pages for the exec image's declared minimum (hardwarejs#6).
• Boot, busybox applets, and fork all unchanged-healthy; the manifest 404 is gone.

SHAs: hardwarejs 42d3ff3 · superproject gitlink 04eb7aa · brintos-web 99fe3c2 (all joel-dev). Closing.