.. SPDX-License-Identifier: GPL-2.0 ================================================================================ arch/wasm32 scheduler model (K4 spec) ================================================================================ :Status: K4 (cooperative kthread scheduling, ``switch_to`` via Asyncify, rest_init → kthreadd → kernel_init to "No working init found"). Stable through K6; revisions require a new section + §15 row. This document is the source of truth for *how the wasm32 port models kernel-thread context switching, cooperative scheduling, idle behaviour, and the kernel-side half of the Two Regimes spec in ``docs/ARCHITECTURE.md`` §17*. It plays the same role for the scheduler that ``mm-model.rst`` plays for memory and ``boot-protocol.rst`` plays for boot handoff: spec-first, code-second. Treat any divergence between this document and ``arch/wasm32/kernel/switch_to.c`` / ``arch/wasm32/kernel/asyncify.c`` / ``arch/wasm32/include/asm/switch_to.h`` / ``arch/wasm32/include/asm/thread_info.h`` as a bug in the code. This doc explicitly does NOT re-derive the Two Regimes framing — read §17 of ``docs/ARCHITECTURE.md`` first; this doc is the K4 implementation contract built on top of it. -------------------------------------------------------------------------------- 0. K4-phase scope at a glance -------------------------------------------------------------------------------- (a) **New cross-module ABI introduced by K4.** Three surfaces. - **kernel-self ↔ Asyncify-transformed-self.** Not kernel ↔ HardwareJS or kernel ↔ userspace; the kernel module's own C code, post-Binaryen-``asyncify``-pass, gains a new contract with itself. Specifically: any function on the Asyncify *whitelist* (§3.1 below) is permitted to suspend via a cooperative yield point and resume later with its C-stack restored. Any function NOT on the whitelist that nonetheless reaches a yield point at runtime triggers ``asyncify-asserts`` (``--pass-arg=asyncify-asserts`` to wasm-opt), surfacing the whitelist gap as a hard runtime trap rather than a silently corrupt stack. The whitelist itself is the ABI artifact, generated by ``scripts/asyncify-whitelist.py`` (K4 introduces this tool, sibling to ``scripts/wasm-add-memory.py`` and the upcoming K5/K6 percpu post-link pass). - **kernel ↔ HardwareJS (scheduler wakeup word).** A new fixed-offset, fixed-name word in ``env.kernel_memory``, documented as ``__sched_wakeup_word`` and exported as a wasm global of i32 type pointing at the offset. Any party that makes a kthread runnable — kernel C code's ``try_to_wake_up`` path, a HardwareJS-side timer expiry, a user Worker's syscall response — writes a nonzero value to the word and calls ``Atomics.notify``. The vmlinux Worker, when its cooperative runqueue is empty, parks via ``Atomics.wait`` on the same word. §6 below specs the word's offset, semantics, and the host-side helper surface. - **kernel ↔ wasm runtime (Asyncify state convention).** The Asyncify pass exposes two wasm globals: ``asyncify_state`` (i32, 0/1/2 = none/unwind/rewind) and ``asyncify_data`` (i32 pointing at the per-task save buffer). These ARE the contract: the kernel's C code in ``arch/wasm32/kernel/switch_to.c`` manipulates these globals directly via wasm-builtin extern declarations, and the Asyncify-transformed binary observes them at function-entry epilogue and prologue. §3.2 below names the wasm globals and the C-level extern declarations that bind them. (b) **Architectural decisions K4 forces.** Q1–Q13 in §2 below. None are reversible without a coupled HardwareJS ↔ kernel bump. (c) **§15 debt this phase predictably opens.** Six rows, all noted inline below and added to the §15 table at K4 close: 1. **Asyncify whitelist is a maintenance contract.** Adding a yielding kernel path that doesn't update ``arch/wasm32/asyncify-whitelist.txt`` (generated by ``scripts/asyncify-whitelist.py``) produces a runtime ``asyncify-asserts`` trap. The K5/K6 commits that add new yielding paths (page faulting, signal handling, blocking I/O) MUST regenerate the whitelist. Owed: each subsequent K-phase. 2. **``current->mm != NULL`` path stays ``BUG()``.** K4 implements Regime 2 only. Regime 1 (user-task park via ``Atomics.wait`` on per-task futex word) is a K5 deliverable. ``wasm_sched_class.enqueue_task`` in ``arch/wasm32/kernel/sched.c`` BUG_ONs if it sees a non-NULL ``->mm`` (see ``docs/ARCHITECTURE.md §17.7`` for the K4 class scope and HISTORICAL NOTE for the original escape-hatch design that was retired). Owed: K5. 3. **Voluntary preemption only.** ``preempt_count`` is compiled in but not driving dispatch (§17.4 already established). K4 makes this real: the cooperative runqueue NEVER preempts a running kthread mid-function; yield happens only at explicit ``schedule()`` / ``cond_resched()`` / ``wait_event_*`` / ``msleep`` / ``schedule_timeout`` call sites. Owed: never (a permanent property of the model). 4. **Asyncify binary-size delta.** Post-Asyncify ``vmlinux.wasm`` is empirically X% larger than pre-Asyncify (X measured at K4 close; recorded in commit body). Whitelisted-only transformation keeps the delta bounded, but it is not zero. Owed: revisit at K7 if size becomes a deploy-cost issue, by either tightening the whitelist or migrating to JSPI per Q7. 5. **Kthread signal handling deferred.** ``signal_pending()`` returns 0 for kthreads at K4 (Q5 below). KILL, freezer, suspend-related signals are out of K4 scope. Owed: K6+ when the freezer / suspend / kthread-stop machinery becomes interesting (it never may, for a browser target). 6. **``wasm_sched_class`` Regime-1 extension.** K4 lands a real ``wasm_sched_class`` for Regime 2 kthreads (see §5 below and ``docs/ARCHITECTURE.md §17.7``). At K4 the class BUG_ONs in ``enqueue_task`` if it sees a task with ``->mm != NULL`` (the Regime-1 marker). K5 extends the same class to handle Regime 1 by recording the per-task futex word on enqueue and firing the notify on dequeue; the class structure stays single — there is NOT a separate ``sched_browser_class`` (the early K3/K4 design name that was retired alongside §17.7's escape-hatch design — see ``docs/ARCHITECTURE.md §17.7`` HISTORICAL NOTE for the original assumption and why it was refuted). Owed: K5. -------------------------------------------------------------------------------- 1. Scope inside §17's framing -------------------------------------------------------------------------------- §17 of ``docs/ARCHITECTURE.md`` defines the Two Regimes. K4 implements exactly half of the spec: ============== ========================================================= Regime 2 **K4 implements in full.** All kthreads inside the (``->mm == 0``) vmlinux Worker. Cooperative FIFO runqueue inside ``wasm_sched_class``. Asyncify-backed ``switch_to``. Idle parks on the master scheduler wakeup word. kthread_create / kthread_run / kthread_stop all work. wait_event / wake_up_process / complete / wait_for_completion all work for kthread-only callers. Regime 1 **K4 leaves as ``BUG()``.** No user-process Workers (``->mm != 0``) exist yet (binfmt_wasm is K5; the userspace toolchain isn't producing user binaries yet). ``wasm_sched_class.enqueue_task`` BUG_ONs if it sees a task with non-NULL ``->mm``. K5 extends the same class to record the per-task ``Atomics.wait`` futex word on enqueue and fire the notify on dequeue. ============== ========================================================= Specifically out of scope for K4: * User-process spawn (no ``do_fork`` walks the ``->mm != NULL`` branch because no Worker exists to run a user process yet). * binfmt_wasm dispatching to a user wasm binary. * The ``/init`` binary itself; the userspace toolchain hasn't been exercised against a real userland. * Regime 1 dispatch policy (the per-task ``Atomics.wait`` parking is K5's ``wasm_sched_class`` extension; see ``docs/ARCHITECTURE.md §17.2`` and the §17.7 HISTORICAL NOTE for the original "deferred to v1.0 via ``sched_browser_class``" design that was retired mid-K4). What this means in the K4 acceptance path: ``rest_init`` runs, ``kernel_init`` is spawned as a kthread, kernel_init attempts to ``run_init_process("/init")``, fails (no rootfs, no binfmt_wasm), falls through to ``panic("No working init found.")``. That panic *is* the K5 boundary — exactly the §17 / K5 framing. -------------------------------------------------------------------------------- 2. Architectural decisions Q1–Q13 (the K4 opening thirteen) -------------------------------------------------------------------------------- Q1. Asyncify scope (whitelist) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: whitelisted Asyncify, generated by ``scripts/asyncify-whitelist.py`` from the reachability closure of the cooperative-yield function set {``schedule``, ``cond_resched``, ``wait_event_interruptible``, ``wait_event_timeout``, ``msleep``, ``schedule_timeout``, ``wait_for_completion``, ``wait_for_completion_interruptible``, ``__wasm32_switch_to``}. Pass: ``wasm-opt --asyncify --pass-arg=asyncify-only-list@arch/wasm32/asyncify-whitelist.txt --pass-arg=asyncify-asserts``.** Considered: * **Full-program Asyncify.** The default. Easy to enable, hard to diagnose. Increases binary size 2-3× (Binaryen issues #2755, #5040). Slows EVERY function (each non-trivial wasm function gains an unwind/rewind switch at the prologue and a state check at every call). Acceptable for size-insensitive applications; unacceptable for a kernel that boots into a browser tab. * **Whitelist + asyncify-asserts (with named topmost-strip exceptions).** Binaryen's ``asyncify-only-list`` pass-arg restricts the transformation to a named subset. Any function NOT on the list (and not in the transitive closure of callers) is left as-is — same size, same speed. The companion ``asyncify-asserts`` arg makes the runtime trap if a non-listed function reaches a yield point at runtime, which is the diagnostic that catches whitelist drift. This is the standard pattern in production Asyncify deployments (Emscripten's documentation explicitly recommends it for performance-critical code paths). **Topmost-runtime caveat (empirically discovered at K4 opening).** Binaryen 125's ``asyncify-asserts`` injects the state-stability check around every call site in non-instrumented functions, including functions that ARE the asyncify state- machine driver (those that call ``asyncify_start_unwind`` / ``asyncify_stop_rewind`` directly — ``__wasm32_switch_to`` and any future helpers). For those functions, the legitimate state change IS the bug the assertion fires on, and the kernel traps on every context switch. Binaryen acknowledges this in source at ``src/passes/Asyncify.cpp:1286-1291`` of version_125 with a FIXME ("the top-most runtime is actually a place that needs neither instrumentation *nor* assertions"). Our resolution is a post-pass strip tool, ``scripts/asyncify-strip-topmost-asserts.py``, that reads a named list at ``arch/wasm32/kernel/asyncify-topmost.txt`` and surgically removes the 9-byte assert pattern from those functions' bodies — preserving asserts everywhere else. The exception list is the maintenance contract: adding a topmost-runtime helper without listing it is a build-time- detectable bug (the strip tool fails the build if any listed name is missing from the binary's name section); calling ``asyncify_start_unwind`` from outside the list is a runtime trap. The probe at ``linux-wasm/tools/k4-asyncify-asserts-probe/`` validates this surgery against the pinned toolchain and detects Binaryen drift before the kernel does. See §15 row "asyncify-asserts enabled with named topmost-strip exceptions" and §18 entry on topmost-runtime as the intended exception. **The host is responsible for ``asyncify_stop_rewind`` (empirically discovered at K4 mid-phase, R4b).** Binaryen 125 does NOT auto-inject ``asyncify_stop_rewind`` at the bottom of the rewind chain. Verified by direct disassembly inspection of the post-pass ``vmlinux.wat``: zero calls to the synthesized ``asyncify_stop_rewind`` function exist anywhere in the binary unless emitted by hand from C code. The Asyncify pass moves the state machine UNWIND→REWIND→ via the rewind chain's prologue/epilogue mechanics (each function's prologue decrements ``cur`` to load its frame), but the REWIND→NORMAL transition is NOT done automatically; it must be performed by the topmost-runtime function (in our self-call design, ``__wasm32_switch_to``) on rewind re-entry. The contract: the topmost-runtime function maintains a per-task gate (``thread.unwound``); on the first call (state is NORMAL, unwound is false) it sets the flag and calls ``asyncify_start_unwind``; the host (the rest of ``__wasm32_yield``'s loop) eventually calls ``asyncify_start_rewind`` on a saved buffer; the rewind chain bottoms out by re-executing the call to the topmost helper, which observes ``unwound == true``, calls ``asyncify_stop_rewind``, clears the flag, and returns. The state is now NORMAL and execution resumes in the caller past the suspended call site. The ``asm/asyncify.h`` header in this tree previously contained a comment claiming auto-injection — that comment was incorrect for Binaryen 125 and has been deleted along with the protocol fix. Future Binaryen versions may change this behavior; the topmost-strip-asserts probe and the asyncify-removelist propagation probe (both at ``linux-wasm/tools/k4-binaryen-asyncify-probe/``) detect drift in either direction before the kernel does. * **Manual``__attribute__((noreturn))``-marker pattern.** Mark yielding functions with a custom attribute; have the build system extract the marker list as the whitelist. Cleaner long- term but requires per-function code changes across upstream Linux source. Out of bounds (§16 PR rejection criterion). * **JSPI (JS Promise Integration).** Q7 below. **Rationale.** Whitelisted Asyncify is the smallest hammer that fits. The whitelist size is ~1500 functions at K4 (closure of the nine root functions through cond_resched / schedule callers); the whitelist regeneration is mechanical (DWARF-walking the K4 ``vmlinux.wasm`` via ``llvm-objdump --syms``, transitively including callers reachable from the yield-root set). The K4 binary-size delta is bounded to ~15% per Binaryen's documented "selective asyncify" overhead on similar-size kernels; the actual number is measured at K4 close and recorded in the commit body alongside the atomic-opcode count. **§15 debt row at K4 close.** "Asyncify whitelist is a per-K-phase maintenance contract." See §0(c)(1). Q2. switch_to ABI between kthreads (the kernel-self ↔ Asyncify-self contract) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: per-task Asyncify save buffer is allocated at ``kthread_create`` time, lives in ``task_struct.thread.asyncify_data``, sized at ``ASYNCIFY_DATA_SIZE = 16 * 1024`` (16 KiB). The kernel stack is separately allocated, sized at THREAD_SIZE per Q4. ``__wasm32_switch_to(prev, next)`` is the only function in the kernel module that touches the ``asyncify_state`` / ``asyncify_data`` wasm globals directly. Everything else above it observes a normal C function call.** Per-task storage (concrete byte layout): :: struct thread_struct { /* Asyncify save buffer. asyncify_data points HERE. The * buffer layout is opaque (defined by Binaryen's pass); * we treat it as a fixed-size void* region. */ void *asyncify_data; unsigned long asyncify_data_bytes; /* = ASYNCIFY_DATA_SIZE */ /* Kernel stack. __stack_pointer for this task starts at * stack_base + stack_size. */ void *stack_base; unsigned long stack_size; /* = THREAD_SIZE */ /* Cooperative runqueue linkage (used by the FIFO scheduler * in arch/wasm32/kernel/asyncify.c). */ struct list_head rq_node; /* R4b (§17.7 FIFTH HISTORICAL NOTE): per-task gate * distinguishing "first call to __wasm32_switch_to (start * the unwind)" from "rewind re-execution of switch_to * (call asyncify_stop_rewind)." Required because * Binaryen 125 does NOT auto-inject stop_rewind at the * bottom of the rewind chain — the topmost helper is * responsible. Cleared on boot via the zero-initialized * thread_struct; toggled by switch_to. Bool semantically; * stored as u32 for alignment-with-other-fields parity. */ unsigned int unwound; /* R5 (§17.7 SIXTH HISTORICAL NOTE): per-task slot through * which the resume-path "actual outgoing task" pointer is * passed. __wasm32_yield's resume branch sets * next->thread.last_task = BEFORE asyncify_start_rewind. * On the rewind chain's bottom, __wasm32_switch_to reads * prev->thread.last_task (where prev == the resuming task * on rewind because the rewind chain restored its saved * locals), clears it, and returns it as the macro's * `last` argument. The upstream context_switch macro * `switch_to(prev, next, prev)` then assigns prev to that * return value, so finish_task_switch(prev) processes the * correct outgoing task. NULL on first-entry (no prior * unwind, no rewound switch_to to consume it). See the * "dual-path" note below for the separation between this * field and __wasm32_pending_prev. */ struct task_struct *last_task; }; How the ABI works on a switch (post-R4b/R5): :: /* Returns the actual outgoing task (== `last` in upstream's * switch_to macro semantic). On unwind: prev. On rewind: * prev->thread.last_task (set by __wasm32_yield before * start_rewind). See "dual-path" note below. */ struct task_struct * __wasm32_switch_to(struct task_struct *prev, struct task_struct *next) { /* On rewind re-entry the wasm trampoline re-executes the * call to __wasm32_switch_to with the SAME arguments. We * distinguish unwind (first call, unwound=false) from * rewind (second call, unwound=true) via the per-task gate * on prev->thread.unwound. */ if (prev->thread.unwound) { struct task_struct *last = prev->thread.last_task; prev->thread.unwound = false; prev->thread.last_task = NULL; asyncify_stop_rewind(); return last; } prev->thread.unwound = true; prev->thread.stack_top = __builtin_wasm_get_stack_pointer(); /* Hand off prev to the post-unwind handler. The unwind chain * propagates upward to __wasm32_yield, which observes * state==UNWIND, picks the next task, and (in the resume * branch) stashes the real outgoing prev into * next->thread.last_task, then calls * asyncify_start_rewind(next->thread.asyncify_data); on the * resulting rewind chain, switch_to is re-executed with * the resuming task's saved arguments, sees unwound=true, * reads last_task, and returns it. */ __wasm32_pending_prev = prev; asyncify_start_unwind(prev->thread.asyncify_data); /* If start_unwind returns to C (Binaryen's pass actually * threads control to the wasm-level epilogue at the post- * call check), this return value is consumed by the * switch_to macro as `last = prev`. */ return prev; } /* The upstream switch_to macro consumes the return value: */ #define switch_to(prev, next, last) do { \ (last) = __wasm32_switch_to((prev), (next)); \ } while (0) **Dual-path: pending_prev vs last_task.** ``__wasm32_pending_prev`` (global, set by ``__wasm32_switch_to``) and ``next->thread.last_task`` (per-task, set by ``__wasm32_yield`` on the resume branch) are TWO DISTINCT MECHANISMS serving two distinct dispatch paths: * **First-entry dispatch** (the dispatched task has never run; ``next->thread.first_entry == true``; no saved Asyncify buffer to rewind from). ``__wasm32_yield`` reads ``__wasm32_pending_prev`` and passes it to ``schedule_tail(prev)`` so the new kthread's first action matches upstream's ret_from_fork contract. ``last_task`` is NULL on this path because no rewound ``switch_to`` is going to consume it. * **Resume dispatch** (the dispatched task has a saved Asyncify buffer; ``next->thread.first_entry == false``). ``__wasm32_yield`` BEFORE calling ``asyncify_start_rewind``: 1. ``next->thread.last_task = __wasm32_pending_prev;`` (stash the real outgoing task into the resuming task's slot, where the rewound ``switch_to`` can find it). 2. ``__wasm32_pending_prev = NULL;`` (clear; resume path does NOT call ``schedule_tail`` — the rewound ``context_switch`` will call ``finish_task_switch`` itself). 3. ``asyncify_start_rewind(next->thread.asyncify_data);`` 4. ``(void)__wasm32_kthread_dispatch(next);`` — triggers the rewind chain, whose bottom is ``switch_to`` which reads ``prev->thread.last_task`` and returns it as the macro's ``last`` arg. The right mechanism for a given dispatch is determined by ``next->thread.first_entry``. Conflating the two (e.g. "just always read pending_prev from switch_to") fails because on rewind, ``switch_to`` runs with restored saved locals — it cannot reach the currently-set value of a global without going through a memory location bound to a saved frame, which would be a different bug. This dual-path emerged from R5 (§17.7 SIXTH HISTORICAL NOTE). The original K4 design used only ``pending_prev``; the resume path read it and threw it away, on the assumption "K5's user-process spawn lifecycle is where last-task tracking matters." That assumption was refuted by the kernel busy-spinning at 100% CPU after the first resume because ``finish_task(pid 1)`` was called with the saved (wrong) prev, leaving the actual outgoing task's ``on_cpu`` stuck at 1 — the next ``try_to_wake_up`` on that task spun forever on ``smp_cond_load_acquire(&p->on_cpu, !VAL)``. **Companion mechanism: deep_sp (R6 / §17.7 SEVENTH HISTORICAL NOTE).** ``__wasm32_yield``'s resume branch must restore TWO pieces of cross-rewind state, not one: * ``next->thread.last_task`` — the actual outgoing task, carried so the rewound ``switch_to`` can return it as the macro's ``last`` arg (described above). * ``next->thread.deep_sp`` — the ``__stack_pointer`` value at the moment of the original unwind (saved in ``__wasm32_switch_to`` BEFORE ``asyncify_start_unwind``), restored by ``__wasm32_yield`` BEFORE ``asyncify_start_rewind``. The second mechanism exists because Binaryen's Asyncify REWIND path does NOT execute each function's C-compiler prologue. Function prologues on REWIND restore wasm locals from the saved buffer and jump DIRECTLY to the saved call site, SKIPPING the SP-descent that would normally occur (``__stack_pointer -= frame_size``). Through the entire rewind chain, no function descends SP. So if yield set ``__stack_pointer = stack_base + THREAD_SIZE`` (a fresh top) before ``asyncify_start_rewind``, the first normal call made from a rewound function (e.g. ``context_switch -> finish_task_switch``) would allocate its frame at ``stack_top - frame_size``, overwriting the resumed task's still-live higher frames — empirically clobbering pid 1's ``DECLARE_COMPLETION_ONSTACK(done)`` at ``stack_top - 0xc0`` in ``__kthread_create_on_node``. The fix is to start the rewind chain at the deep SP value the original unwind captured, so rewound calls' frames land BELOW the live region. As each rewound function returns, its epilogue does ``__stack_pointer += frame_size``, and SP climbs back to ``stack_top`` by the time dispatch returns to yield. Concrete sequence in ``__wasm32_yield`` (resume branch), showing both pieces of dual-path state being set up together: :: /* Snapshot yield's own SP so we can restore it after * dispatch returns. saved_yield_sp is a scalar with no * taken address; clang keeps it in a wasm local across * the dispatch call, so it survives the SP swap. */ saved_yield_sp = __wasm32_get_sp(); if (next->thread.first_entry) { /* fresh stack — no prior rewind chain to preserve */ __wasm32_set_sp(next->thread.stack_base + THREAD_SIZE); /* ... schedule_tail + dispatch ... */ } else { /* preserve the deep SP from this task's last unwind */ __wasm32_set_sp(next->thread.deep_sp); next->thread.last_task = __wasm32_pending_prev; __wasm32_pending_prev = NULL; asyncify_start_rewind(next->thread.asyncify_data); (void)__wasm32_kthread_dispatch(next); } /* After dispatch returns via the next yield's unwind * chain, each frame's epilogue restored SP, so SP is * back at stack_top. Restore yield's own SP. */ __wasm32_set_sp(saved_yield_sp); The ``__wasm32_get_sp`` / ``__wasm32_set_sp`` accessors are implemented in ``arch/wasm32/lib/current.S`` as direct ``global.get $__stack_pointer`` / ``global.set $__stack_pointer`` ops. C-level inline assembly was attempted first and silently elided by LLVM 22 even with ``volatile`` and ``"memory"`` clobber. The ``.S`` boundary is opaque to the optimizer and each accessor has no shadow-stack prologue, so calling it does not itself adjust SP beyond the assignment. The two cross-rewind pieces are independent: * ``last_task`` matters for ``finish_task_switch`` / ``finish_task`` semantics (correct ``on_cpu`` clearing). Without it: 100% CPU busy-spin on ``smp_cond_load_acquire``. * ``deep_sp`` matters for memory safety of stack-resident data in the resumed task's still-live frames. Without it: silent stack corruption of higher frames, surfacing as data races on completion objects, locks, kernel-local variables. The corruption is non-deterministic from the test harness's view: which frame gets clobbered depends on which task makes the first normal call after rewind and what frame size that call requires. Both pieces of state live in ``struct thread_struct`` (``arch/wasm32/include/asm/processor.h``) so they're naturally per-task and persist across the unwind / dispatch / rewind cycle. Future K-phase Asyncify users (K5/K6/U-series) inherit this mechanism: every cooperative-yield site must either save+restore deep_sp explicitly OR run on a stack that is private to a single resumable continuation. The "control transfers to the wasm-level epilogue" comment hides the actual mechanism, which is the subject of Q11 and is fundamentally Binaryen-pass-internal: the Asyncify pass augments switch_to's caller (``__schedule``'s context_switch block) with a post-call check that observes ``state==UNWIND``, saves the caller's frame into ``asyncify_data``, and returns. The chain propagates up through every instrumented function until it reaches the outermost ``__wasm32_yield``. **R4b correction (§17.7 FIFTH HISTORICAL NOTE).** Earlier drafts of this section had ``switch_to`` setting ``asyncify_state = ASYNCIFY_STATE_NONE`` on rewind re-entry *without* explicitly calling ``asyncify_stop_rewind``, on the assumption that Binaryen would inject the state transition automatically at the rewind chain's bottom. That assumption was empirically refuted: ``asyncify_stop_rewind`` is never called from generated code; the host (in our self-call design, ``__wasm32_switch_to`` itself) must call it explicitly. The gate flag distinguishes unwind from rewind re-entry; setting ``asyncify_state`` by hand is NOT a substitute for ``asyncify_stop_rewind`` because the latter performs additional bookkeeping inside the synthesized implementation. **Failure mode if a non-whitelisted function reaches a yield.** ``asyncify-asserts`` makes wasm-opt emit runtime checks at every function prologue: if ``asyncify_state == UNWIND`` and this function isn't on the whitelist, trap. The kernel sees this as ``__builtin_trap()`` → Worker exit → HardwareJS observes "kernel trap." Better than silent stack corruption. Q3. HardwareJS ↔ vmlinux-Worker idle protocol ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: master scheduler wakeup word at fixed offset in ``env.kernel_memory``. The vmlinux Worker parks on ``Atomics.wait(env.kernel_memory, sched_wakeup_offset, expected_value)`` when its cooperative runqueue is empty. Any party that makes a kthread runnable bumps the word and notifies. The kernel exports the offset as a wasm global ``__sched_wakeup_word`` so HardwareJS doesn't have to hardcode it.** Considered: * **(i) ``Atomics.wait`` on a master futex word in env.kernel_memory.** Correct, scalable, lets the host OS truly suspend the Worker. Requires the contract: every code path that makes a kthread runnable must notify. The contract is small (single chokepoint is ``try_to_wake_up`` on kthreads + the scheduler tick's TIF_NEED_RESCHED set path). * **(ii) Spin loop / setTimeout poll.** Trivially wrong: wastes a CPU core, doesn't let the host OS sleep, makes the laptop fan spin in an idle browser tab. Rejected. * **(iii) Per-kthread wake words.** Could work but explodes the ABI surface (one offset per task, dynamic enrolment with HardwareJS at kthread_create). And the actual question the vmlinux Worker is asking is "ANY kthread runnable" — a single bitmask-like wake word is the right granularity. **Rationale for (i).** Standard Linux UML / pre-KVM model mechanism, transliterated to the browser. The single chokepoint makes the contract auditable; the wake-word ABI is one offset and one global symbol, not a fanout of per-task state. Concrete spec: * **Offset:** within ``env.kernel_memory``, picked by the kernel C code at allocation time and exported as wasm global ``__sched_wakeup_word`` (i32, immutable, pointing at the offset). HardwareJS reads the global at instantiation time and records the offset; no hardcoded constant in either party. Same pattern as ``__heap_base`` discovery at K2. * **Type:** ``u32`` in ``env.kernel_memory``. Initial value: 0. * **Semantics:** any positive value means "kernel C code has set TIF_NEED_RESCHED on at least one kthread; vmlinux Worker should re-enter the runqueue scan." The vmlinux Worker, when it parks, reads the current value and waits on it; any party that wakes a kthread increments the word and calls ``Atomics.notify(env.kernel_memory, sched_wakeup_offset, 1)``. The kernel C side decrements the word back to 0 once it has fully processed all wakeups (a single thread-safe ack pattern via ``atomic_xchg(&__sched_wakeup_word, 0)`` at the top of the cooperative-yield loop). Concrete contract for non-kernel parties (HardwareJS, user Workers, the JS-side timer fire path): * HardwareJS exposes a host helper: ``linux.kick_scheduler()``. The kernel never calls this itself — it manipulates the word directly. The helper is for HardwareJS-side timers / process Workers / virtio backends / anything that wants the kernel to notice their effect on shared state. * ``linux.kick_scheduler`` reads the kernel-exported ``__sched_wakeup_word`` global once at boot, then performs ``Atomics.add(env.kernel_memory, offset, 1)`` followed by ``Atomics.notify(env.kernel_memory, offset, 1)``. No per-call indirection. **§15 debt row.** "Master scheduler wakeup word's offset discovery currently requires HardwareJS to call into the ``__sched_wakeup_word`` global at instantiation time. If HardwareJS-bookkept process Workers ever need to do this discovery themselves (rather than going through the parent HardwareJS), the offset must propagate via the process-Worker spawn protocol." K7+. Q4. THREAD_SIZE for wasm ~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: ``THREAD_SIZE = 32 KiB`` (``THREAD_SIZE_ORDER = 3``, with ``PAGE_SIZE = 4096``). Asyncify save buffer is a separate 16 KiB allocation. Per-task footprint: 48 KiB + ``task_struct``.** Current state: ``arch/wasm32/include/asm/thread_info.h`` already declares ``THREAD_SIZE_ORDER = 2`` (16 KiB). K4 bumps this to ``3``. Documented here so the bump is justified, not silent. Considered: * **16 KiB (status quo, Linux default).** Adequate for most arches because native code packs locals densely into registers + stack slots; ~64 frames @ 256 bytes is typical depth. wasm code emits every local as a typed wasm slot (i32 = 4 bytes minimum, i64 = 8 bytes, no register coloring at the wasm IR level), so per-frame stack consumption is ~1.5–2× heavier than native. Real-world: a printk path with vsnprintf + console_lock takes ~3 KiB of native stack; on wasm it's ~5 KiB. Lockdep on top of that adds ~6–8 KiB more. 16 KiB has zero headroom for any double-fault-grade diagnostic path. * **32 KiB.** Doubles the headroom. ~7× per-kthread storage vs ``task_struct`` alone (32 KiB + 16 KiB asyncify_data + ~4 KiB task_struct ≈ 52 KiB), which is fine at K4 where we have at most ~6 kthreads at steady state (kthreadd, kworker, rcu_*, ksoftirqd, kernel_init). Worst case ~300 KiB of per-task storage; comfortable on a 64 MiB heap. * **64 KiB.** Linus would beat us with a stick (this is one wasm-page, not a kernel-stack page). Overkill; reserved for K6+ if specific paths blow 32 KiB. * **Dynamic per-task sizing via Kconfig.** Possible (NR_CPUS is already compile-time per Q4 of K3) but unnecessary at K4. Tunable post-v0.1.0 if measurement shows variance. **Rationale for 32 KiB.** Doubles the headroom over native Linux default to absorb wasm's heavier per-frame cost. Measured against the K3 closing dmesg path (Mount-cache hash init → kmalloc-driven slab traversal), the deepest observed native stack on x86_64 for the equivalent path is ~6 KiB. We budget 5× headroom (32 KiB) instead of 2.7× (16 KiB) so that RCU + lockdep + a future trap-from-deep diagnostic don't overflow. Update ``arch/wasm32/include/asm/thread_info.h``'s ``THREAD_SIZE_ORDER`` from 2 to 3 as part of the K4 implementation commit (this file currently says "K6 may revisit"; this IS the revisit, brought forward to K4 because Asyncify-aware kthreads are the first real users). **§15 debt row.** None — this is a justified configuration decision, not a workaround. The ``thread_info.h`` comment is updated in the K4 implementation commit. Q5. Kernel-thread signal handling (KILL / STOP / freezer) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: OUT of K4 scope. ``signal_pending()`` returns 0 for kthreads at K4. ``kthread_stop()`` works (it uses ``KTHREAD_SHOULD_STOP`` bit + ``wake_up_process``, both of which are kthread-mechanism not signal-mechanism). ``KILL``, ``STOP``, ``CONT``, and the freezer/suspend signal paths are stubbed.** Considered: * **In-scope.** Kthread signals are used by the freezer (suspend/resume), by deliberate test code, and by out-of-process kernel debuggers. None of these are K4-acceptance-relevant. * **Out-of-scope, stubbed.** Saves work, doesn't block any K4 acceptance criterion, defers the decision to whenever a consumer becomes interesting (probably K6 if freezer comes in, possibly never if the browser target makes "suspend the whole virtual machine" unnecessary). **Rationale.** The browser target's "suspend" model is the host hiding the tab; the wasm runtime stops scheduling time to the Worker; on resume, the wasm Worker just picks up where it was. Linux's freezer infrastructure is geared toward managing suspend on devices with persistent state across cold boot — NOT our situation. Defer until something concrete needs it. **§15 debt row.** "Kthread signal handling deferred." See §0(c)(5). Q6. kernel_init failure shape ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: ``kernel_init`` runs as a kthread, attempts ``run_init_process(/init)`` and the other usual paths (``/sbin/init``, ``/etc/init``, ``/bin/init``, ``/bin/sh``), all fail with -ENOENT, and the kernel panics with the upstream "No working init found." message. This panic IS the K5 boundary.** Considered: * **Panic at "No working init found."** Upstream-Linux-exact. Triggers ``panic()`` → ``machine_restart`` (which currently ``__builtin_trap()``-s per §15) → vmlinux Worker exit. The K4 vitest observes "kernel-trap" with the panic message in dmesg. * **Hang.** Would also satisfy "reaches kernel_init"; less diagnostic. Rejected. * **Synthesize a fake successful exec.** Would let K4 boot proceed past kernel_init; would be a lie. Rejected. **Rationale.** "No working init found" is the canonical "kernel is fully booted, userland is missing" Linux signal. Anyone reading our K4 dmesg recognizes it instantly. K5 satisfies it by providing /init (which is the userland binfmt_wasm-spawned first user process). **§15 debt row.** None — this is the K5 boundary, not debt. Q7. Asyncify tool selection (Binaryen vs JSPI vs custom) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: Binaryen's ``wasm-opt --asyncify`` pass, pinned toolchain version recorded alongside the LLVM 22 / Node 25 pin already established at K3 (mm-model.rst §2 Q3).** Considered: * **Binaryen asyncify.** Mature (~5 years in production at Emscripten / Cheerp). Tool-independent — works on any wasm runtime that implements the threads / atomics proposals; no engine-level cooperative-suspension support required. State stays in linear memory under our control. Generates enough hooks for our needs. Pinned version recorded for K4 reproducibility (Binaryen ≥ 119; the upcoming K4 commit records the exact version against the Q3 ``clang/wasm-ld 60513b8d``-pinned toolchain). * **JSPI (JS Promise Integration).** Engine-level cooperative-suspension support, shipping in V8 since v25 and in SpiderMonkey since 132. Requires marking imports as ``suspending`` at JS instantiation time (a §18-flavour JS-object-shape ABI constraint); requires the kernel to surface yield points as imports rather than internal function calls; pulls Promise plumbing into the kernel- HardwareJS boundary. Re-evaluable at K7+ once we have perf data from Binaryen-asyncify in real workloads. Right now JSPI's cost (ABI surface area, host-side promise glue, engine-version requirement) outweighs its benefit (~25% less binary size than asyncify). * **Custom continuation-passing transformation.** Reinvent Binaryen's wheel. Rejected — categorically the kind of YAK §16 rejects. **Rationale.** Binaryen's asyncify is the standard, maintained, runtime-agnostic answer. K4 records the pinned version in commit body and in ``Documentation/wasm/toolchain.rst`` (or wherever the toolchain pin spec lives at K4 close — if that doc doesn't exist yet, K4 introduces a one-page version). **§15 debt row.** "Re-evaluate Binaryen asyncify vs JSPI at K7+ when we have real-workload perf data." Soft debt — not blocking, but flag for revisit. Q8. cond_resched() maintenance contract ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: every long-running kernel-C loop MUST contain explicit ``cond_resched()`` calls at intervals; the K4 cooperative model has no other yield mechanism.** Status: this is upstream Linux convention already (RT and PREEMPT_NONE configs both rely on it). The K4 thing is: on this port, ``cond_resched()`` is the ONLY yield primitive short of a full block. Any kernel C path that loops without ``cond_resched()`` will starve other kthreads. Implications: * upstream Linux's already-present ``cond_resched()`` calls in ``mm/`` and ``kernel/`` Just Work. * If a future K-phase exercises a kernel-internal loop that is rare on real arches and so was never instrumented with ``cond_resched()`` upstream, the K-phase commit MUST add ``cond_resched()`` (preferably push the fix upstream). * User code in arch/wasm32 (currently just K2's setup_arch validation loops; nothing K3 added except memblock-walk loops which all have ``cond_resched()`` upstream) follows the same rule. **§15 debt row.** None — this is a convention, not a debt item; it's documented here so future K-phase reviewers know to look for it. Q9. printk-from-asyncify and the Never-Yields blacklist ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: ``printk`` and all functions it calls (``vsnprintf``, ``console_drivers``' write callbacks, ``__wasm32_dmesg_write``) are explicitly excluded from the Asyncify whitelist. ``asyncify-asserts`` will trap if any of them reaches a yield point.** Why: ``printk`` runs from any context — including ones where the caller has already started an unwind. If printk itself yielded, the unwind chain would re-enter printk during rewind, and printk's locks would deadlock. This is the inverse of Q8: cond_resched is mandatory on allowed-to-yield paths; printk is mandatory on the NEVER-yields list. Both are convention contracts; both are enforced at runtime by Binaryen's asyncify-asserts pass. **The blacklist** (initial K4 set, expandable): * ``printk`` and ``vprintk`` family. * ``vsnprintf``, ``snprintf``, ``sprintf``. * ``__wasm32_dmesg_write`` (host-import callback target). * ``console_drivers``' ``write`` and ``flush`` callbacks (just the bootconsole at K4). * ``rcu_read_lock`` / ``rcu_read_unlock`` and the rcu_read-side fast paths (yielding inside an RCU read-side critical section breaks the grace period contract). * ``spin_lock`` / ``spin_unlock`` / ``raw_spin_lock`` / ``raw_spin_unlock`` (yielding while holding a spinlock deadlocks any waiter). * ``BUG`` / ``BUG_ON`` / ``WARN`` / ``WARN_ON`` (no point yielding from a diagnostic-emit path). Generation: ``scripts/asyncify-whitelist.py`` generates ``arch/wasm32/asyncify-whitelist.txt`` from the yield-root reachability closure; the blacklist is the COMPLEMENT of that whitelist by construction. Asyncify-asserts only emits checks against the whitelist, so the blacklist's "never yields" property follows from list-membership. **§15 debt row.** None — the rule is implementation-level and lives in the whitelist generator. Q10. workqueue / kworker ~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: ``kworker`` per-CPU threads are normal kthreads under our K4 machinery; no special arch hook needed. Upstream ``workqueue.c`` calls ``kthread_create`` / ``kthread_create_on_cpu`` for each kworker; K4's ``kthread_create`` implementation is complete enough that these kworkers spawn, park on their wait queues, and run work items cooperatively when an item is queued.** Status check: K3 already linked ``workqueue.c`` (and panicked in ``init_pwq`` until ``CONFIG_SLUB_TINY=y`` made the per-CPU slab allocations work). K4's contribution is that the kthread the workqueue infrastructure tries to spawn at the bottom of the K3 progression — ``kworker/0:0``, ``kworker/0:0H``, etc. — actually run for the first time. **§15 debt row.** None — this Just Works through composition of K3 and K4. Q11. Unwind/rewind reentry pattern ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: self-call reentry from inside the kernel C code (``arch/wasm32/kernel/asyncify.c::__wasm32_yield``). NO host- side reentry loop. The host (HardwareJS, vmlinux Worker) sees the kernel as a single ``wasm_start_kernel`` call that keeps internally yielding to itself.** Considered: * **(i) Self-call from kernel C.** The yield function does: :: while (1) { /* asyncify_state = UNWIND set by __wasm32_switch_to */ asyncify_stop_unwind(); /* prev's stack is now saved in prev->thread.asyncify_data. */ struct task_struct *next = pick_next_runnable(); if (!next) { /* Park on master wakeup word. */ __wasm32_park_until_runnable(); continue; } /* Set up rewind. */ asyncify_state = ASYNCIFY_STATE_REWIND; asyncify_data = next->thread.asyncify_data; /* Call back into the kernel — Asyncify rewinds into * next's saved continuation. */ __wasm32_kernel_resume(); // a separate function so // Asyncify treats it as a // new entry point. /* (control never returns here; rewind transfers into * next's frame instead.) */ } Self-contained: HardwareJS doesn't have to know that switch_to happens. The vmlinux Worker enters ``wasm_start_kernel`` once and stays there until panic / trap. * **(ii) Host-loop reentry.** The kernel returns to HardwareJS at every switch_to, HardwareJS observes the state, calls back in. Trivially exposes the Asyncify-state ABI to HardwareJS — bad: §18 says JS-side shape constraints belong in docs; Asyncify-state shape is one. Also slower (one host round-trip per context switch). Rejected. **Rationale.** Keep Asyncify state entirely inside the kernel module; HardwareJS sees no change from K3 except that ``wasm_start_kernel`` now runs forever (until panic), parking on the scheduler wakeup futex when idle, instead of returning when it reached the K3 hang. **§15 debt row.** None — this is the implementation choice the K4 code implements; nothing later phases need to revisit. Q12. Kthread first-entry shim ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: ``kthread_create`` allocates the per-task asyncify_data buffer, kernel stack, and ``thread_struct``; then enqueues the kthread in the cooperative runqueue with an "initial" Asyncify state that causes the first rewind into the kthread's entry function (``kthread`` -> ``threadfn(data)``) rather than into a previously-saved continuation.** Mechanism: the initial asyncify_data buffer is hand- constructed by ``__wasm32_kthread_init`` to look like a saved state at the entry of ``kthread()`` (the kernel's ``kthread/kthread.c::kthread`` function, the universal kthread entry trampoline). Specifically: * The save buffer's header indicates "rewind into ``kthread`` at function entry." * The buffer's locals area holds ``data`` and ``threadfn`` as if they had just been pushed for a normal call. This is a one-shot hand-construction. Subsequent yields and resumes use the normal Asyncify-save mechanism. **§15 debt row.** "Hand-constructed initial asyncify_data buffer depends on Binaryen's per-function save-buffer layout, which is a Binaryen-private implementation detail. A Binaryen version bump could break the layout assumption. Detection: ``asyncify-asserts`` catches it as a corrupt restore at the first kthread run." Owed: pinned-toolchain discipline (Q7). Not K-phase-tied. Q13. Asyncify state buffer sizing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Decision: ``ASYNCIFY_DATA_SIZE = 16 KiB`` per kthread. Empirically validated at K4 close against the deepest observed save-state depth from the K4 acceptance run.** Considered: * **4 KiB.** Adequate for shallow callers (kthreadd's main loop is ~4 frames deep). Insufficient for any path that goes through ``schedule_timeout`` → ``hrtimer`` → RCU. * **16 KiB.** Generous enough to handle the deepest observed K4 path (~20 frames, ~9 KiB save state). Matches THREAD_SIZE_ORDER's reasoning of "5× headroom over observed worst case." * **THREAD_SIZE.** Tempting to colocate them (one allocation of 32 KiB total). Slightly wasteful — the save buffer doesn't grow with stack depth linearly, it grows with number of active call frames × per-frame locals, which is roughly half the C stack consumption. Separate allocations keep the buffer sizing tunable independently of stack sizing. **Rationale.** 16 KiB is the smallest power-of-two that provides 5× headroom over the deepest observed save-state in the K4 boot path. Recorded at K4 close with the actual high-water-mark. **§15 debt row.** None — tunable Kconfig constant; revisit if K-phase boot paths grow deeper. -------------------------------------------------------------------------------- 3. Asyncify mechanism -------------------------------------------------------------------------------- 3.1 Whitelist generation ~~~~~~~~~~~~~~~~~~~~~~~~ **K4 STATUS:** The curated-whitelist approach below is the sched-model design target. **K4 ships without** a curated whitelist or the generator script: Binaryen's default instrumentation scope (every function transitively calling ``__wasm32_switch_to`` gets the save/restore prologue) is used instead. This results in a ~2-3× larger ``vmlinux.wasm`` post-Asyncify than the whitelist would produce. Deferral rationale + retirement protocol is recorded as a §15 row in ``docs/ARCHITECTURE.md`` ("Asyncify pass runs with Binaryen DEFAULT instrumentation scope"). K6 lands the whitelist generator and reduces the binary; K4 records the no-whitelist baseline as the K6-target reduction reference. The DESIGN (K6 target): ``scripts/asyncify-whitelist.py`` (introduced at K4 in spec, deferred to K6 in implementation, sibling to ``wasm-add-memory.py``, ``wasm-inspect.py``, and the upcoming K5/K6 percpu post-link pass) emits ``arch/wasm32/asyncify-whitelist.txt``. Algorithm: 1. Walk the pre-Asyncify ``vmlinux.wasm`` symbol table via ``llvm-objdump --syms``. 2. Build a call graph from the wasm ``code`` section's ``call`` and ``call_indirect`` instructions, mapping each function index to its callees. 3. Seed the closure with the YIELD ROOTS set: :: schedule cond_resched wait_event wait_event_interruptible wait_event_timeout wait_event_interruptible_timeout msleep msleep_interruptible schedule_timeout schedule_timeout_interruptible wait_for_completion wait_for_completion_interruptible wait_for_completion_killable __wasm32_switch_to 4. Transitively include all CALLERS of yield roots (because any caller of a yielding function might observe an unwind through its own frame). 5. Apply the BLACKLIST exclusion (Q9), removing the printk / spinlock / RCU-read-side functions from the whitelist even if they appear transitively. ``asyncify-asserts`` will catch any indirect-call escape at runtime. 6. Emit the whitelist as one function-name-per-line, alphabetized, in ``arch/wasm32/asyncify-whitelist.txt``. The build system invokes ``wasm-opt`` at K4 with:: wasm-opt -g vmlinux.wasm \ --asyncify \ --pass-arg=asyncify-asserts \ --pass-arg=asyncify-removelist@__wasm32_yield \ --pass-arg=asyncify-removelist@wasm_start_kernel \ -o vmlinux.wasm.asyncify-with-asserts (One ``--pass-arg=asyncify-removelist@NAME`` per function: Binaryen 125 does not accept ``@file`` syntax for the removelist, as verified by the K4 asserts-probe and per the build script's read of ``arch/wasm32/kernel/asyncify-removelist.txt``.) The K6 target invocation will add:: --pass-arg=asyncify-only-list@arch/wasm32/asyncify-whitelist.txt The ``-g`` flag preserves the function-name section through the pass; the post-pass strip tool requires it to resolve names from ``asyncify-topmost.txt``. Then the topmost-runtime asserts get surgically stripped:: scripts/asyncify-strip-topmost-asserts.py \ vmlinux.wasm.asyncify-with-asserts \ --topmost-list arch/wasm32/kernel/asyncify-topmost.txt \ -o vmlinux.wasm.asyncify The Asyncify pass runs AFTER the K3 percpu-anchor-bracket final link and BEFORE the K3 user_memory splice (so ``wasm-add-memory.py`` operates on the strip-tool's output; the splice has no opinion about Asyncify). Toolchain drift in either the asyncify pass or the strip tool is caught at CI by the two sibling probes, ``tools/k4-binaryen-asyncify-probe/`` (asyncify_data layout) and ``tools/k4-asyncify-asserts-probe/`` (strip-tool surgical behavior). Both run on every build, so a Binaryen bump that changes either the assert encoding or the state machine fails the probe before vmlinux mysteriously traps in ``__wasm32_switch_to``. 3.1a Topmost-list vs removelist: which list, when? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Two adjacent ABI files cooperate with the Asyncify pass + strip tool: ``arch/wasm32/kernel/asyncify-topmost.txt`` (read by ``scripts/asyncify-strip-topmost-asserts.py``) and ``arch/wasm32/kernel/asyncify-removelist.txt`` (read by ``scripts/link-vmlinux-wasm.sh`` and emitted as a single ``--pass-arg=asyncify-removelist@A,B,C`` comma-joined argument; see R4a in ``docs/ARCHITECTURE.md`` §17.7 FOURTH HISTORICAL NOTE for why one-per-arg silently drops all but the last). The interaction is subtle enough that the next person adding a topmost helper should NOT have to re-derive it from a confusing build failure. Decision table (post-R4b): ============================================ ============== ============== ============================================ Function role topmost-list removelist What goes wrong if either is omitted / mistakenly included ============================================ ============== ============== ============================================ Pure topmost helper: calls ``asyncify_start_ YES **NO** topmost-list-only is correct. unwind`` (or ``asyncify_stop_rewind``) directly and is otherwise call-graph-leaf on the asyncify side (e.g. ``__wasm32_switch_to``). Binaryen's scanner sets ``info.isTopMostRuntime=true`` for any function calling ``START_UNWIND`` or ``STOP_REWIND`` directly; ``needsInstrumentation`` then returns false unconditionally. The auto-detection is sufficient — no removelist entry needed. ``asyncify-asserts`` injects asserts; we strip them via topmost-list. --- --- **Omit topmost-list:** asserts remain; function traps at runtime on the legitimate state change. --- --- **Mistakenly add to removelist:** see R4b (``§17.7`` FIFTH HISTORICAL NOTE). ``asyncify-removelist`` ALSO sets ``info.canChangeState=false``, which propagates back through the call graph and suppresses the post-call state-check that *callers* of the listed function would otherwise receive. The caller's frame is never saved on unwind; on rewind the buffer is one frame short and the caller's prologue reads garbage. Symptom: kernel trap inside ``vprintk_emit`` via ``__schedule_bug`` triggered by a garbage call-site-idx in the caller's saved-but-corrupted locals. -------------------------------------------- -------------- -------------- -------------------------------------------- Driver function: observes ``state==UNWIND`` YES YES BOTH required. across a child call and explicitly calls ``asyncify_stop_unwind`` / ``asyncify_start_rewind`` (e.g. ``__wasm32_yield``, ``wasm_start_kernel``). Binaryen would normally INSTRUMENT this for save/restore (because it transitively reaches ``asyncify_start_unwind`` via the child call) AND has ``canChangeState=true`` because no removelist entry suppresses it; we must force NON-instrumentation via removelist AND strip the resulting asserts via topmost-list. The post-call check suppression on removelisted driver callers is fine here: the driver IS the bottom of the asyncify call chain (post-unwind it re-enters the yield loop in the host, not a higher-level kernel function); no upward propagation matters because there is no caller frame that needs saving. --- --- **Omit removelist:** Binaryen inserts save/restore prologue; on unwind, the driver silently saves its own frame and propagates upward — the unwind escapes the kernel (host-loop reentry, Q11 rejected). Symptom: wasm_start_kernel returns to HardwareJS with state==UNWIND, kernel exits. --- --- **Omit topmost-list:** asserts remain in non-instrumented function; driver traps the moment its child returns with state==UNWIND. Symptom: kernel trap on the FIRST schedule() call. -------------------------------------------- -------------- -------------- -------------------------------------------- Plain whitelisted callee: appears on the no no neither required. Binaryen instruments yield call chain (e.g. upstream for save/restore, asserts don't apply ``context_switch``, ``__schedule``). No to instrumented functions. direct asyncify_* state-machine calls. ============================================ ============== ============== ============================================ Cheat sheet for adding a new function: * "I call ``asyncify_start_unwind`` directly and that's all, the rest of the asyncify state-machine is somebody else's problem" → topmost-list ONLY. **Do NOT add to removelist — Binaryen's auto-detection already does the right thing, AND removelist propagates canChangeState=false to your callers in a way that breaks their frame saves.** * "I'm a driver that needs to observe state changes across child calls" → topmost-list AND removelist. * "I just need to be in the unwind path" → neither (let Binaryen's default instrumentation handle it). The build fails loudly if a topmost-list name doesn't appear in the binary (typo / dead-code stripping detection). The removelist does NOT fail loudly on typos — Binaryen emits a warning and continues. Cross-check by reading the linker log for ``warning: Asyncify removelist contained a non-existing function name`` and fix the spelling. The ``linux-wasm/tools/k4-binaryen-asyncify-probe/`` sub-tests verify both the removelist's caller-side-propagation behavior and the multi-entry comma-syntax against the pinned Binaryen; they fail loudly if a version bump changes either semantic. 3.2 wasm globals ~~~~~~~~~~~~~~~~ After Binaryen's asyncify pass, ``vmlinux.wasm`` contains the following NEW wasm globals: ============== ====== =================== ===================================== Name Type Mutable Semantics ============== ====== =================== ===================================== asyncify_state i32 mut 0=NONE, 1=UNWIND, 2=REWIND. Set by ``__wasm32_switch_to``, read by every whitelisted function at prologue/epilogue. asyncify_data i32 mut Offset in default memory of the save buffer for the current unwind/rewind operation. Set by ``__wasm32_switch_to`` to ``{prev,next}->thread.asyncify_data``. ============== ====== =================== ===================================== Plus the existing K3 globals (``__heap_base``, ``__stack_pointer``, ``__data_end``). The kernel C side accesses these via Binaryen's clang builtins: :: extern unsigned int asyncify_state __attribute__((import_module("env"), import_name("asyncify_state"))); Wait — Binaryen's asyncify pass DEFINES these globals as part of the module's local globals, not as imports. The correct C access pattern is via inline asm: :: static inline u32 __wasm32_asyncify_get_state(void) { u32 r; asm volatile ("global.get $asyncify_state\n local.set %0" : "=r"(r)); return r; } The K4 implementation commit pins the exact accessor pattern — this section will be updated at K4 close with the actual idiom that landed (the same one the Q3 cross-memory atomic probe pioneered: empirical proof at the K4 opener that the C-side access works in our toolchain). 3.3 Binary-size delta tracking ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Every K-phase's close-commit body from K4 onward includes a ``vmlinux.wasm size`` line and a per-kthread overhead line. **K4 close commit body — REQUIRED format** (this is not optional; the K4-close discipline depends on the K6 reduction target being a measurable named number, not a fuzzy memory): :: vmlinux.wasm size (post-Asyncify, post-user_memory-splice): K3: N bytes (pre-Asyncify, recorded retrospectively at K4 open). K4: N bytes (Δ +X% vs K3 baseline) K4 is the K6 reduction BASELINE: Q1 curated whitelist deferred to K6 (see §15 row "Asyncify pass runs with Binaryen DEFAULT instrumentation scope"). K6 reduction target: −N% of post-Asyncify section size, OR an absolute (X − Y) MiB cut, whichever is the tighter number K6 commits to. Without recording the K4 number explicitly NOW, the K6 measurement loses its reference. K5: N bytes (Δ +Y% vs K4) K6: N bytes (Δ +Z% vs K4 baseline; expected NEGATIVE) ... kthread overhead bytes: formula: nr_kthreads × (THREAD_SIZE + ASYNCIFY_DATA_SIZE) = nr_kthreads × (32 KiB + 16 KiB) = nr_kthreads × 48 KiB K4: nr_kthreads_at_steady_state × 48 KiB = ??? KiB K5: ... K6: ... Notes: - With NR_CPUS=8 and the kworkers + RCU helpers + ksoftirqd + the usual zoo, expect ~50–80 kthreads at full kernel bring-up in late K-phases, so ~2.5–4 MiB of pure kthread overhead before any userspace runs. - That's fine on a 64 MiB SAB but would dominate on a smaller one. If this line ever exceeds 8 MiB before userspace, revisit THREAD_SIZE_ORDER and ASYNCIFY_DATA_SIZE. Companion to the K3-instituted ``wasm-inspect.py --atomic-count`` line. Becomes a permanent line in every phase's close-commit metrics block, alongside ``wasm-inspect.py --asyncify-info`` for the asyncify-callsite tally introduced at K4. The K6 close-commit MUST cite the K4 baseline by number when recording its reduction. "Reduced by N%" without naming the absolute number against which N is measured is not acceptable under this convention — the K4 number IS the contract. -------------------------------------------------------------------------------- 4. ``switch_to`` and the per-task ABI -------------------------------------------------------------------------------- Per Q2 above. The on-the-wire ABI between ``__wasm32_switch_to(prev, next)`` and the rest of the kernel (call sites in ``kernel/sched/core.c::context_switch``) is the standard upstream contract — no changes. The novelty is entirely in ``__wasm32_switch_to``'s implementation. Files (all K4-introduced): * ``arch/wasm32/include/asm/switch_to.h`` (already exists from K3; K4 makes its body real). * ``arch/wasm32/include/asm/thread_info.h`` (extends with ``stack_top`` per Q2; bumps ``THREAD_SIZE_ORDER`` to 3 per Q4). * ``arch/wasm32/include/asm/processor.h`` (extends ``struct thread_struct`` with the layout in Q2). * ``arch/wasm32/kernel/switch_to.c`` (the function body — sets the two Asyncify globals, calls ``asyncify_start_unwind``, returns; the rewind happens via Q11's reentry pattern). * ``arch/wasm32/kernel/asyncify.c`` (the cooperative runqueue + reentry loop in ``__wasm32_yield``). The ABI is single-Worker (vmlinux), single-CPU-fiction (num_online_cpus() == 1) at K4. ``switch_to`` is never invoked from any other Worker. -------------------------------------------------------------------------------- 5. Cooperative runqueue (Regime 2 implementation) -------------------------------------------------------------------------------- A small FIFO ring buffer in ``arch/wasm32/kernel/asyncify.c``, exposed through ``wasm_sched_class`` (``arch/wasm32/kernel/ sched.c``) per ``docs/ARCHITECTURE.md §17.7``: :: static LIST_HEAD(wasm32_rq); static DEFINE_RAW_SPINLOCK(wasm32_rq_lock); void wasm32_rq_enqueue(struct task_struct *p) { unsigned long flags; raw_spin_lock_irqsave(&wasm32_rq_lock, flags); list_add_tail(&p->thread.rq_node, &wasm32_rq); raw_spin_unlock_irqrestore(&wasm32_rq_lock, flags); } struct task_struct *wasm32_rq_peek(void) { struct task_struct *p = NULL; unsigned long flags; raw_spin_lock_irqsave(&wasm32_rq_lock, flags); if (!list_empty(&wasm32_rq)) p = list_first_entry(&wasm32_rq, struct task_struct, thread.rq_node); raw_spin_unlock_irqrestore(&wasm32_rq_lock, flags); return p; } There is no ``wasm32_rq_dequeue`` primitive at the rq level. Removal from ``wasm32_rq`` happens *exclusively* through ``dequeue_task_wasm`` (the ``wasm_sched_class`` method), which is itself called *exclusively* from ``deactivate_task`` along the state-changing yield path of ``__schedule`` (path A — see §5.3 below). This is the peek-not-pop discipline. Concurrency: the spinlock is needed because process Workers (Regime 1, K5+) can ``wake_up_process`` a kthread, and ``try_to_wake_up`` lands in our ``wasm_sched_class.enqueue_task`` which is the front-door for ``wasm32_rq_enqueue``. At K4 there's only the vmlinux Worker, so the lock is uncontended; the cost is one ``i32.atomic.rmw.cmpxchg`` per peek / enqueue (K3-real atomic). K5 starts to actually contend it. The "irqsave" variant is what kernel locking convention mandates here even though we have no IRQs — the macros compile to non-IRQ-flag-saving versions on ``CONFIG_PREEMPT_NONE``-equivalent configs but the discipline keeps the kernel C code uniform across arches. ``wasm_sched_class`` wires these primitives through: * ``enqueue_task_wasm(rq, p, flags)`` → ``wasm32_rq_enqueue(p)``. Called from ``activate_task`` along the ``wake_up_new_task`` / ``try_to_wake_up`` paths. At K4 BUG_ONs if ``p->mm != NULL`` (Regime 1 is K5+). * ``dequeue_task_wasm(rq, p, flags)`` → removes ``p`` from ``wasm32_rq`` via ``list_del_init(&p->thread.rq_node)`` and decrements ``rq->nr_running``. Called from ``__schedule → block_task → dequeue_task`` when a task goes to sleep (state set to a non-RUNNABLE state by the caller before ``schedule()``), and from ``deactivate_task`` during task migration (K5+). This is the **only** removal site from ``wasm32_rq``; see §5.3 (peek-not-pop) for why. * ``pick_next_task_wasm(rq, prev)`` → returns ``list_first_entry(&wasm32_rq, ...)`` without removing (peek). If ``wasm32_rq`` is empty, returns NULL and ``__schedule`` selects ``rq->idle``. The K5+ contract: a task is on ``wasm32_rq`` if and only if it is runnable; ``pick_next_task_wasm`` is allowed to return ``prev`` itself (which then short-circuits ``__schedule``'s ``prev != next`` branch — no ``context_switch``, no unwind, the running task continues). * ``task_fork_wasm(p)`` → ensures the new task's ``thread.rq_node`` is a valid ``LIST_HEAD_INIT``-equivalent empty node and its ``first_entry`` flag is true. ``__wasm32_yield`` (the cooperative dispatcher, see §3.2 and §4 above) also peeks rather than pops. The dispatched task remains at the head of ``wasm32_rq`` while it is executing. When the task voluntarily yields back through ``__schedule``, the state-of-the-runqueue decides what happens next; see §5.3. All other ``sched_class`` methods (``task_tick``, ``check_preempt_curr``, ``select_task_rq``, ``update_curr``, ``prio_changed``, ``switched_from``, ``switched_to``, ``balance``, ``yield_task``, ``yield_to_task``, ``migrate_task_rq``, ``rq_online``, ``rq_offline``, ``task_dead``) are no-ops, each guarded by a ``WARN_ONCE`` so the first call from a path we didn't anticipate is observable. The no-op set is the K4 maintenance contract: a method that WARN_ONCE-fires in CI is a method we owe a real implementation of, OR a path we owe a "why this never fires" note in this section. 5.1 Pinning ``p->sched_class`` to ``wasm_sched_class``: two arch hooks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Routing the class methods correctly requires that every task's ``p->sched_class`` actually point at ``&wasm_sched_class``. The class is correctly wired into the section ordering and is returned by the cooperative dispatcher, but upstream code in ``kernel/sched/`` writes ``p->sched_class`` directly at multiple sites. Each such site is hooked, per ``docs/ARCHITECTURE.md §17.7`` (post-second-amendment). At K4 there are two such sites, and therefore two arch hooks. See §17.7 SECOND HISTORICAL NOTE for the META-LESSON about enumerating all assignment sites before declaring an arch hook strategy complete. **Site 1: end of ``sched_fork()``** (creation-time class assignment, policy-driven). Upstream ``sched_fork`` ends with: :: if (rt_prio(p->prio)) { p->sched_class = &rt_sched_class; } else if (task_should_scx(p->policy)) { p->sched_class = &ext_sched_class; } else { p->sched_class = &fair_sched_class; } The arch hook ``arch_sched_fork(p)`` is added by ``linux-wasm/patches/0001-sched-add-arch_sched_fork-hook.patch`` as a ``__weak`` no-op default; ``arch/wasm32/kernel/sched.c`` provides the strong override that sets ``p->sched_class = &wasm_sched_class`` unconditionally. The hook fires at the very end of ``sched_fork``, *after* the upstream policy → class mapping above runs, so the override is the last write that wins. **Site 2: ``__setscheduler_class()``** (any-time class reassignment, also policy-driven). Upstream ``__setscheduler_class(policy, prio)`` returns the class that ``__sched_setscheduler`` and ``__rt_mutex_setprio`` should assign: :: if (dl_prio(prio)) return &dl_sched_class; if (rt_prio(prio)) return &rt_sched_class; if (task_should_scx) return &ext_sched_class; return &fair_sched_class; This is invoked at every ``sched_setscheduler*()`` call, every ``sched_setattr()``, and every PI-mutex priority inheritance. The notable case for K4 is upstream ``kthread()`` (the body that runs as the entry function of every kthread, in ``kernel/kthread.c``): it calls ``sched_setscheduler_nocheck(current, SCHED_NORMAL, ¶m)`` as its first action, to reset whatever priority the new kthread inherited from ``kthreadd``. Without a hook at this site, the first instruction every kthread executes re-routes ``p->sched_class`` from ``wasm_sched_class`` to ``fair_sched_class``, and the next ``enqueue_task`` traps in ``enqueue_task_fair`` against an uninitialized ``p->se`` / CFS state. The arch hook ``arch_setscheduler_class(int policy, int prio)`` is added by ``linux-wasm/patches/0003-sched-add-arch_setscheduler_class-hook.patch`` as a ``__weak`` default returning ``NULL`` (upstream behavior preserved); ``arch/wasm32/kernel/sched.c`` provides the strong override that returns ``&wasm_sched_class`` unconditionally. Upstream ``__setscheduler_class`` is patched to consult the hook first and short-circuit on non-NULL: :: const struct sched_class *arch_class; arch_class = arch_setscheduler_class(policy, prio); if (arch_class) return arch_class; /* upstream policy → class mapping continues */ The hook signature is intentionally minimal — returning ``const struct sched_class *`` with ``NULL = passthrough`` — and NOT a boolean-veto-with-out-param shape. The rationale for the chosen shape is documented in ``docs/ARCHITECTURE.md §19`` (no-premature-flexibility rule), with this hook as the worked example. **Site 3 (boot-only): ``init_task``.** The boot CPU's idle task ``init_task`` is constructed at link time, not via ``sched_fork``, so the ``arch_sched_fork`` hook does not pick it up. ``arch/wasm32/kernel/head.c::wasm_start_kernel`` explicitly sets ``init_task.sched_class = &wasm_sched_class`` after ``__wasm32_set_current(&init_task)`` and before the first ``schedule()`` call. (Site 3 is not an arch hook, just a direct arch-side write at boot.) The two-hook design plus the boot-time direct write together guarantee the invariant: *for every task that ever runs on a wasm32 CPU,* ``p->sched_class == &wasm_sched_class`` *for the task's entire lifetime.* The other sched classes (``fair_sched_class``, ``rt_sched_class``, ``deadline_sched_class``, ``stop_sched_class``, ``idle_sched_class``, ``ext_sched_class``) remain compiled in but are never assigned to any ``p->sched_class`` at runtime. 5.2 K4 close metric: arch hook count ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The K4-close commit body records the count of arch hooks added to upstream ``kernel/sched/`` along with the count of arch hook *overrides* in ``arch/wasm32/kernel/sched.c``. At K4 close: * 2 weak hooks added upstream (``arch_sched_fork``, ``arch_setscheduler_class``). * 2 strong overrides in ``arch/wasm32/``. * 1 boot-time direct write (``init_task.sched_class`` in ``head.c``). * Total ``p->sched_class`` assignment sites hooked: 3 of 3 (sites 1, 2, 3 above). Future K-phases that surface a previously-missed ``p->sched_class`` assignment site increment this count and extend ``docs/ARCHITECTURE.md §17.7``. The grep-then-trust discipline (§17.7 META-LESSON, promoted to ``docs/ARCHITECTURE.md §20``) is the preventive measure. 5.3 Peek-not-pop: ``__wasm32_yield`` and the two yield paths ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A task is on ``wasm32_rq`` if and only if it is runnable. The running task is on ``wasm32_rq`` (at the head) for the entire duration of its execution; it is removed only when it transitions to a non-RUNNABLE state via ``deactivate_task → dequeue_task_wasm``. This is the **peek-not-pop** discipline, and it matches upstream CFS semantics (the running task remains in the CFS rb-tree until it sleeps) using a list-head rather than an rb-tree. Equivalently: ``__wasm32_yield``'s dispatch loop reads ``wasm32_rq_peek()``, not a ``wasm32_rq_dequeue()`` (there is no such primitive). The dispatched task is referenced in ``current`` and ``rq->curr`` but is still on ``wasm32_rq`` while executing. The rationale is the path-A / path-B distinction in upstream ``__schedule``: :: __schedule(sched_mode) ... prev_state = READ_ONCE(prev->__state) # Path A: state-changing voluntary yield. # The caller set prev->__state to TASK_*INTERRUPTIBLE # / TASK_UNINTERRUPTIBLE / etc. *before* calling # schedule(). __schedule blocks prev. if (!preempt && prev_state && !signal_pending_state(prev_state, prev)) { block_task(rq, prev, DEQUEUE_NOCLOCK) -> dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK) -> dequeue_task_wasm -> list_del_init(&prev->thread.rq_node) } # Path B: state-preserving voluntary yield. # cond_resched(), bare schedule() with state still # RUNNING, preemption, or signal_pending forcing the # stay-runnable branch. __schedule does NOT block prev. next = pick_next_task(rq, prev, &rf) -> pick_next_task_wasm -> wasm32_rq_peek() # head of rq if (likely(prev != next)) context_switch(rq, prev, next, &rf) -> switch_to -> __wasm32_switch_to -> asyncify_start_unwind If the dispatcher popped on dispatch and re-enqueue only happened on ``wake_up_process``, path B would be broken: ``pick_next_task_wasm`` would see an empty rq (because the dispatcher had already removed the running task), return ``rq->idle``, and ``prev != next`` would unwind ``prev`` permanently — there is no other dispatcher path that would re-enter ``prev`` later, because ``prev`` was never put back on the rq (path B's whole point is that the task is *still runnable*). With peek-not-pop: * **Path A** behaves identically before and after the fix: ``deactivate_task`` removes ``prev`` from ``wasm32_rq``, ``pick_next_task_wasm`` returns the new head (or NULL → ``rq->idle``), ``context_switch`` unwinds ``prev`` and the dispatcher picks the next runnable task or parks. * **Path B** is handled correctly: ``prev`` is still at the head of ``wasm32_rq`` (nobody removed it), ``pick_next_task_wasm`` returns ``prev``, ``__schedule`` sees ``prev == next`` and short-circuits. ``context_switch`` is not called. No unwind. The running task continues past the cooperative-yield point — the correct outcome for a state-RUNNING voluntary yield in a single-runnable-task scenario. * **Path B with another runnable task**: when another task was enqueued (via ``wake_up_process``) before the yield, the head of ``wasm32_rq`` is *that* task, not ``prev``. ``pick_next_task_wasm`` returns the other task, ``prev != next``, ``context_switch`` unwinds ``prev``, the dispatcher resumes the other task. ``prev`` is still on ``wasm32_rq`` (we peeked, didn't pop), so the next time the dispatcher peeks it will pick ``prev`` back up (after any higher-priority head ahead of it has either been ``deactivate_task``'d via path A or dispatched and yielded itself via path B). This is FIFO fairness with the correctness property "no runnable task ever falls off the rq." The K4-close ``wasm32_rq`` invariants are therefore: 1. A task ``p`` is on ``wasm32_rq`` ⇔ ``p`` is runnable (``READ_ONCE(p->__state) == TASK_RUNNING`` AND the task has not exited). 2. ``wasm32_rq`` is a FIFO list ordered by enqueue time (``list_add_tail``). 3. The running task — when there is one — is at the head. ``__wasm32_yield`` peeks the head and dispatches it; on re-entry to ``__wasm32_yield``, the next peek either returns the same head (path B, no-op cycle) or a different head (path A removed prev OR a wakeup enqueued a new task ahead of prev). 4. Removal from ``wasm32_rq`` happens *exclusively* through ``dequeue_task_wasm``, which is called *exclusively* from ``deactivate_task`` along ``__schedule``'s path A (and from ``do_exit``'s scheduler teardown — also path A semantically, since the exiting task transitions to ``TASK_DEAD``). See ``docs/ARCHITECTURE.md §17.7 THIRD HISTORICAL NOTE`` for the empirical refutation that produced this design (a state-RUNNING voluntary yield inside ``create_worker``'s post-completion resume chain caused pid 1 to fall off the runqueue under the pre-R3 pop-on-dispatch model). The META-PRINCIPLE (``§20``) about all-sites enumeration applies analogously to *path* enumeration: every code path through ``__schedule`` had to be examined, not just the obvious voluntary-sleep path. R3 is the worked example for behavior-enumeration (§20.3). -------------------------------------------------------------------------------- 6. HardwareJS ↔ vmlinux idle/wakeup futex (cross-module ABI) -------------------------------------------------------------------------------- Per Q3 above. 6.1 ABI spec ~~~~~~~~~~~~ ============================== ========================================= Name ``__sched_wakeup_word`` Type ``u32`` in ``env.kernel_memory`` Initial value ``0`` Allocated by Kernel C code (``__wasm32_sched_init`` at boot) via a small bss-resident variable; the wasm-global ``__sched_wakeup_word`` is set to its offset and exported. Exposed to HardwareJS via Wasm global ``__sched_wakeup_word`` (i32, immutable, value = offset in ``env.kernel_memory``). Read-write party (kernel) Writes (clears) at top of cooperative yield loop. Reads (waits) when runqueue empty. Read-write party (HardwareJS) Writes (increments) + notifies when external event makes a kthread runnable. Never reads. Read-write party (user Worker) Same as HardwareJS via ``linux.kick_scheduler``. ============================== ========================================= 6.2 Host-side helper ~~~~~~~~~~~~~~~~~~~~ A new HardwareJS export ``HardwareJS.kickKernelScheduler()``: * Reads ``__sched_wakeup_word`` global from the vmlinux instance (once, cached at instantiation). * Performs ``Atomics.add(int32View, offset/4, 1)`` followed by ``Atomics.notify(int32View, offset/4, 1)`` on the ``env.kernel_memory``-backing ``Int32Array``. The vmlinux Worker imports this as ``linux.kick_scheduler`` for kernel C code that wants to wake the kernel from a HardwareJS- side context. At K4 the only kernel C call site is the hrtimer host-callback (when an hrtimer fires in HardwareJS JS-land, it calls ``linux.kick_scheduler`` to make the timer softirq runnable). 6.3 Idle protocol on the vmlinux side ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: void __wasm32_park_until_runnable(void) { /* Snapshot the current value, clear it, then wait if no * one has set it since the snapshot. The atomic-xchg * pattern below is the standard race-free idle-park. */ u32 v = atomic_xchg(&__sched_wakeup_word, 0); if (v != 0) { /* Someone already raised the flag. Don't park. */ return; } /* Atomically wait if the word is still 0. */ __wasm32_atomic_wait_i32(&__sched_wakeup_word, /*expected=*/0, /*timeout_ns=*/-1); } Where ``__wasm32_atomic_wait_i32`` is a thin wrapper around the wasm ``memory.atomic.wait32`` instruction (already implementable in K3 — the opcode is on the K3 atomics inventory). This is the only place in the kernel that EVER calls ``memory.atomic.wait32``. The kernel's spinlock paths use ``i32.atomic.rmw.cmpxchg`` (K3 inventory) and don't ``wait`` because spinlocks are uncontended in a single-Worker model. -------------------------------------------------------------------------------- 7. Per-task storage layout -------------------------------------------------------------------------------- At ``kthread_create`` time: :: struct task_struct *p = kmem_cache_alloc(task_struct_cache, GFP_KERNEL); p->thread.stack_base = kvmalloc(THREAD_SIZE, GFP_KERNEL); p->thread.stack_size = THREAD_SIZE; p->thread.stack_top = p->thread.stack_base + THREAD_SIZE; /* wasm stack grows DOWN, kernel-side */ p->thread.asyncify_data = kvmalloc(ASYNCIFY_DATA_SIZE, GFP_KERNEL); p->thread.asyncify_data_bytes = ASYNCIFY_DATA_SIZE; INIT_LIST_HEAD(&p->thread.rq_node); __wasm32_kthread_init_save_buffer(p, threadfn, data); Lifetime: freed on ``free_thread_stack`` (kernel/fork.c already calls this; arch_release_thread_stack is the hook). ``arch/wasm32/kernel/process.c`` (K4 introduces) provides: :: void arch_release_thread_stack(struct task_struct *p) { kvfree(p->thread.asyncify_data); kvfree(p->thread.stack_base); } Concurrency: as long as a kthread can be on the runqueue (its ``rq_node`` is linked), its storage is alive. ``kthread_stop`` ensures the kthread runs to completion before ``put_task_struct`` fires the release. -------------------------------------------------------------------------------- 8. K4 acceptance shape -------------------------------------------------------------------------------- (Copied verbatim from the K4 instructions so this doc holds the spec.) * K0–K3 dmesg progression still appears (regression). * ``rest_init`` runs to completion. * ``kthreadd`` is reachable; runs at least one full "wait for work → no work → park" loop without busy-spinning the host CPU (verified by host-side CPU usage during a 5s idle window — should be near zero, NOT pinned to 100%). * ``kernel_init`` runs as a kthread, attempts to mount rootfs and find /init, and panics cleanly with the upstream "No working init found." message (or the 6.12 equivalent string). * A vitest exercises ``kthread_create`` + ``complete`` + ``wait_for_completion`` across two kthreads (producer completes, consumer waits, scheduler runs both cooperatively) — passes. * Post-Asyncify binary size is recorded in commit body alongside the K3-instituted atomic-opcode count. * ``wasm-inspect.py`` (extended in K4) reports Asyncify globals present (``asyncify_state``, ``asyncify_data``) and a non-zero count of ``asyncify_start_unwind`` / ``asyncify_stop_rewind`` call sites inside the whitelisted region. * The K4-close commit body records ``wasm_sched_class``'s method shape as ``N real / M no-op``. ``N`` counts methods with a non-empty body that participates in the dispatcher contract (at K4: ``enqueue_task``, ``dequeue_task``, ``pick_next_task``, ``task_fork``). ``M`` counts methods with a ``WARN_ONCE`` no-op body that future work may promote to real. The count is the K6 baseline for Kconfig-deselecting unused upstream classes (``fair``, ``rt``, ``deadline``, ``ext``, ``stop``) — that optimization shrinks the binary by approximately ``5 × sizeof(struct sched_class) ≈ 5 × 112 bytes ≈ 560 bytes plus the function-body weight of each class's real methods``, which is small per-class but cumulative across the upstream class set. * The K4-close commit body records the count of arch hooks added to upstream ``kernel/sched/`` as ``H weak / H override``, where ``H`` is the number of upstream call sites at which the wasm32 arch overrides class selection or assignment. At K4 close ``H = 2`` (``arch_sched_fork``, ``arch_setscheduler_class``); see §5.1 above for the per-hook accounting. Future K-phases that surface a previously-missed ``p->sched_class`` assignment site increment ``H``, extend ``docs/ARCHITECTURE.md §17.7``, and add a new ``patches/000N-*.patch`` for the upstream weak default. This count is the trailing indicator of ``§17.7`` SECOND HISTORICAL NOTE's META-LESSON: a third refutation (``H = 3``) would mean we missed *another* site during the K4 grep-then-trust pass; if K5 ever has to bump ``H`` for ``sched_class``, that's signal to re-audit the grep methodology, not just to add the missing hook. Do NOT chase past "No working init found." K5 starts there. Tag ``v0.1.0-rc5`` at K4 close. -------------------------------------------------------------------------------- 9. Constraint on subsequent K-phases -------------------------------------------------------------------------------- K5 cannot start until this section is committed and the K5 implementation notes reference Q1–Q13 by number. K5's Regime-1 implementation (``current->mm != NULL`` parking via per-task futex word) is a strict extension of §6 above: each user task gets its own per-task ``Atomics.wait`` word in ``env.kernel_memory``, and ``try_to_wake_up`` of a user task notifies that word. The master scheduler-wakeup word ``__sched_wakeup_word`` stays the same; it's about kthread runqueue work. K5+ commits that conflict with §2 here force a separate doc commit first updating §2. K6 (the post-link Python pass for percpu / sched_class anchors per ``mm-model.rst`` §0(c)(4)) is orthogonal to the scheduler work but interacts at one point: the percpu post- link pass MUST run BEFORE Asyncify, so the Asyncify pass sees the final symbol layout. Recorded here so K6 doesn't order its own pipeline wrong.