522 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3================================================================================4arch/wasm32 memory model5================================================================================6 7:Status: K0 (build glue). Stable for K0–K2. Revisited at K3 (mm_init).8 9This document is the source of truth for how ``arch/wasm32`` maps Linux's10kernel/user split onto WebAssembly. Two related questions are answered here:11 121. *Where does kernel memory live, and where does user memory live, at the13 wasm-module level?* (Section 1.)142. *What page-table shape do we claim to have, given that wasm has no hardware15 MMU?* (Section 2.)16 17Both questions are answered deliberately, not as a side-effect of which18``asm-generic`` headers shut clang up fastest. Changing either model is a19breaking change to the arch ABI.20 21--------------------------------------------------------------------------------221. Two-memory model23--------------------------------------------------------------------------------24 25The cross-module contract — the arch ABI proper — is expressed in **names**,26not in memory indices. Indices are local to each module's import order and27will differ between vmlinux and userspace binaries by design.28 29There are exactly two named ``WebAssembly.Memory`` objects in the system:30 31.. list-table::32 :header-rows: 133 :widths: 22 18 14 4634 35 * - Import name36 - Kind37 - Cardinality38 - Description39 * - ``env.kernel_memory``40 - shared (``WebAssembly.Memory({shared: true})``)41 - **1** for the entire system42 - The kernel's address space. Holds kernel ``.text`` / ``.rodata`` /43 ``.data`` / ``.bss``, the kernel heap (slab, vmalloc area), per-task44 kernel stacks, page-table arrays, virtq rings, signal-pending words,45 and every other Linux datum that crosses Worker boundaries. Imported46 by every wasm module instance in the system — every userspace47 ``.wasm`` instance AND every per-Worker instance of ``vmlinux.wasm``.48 * - ``env.user_memory``49 - shared (``WebAssembly.Memory({shared: true})``)50 - **1 per process Worker** (plus a 1-page sentinel for the51 standalone vmlinux Worker, see §1.1)52 - That Worker's userspace pages: user ``.text`` / ``.rodata`` / ``.data`` /53 ``.bss``, user heap, user stack, mapped files. Imported by the54 userspace ``.wasm`` running in that Worker, AND by the kernel55 ``.wasm`` instance running in that Worker (so that kernel code56 executing on behalf of this process can do uaccess against the57 calling process's user memory).58 59 **Why ``shared: true`` for what the model calls a "per-process60 private" memory.** "Private" here is an address-space property61 (no other process has a window into this Worker's user pages),62 not a SharedArrayBuffer-sharability claim. The backing SAB has63 to be visible to (a) the running userspace ``.wasm`` instance,64 (b) the kernel ``vmlinux.wasm`` instance in the same Worker,65 and (c) the main-thread syscall router that resolves syscall66 buffers without having to bounce through a wasm export. (a)67 and (b) live in the same Worker so they would share a68 non-shared Memory too; (c) requires SAB-backing. Additionally,69 V8 (and SpiderMonkey) reject ``WebAssembly.instantiate`` with70 "mismatch in shared state of memory" if the wasm module's71 import declaration says ``shared`` and the JS-side Memory is72 not — and since the post-link splice73 (``scripts/wasm-add-memory.py``) declares this import as74 ``shared``, every JS-side allocation MUST be ``shared: true``75 too. K1 discovered this the hard way; the spec now matches76 reality.77 78Two consequences worth pinning here, before talking about indices:79 80* HardwareJS hands the **same** ``env.kernel_memory`` JS object to every81 Worker at instantiation. That is what gives Linux real SMP semantics82 against shared memory in this port: per-CPU variables, spinlocks, RCU,83 IPIs all carry their full upstream meaning because the wasm engine84 enforces the cross-thread memory model on the underlying85 ``SharedArrayBuffer``.86 87* HardwareJS allocates a **fresh** ``env.user_memory`` per process Worker.88 The kernel never directly addresses *another* process's user memory; a89 ``copy_to_user`` happens only while the kernel is executing in *this*90 Worker, where ``env.user_memory`` is bound to *this* process's userspace91 pages.92 93What each module's local memory indices end up being is a derived fact:94 95.. list-table::96 :header-rows: 197 :widths: 32 34 3498 99 * - Module100 - Memory index 0101 - Memory index 1102 * - Userspace ``.wasm``103 - ``env.user_memory`` (own)104 - ``env.kernel_memory``105 * - Kernel ``vmlinux.wasm`` instance running in a process Worker106 - ``env.kernel_memory`` (own)107 - ``env.user_memory`` (that Worker's user pages)108 * - Kernel ``vmlinux.wasm`` instance running in the vmlinux Worker109 (kthreads only — see §1.1)110 - ``env.kernel_memory`` (own)111 - ``env.user_memory`` bound to a 1-page sentinel — see §1.1112 113This pattern is just "``memory 0`` is whichever memory the module's own114compiled ``.data``/``.bss``/stack land in (because that is the memory115``wasm-ld`` emits default loads/stores against), ``memory 1`` is the116other side." It is not a special rule; it is what falls out once you117accept that **names are the ABI, indices are an artefact of link order**.118 119Cross-module portability of source code follows directly. The address-space120attributes used in ``uaccess.S`` are written from the *kernel* module's index121space (because they describe kernel→user / user→kernel copies); the same122copies *cannot* be expressed from userspace because userspace never crosses123into kernel memory in the source language — only via syscall traps which124HardwareJS dispatches without a ``memory.copy``.125 1261.1 Per-Worker kernel instantiation and the kthread sentinel127~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~128 129Every Worker — every process Worker and the dedicated vmlinux Worker —130holds its **own** ``WebAssembly.Instance`` of ``vmlinux.wasm``. They share131exactly what they import. So:132 133* All instances import the same ``env.kernel_memory`` and therefore see134 identical ``current``, ``init_mm``, runqueues, slab arenas, kernel135 ``__bss``, etc., subject to the wasm memory model's atomic-access rules.136* Each instance has its own ``__stack_pointer`` global, its own non-shared137 globals, its own Asyncify control state. This is what makes the138 "a process Worker IS a CPU" identity literally true at the wasm level:139 each Worker is a distinct execution context with its own register-state140 analogue, executing against shared kernel memory.141 142The **vmlinux Worker** is the sole Worker without a userspace process143in it. It hosts only kthreads (``kthreadd``, ``ksoftirqd``, ``kworker``,144the idle thread, and so on). It must still satisfy the ``env.user_memory``145import to keep ``vmlinux.wasm``'s import shape uniform; calling into146the kernel from kthread context with no user memory bound is otherwise147an invariant that the wasm loader cannot express. HardwareJS therefore148hands this Worker a **1-page, ``shared:true``-backed sentinel Memory**149under the same ``env.user_memory`` import field name. Its only purpose150is to fault loudly if anything ever tries to use it. Any151``copy_to_user`` / ``copy_from_user`` issued from kthread context is a152kernel bug; the sentinel ensures the bug materialises as a wasm trap153(out-of-bounds at offset > 65535) on the first cross-memory154``memory.copy`` rather than as silent corruption of an unrelated155process's pages.156 157**Why the same import name instead of ``env.user_memory_sentinel``?**158A wasm module declares each import once at compile time; we cannot159swap import names per Worker class. The kernel's ``vmlinux.wasm``160binary has a single ``env.user_memory`` import slot, and HardwareJS161chooses *what to bind to it* at instantiation time: a real process's162user pages for process Workers, the 1-page sentinel for the vmlinux163Worker. The distinction is in the binding (a HardwareJS-side decision164keyed off "is this Worker hosting a userspace process or running165kthreads?"), not in the import name.166 167**Why ``shared:true`` for the sentinel?** Same reason as for real168``env.user_memory`` (see §1 table): V8/SpiderMonkey require the JS-side169Memory's ``shared`` flag to match the wasm module's import declaration,170and that declaration is fixed to ``shared`` by the171``scripts/wasm-add-memory.py`` splice. The sentinel is 1 page, owned172by exactly one Worker, never handed elsewhere — its SAB-backing is173inert.174 1751.2 Syscall entry sequence (kernel-stack swap)176~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~177 178A kernel stack is per ``task_struct``, slab-allocated out of179``env.kernel_memory`` exactly like upstream Linux. The wasm runtime's180notion of "the stack" is a single ``__stack_pointer`` global per181instance; the kernel must therefore swap that pointer at every182kernel-mode entry and exit.183 184The required dance on a syscall trap from userspace into the kernel185module instance running in the same Worker:186 1871. Userspace ``.wasm`` calls the ``linux.syscall`` import.1882. HardwareJS calls the kernel instance's exported189 ``wasm_syscall(nr, a0..a5)``. This crosses no Worker boundary; the190 kernel instance lives in the same Worker as the userspace instance.1913. ``wasm_syscall`` is implemented in ``arch/wasm32/kernel/entry.c``.192 Before any C code runs, it reads ``current->stack`` from193 ``env.kernel_memory`` and writes it into the instance's194 ``__stack_pointer`` global; saves the old (userspace) value as a195 per-task field. From this point all C locals and saved registers196 land in this task's kernel stack in ``env.kernel_memory``.1974. ``do_syscall(nr, a0..a5)`` runs as in upstream Linux.1985. On return, the saved userspace ``__stack_pointer`` is restored and199 control returns through the import to the userspace ``.wasm``.200 201Without that swap, two Workers concurrently entering the kernel would202each write C locals into the *userspace* stack of their own process —203which is private memory, so they wouldn't clobber each other directly,204but they also wouldn't be writing anywhere the kernel can later find205them. Locks, ``schedule()``'s stack-snapshotting, the entire kernel206stack-walk and unwind machinery, and Asyncify's saved-state pointer all207assume the running C code is using a slab-allocated stack in208``env.kernel_memory``. The swap is non-negotiable.209 210The kthread-context path (vmlinux Worker) does not swap from userspace211into kernel; it always runs against the kernel stack of whichever212kthread is currently scheduled. The Asyncify-driven ``switch_to`` (see213``docs/ARCHITECTURE.md`` §17.3) saves the outgoing kthread's stack214pointer into its ``task_struct`` and loads the incoming one.215 2161.3 Cross-memory boundary copies217~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~218 219Every ``copy_to_user`` / ``copy_from_user`` (and the ``get_user`` /220``put_user`` family) is implemented as a wasm multi-memory221``memory.copy`` from the kernel module's index space. From the kernel222instance's perspective (``memory 0 = env.kernel_memory``,223``memory 1 = env.user_memory``):224 225* ``copy_to_user(dst_user, src_kernel, n)`` →226 ``memory.copy 1, 0`` (dst=user, src=kernel)227* ``copy_from_user(dst_kernel, src_user, n)`` →228 ``memory.copy 0, 1`` (dst=kernel, src=user)229* ``clear_user(dst_user, n)`` →230 ``memory.fill 1`` (target=user)231 232Each translates to one wasm instruction (opcode prefix ``0xfc``, opcode233``0x0a`` for copy, ``0x0b`` for fill, then ULEB128 memory indices). They234are NOT ``memcpy``. A ``memcpy`` that pretends both pointers live in the235same linear memory will silently read garbage the first time a real236syscall argument arrives.237 238The primitives live in ``arch/wasm32/lib/uaccess.S`` because clang's C239front-end cannot today lower address-space-annotated pointer copies to240multi-memory ``memory.copy`` (LLVM 22 crashes in instruction selection241on ``addrspacecast``). The LLVM **assembler**, however, accepts242``memory.copy <dst>, <src>`` and ``memory.fill <dst>`` syntax and emits243the correct opcode bytes; ``wasm-ld`` preserves those bytes unchanged244through linking.245 246Userspace binaries never emit cross-memory copies. A userspace process247that needs to ``read()`` from the kernel does so via the syscall trap,248which is structurally a ``call`` to the ``linux.syscall`` import; the249kernel side does the cross-memory copy on the userspace's behalf.250 251Toolchain note: wasm-ld single-memory limitation252~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~253 254LLVM 22's ``wasm-ld`` does not (yet) emit multi-memory modules: it255preserves ``memory.copy 0, 1`` opcode bytes in the CODE section but256only writes a single entry in the MEMORY/IMPORT section. The resulting257module is structurally invalid (``[parse exception: invalid memory258index]``).259 260``linux-wasm/scripts/link-vmlinux-wasm.sh`` therefore runs a small261post-link rewriter (``linux-wasm/scripts/wasm-add-memory.py``) on every262produced wasm artifact to splice in the second memory as an imported263memory entry. The rewriter is ~280 lines of self-contained wasm-format264code; the wasm spec is final, V8 / SpiderMonkey / Wasmtime all accept265the result, and the rewriter goes away when ``wasm-ld`` grows native266multi-memory linking. Nothing about the kernel source has to change267at that point. This is NOT shortcut debt.268 269The same rewriter is reused by ``wackywasm-tools/bin/wasmld`` for270userspace binaries — they import ``env.kernel_memory`` as their second271memory.272 273--------------------------------------------------------------------------------2742. Page-table model275--------------------------------------------------------------------------------276 277WebAssembly has no hardware MMU. A wasm linear memory is a flat,278byte-addressed array of bytes; there are no page tables, no TLB, no279faults, no PROT_* bits the hardware enforces. From the kernel's point280of view, calling ``set_pte()`` on wasm32 cannot stop ``memory.load``281from succeeding at that address.282 283What Linux still needs from the arch even on flat memory is a284*bookkeeping data structure* per address space, so that ``do_mmap``,285``do_munmap``, ``vm_area_struct`` linkage, ``copy_page_range`` on fork,286and the rmap / OOM killer accounting all have something to walk. The287wasm32 page tables are exactly that: a degenerate single-level array288of bookkeeping ``pte_t`` entries per process, with PUD and PMD folded289away.290 291Levels claimed292~~~~~~~~~~~~~~293 294::295 296 PGD ──> (folded PUD) ──> (folded PMD) ──> PTE ──> physical page297 298 CONFIG_PGTABLE_LEVELS = 2299 pgtable-nopud.h — folds PUD into PGD300 pgtable-nopmd.h — folds PMD into PUD (== PGD because of nopud)301 302Both fold headers are pulled in via ``arch/wasm32/include/asm/pgtable.h``303explicitly, NOT via ``generic-y``, so the level shape is reviewable here304in one place.305 306Page granularity307~~~~~~~~~~~~~~~~308 309::310 311 PAGE_SHIFT = 12 (4 KiB, matches Linux's everywhere-default)312 PAGE_SIZE = 4096313 PAGE_MASK = ~(PAGE_SIZE - 1)314 315A wasm linear memory page is 64 KiB. We do not export the wasm page316size to the rest of the kernel; we use 4 KiB everywhere and let317``memory.grow`` allocate sixteen Linux pages per wasm page. ``mm_init``318will see this as "physical memory comes in 16-page contiguous chunks"319and the buddy allocator handles that case natively.320 321PTE shape322~~~~~~~~~323 324``pte_t`` is a single ``unsigned long``. It encodes:325 326* bits 12+ — page-frame number into the per-process ``env.user_memory``327 of this VMA's owning Worker. Because ``env.user_memory`` is flat,328 PFN == byte offset >> 12.329* bits 0..11 — software-only flags: ``_PAGE_PRESENT``, ``_PAGE_USER``,330 ``_PAGE_RW``, ``_PAGE_DIRTY``, ``_PAGE_ACCESSED``. These are331 advisory; the wasm runtime cannot enforce them. Real protection332 comes from the Worker boundary (a process Worker cannot reach into333 another Worker's ``env.user_memory`` at all, regardless of PTE334 flags).335 336The fact that protection is enforced by the Worker boundary rather337than by PTE bits is what makes "everything in flat memory is technically338accessible to the running wasm code" not a security hole: the running339wasm code is one process, and there is no other process in the same340address space to protect.341 342Address space size343~~~~~~~~~~~~~~~~~~344 345``env.user_memory`` is a wasm32 linear memory: max 4 GiB. We cap346``TASK_SIZE`` at 3 GiB, leaving the top 1 GiB of the wasm32 address347range as a guard / future kernel-virtual-mapping window. ``PAGE_OFFSET``348is therefore ``0xc0000000``; addresses ≥ ``PAGE_OFFSET`` in a user349pointer mean "this is a kernel pointer that someone passed to350userspace by mistake, EFAULT."351 352(Inside the kernel, kernel data structures are addressed by their353natural offsets into ``env.kernel_memory``. The "kernel virtual354address space" Linux normally talks about is collapsed into "indices355into env.kernel_memory", which has its own 4 GiB range; we never have356to translate between the two.)357 358Fork / COW359~~~~~~~~~~360 361``fork`` cannot use copy-on-write because wasm has no page faults: the362child Worker's ``env.user_memory`` is allocated fresh and the parent's363data is bulk-copied into it at clone time. The PTE bookkeeping in364``copy_page_range`` still runs so that the child's mm accounting agrees365with the buddy allocator, but ``pte_mkwrite`` / ``pte_wrprotect`` are366advisory.367 368This is the only place the model loses anything significant compared369to a real MMU arch. K3 will revisit whether to expose a370"snapshot+lazy-copy" primitive from HardwareJS that gets the wasm371engine to deduplicate physical pages until first write; if so, COW372becomes a HardwareJS-level optimization under an unchanged PTE373bookkeeping layer.374 375Acceptance of this model376~~~~~~~~~~~~~~~~~~~~~~~~377 378This model is sufficient for K0–K2 (build, head.c, setup_arch). It379commits to a single-level real PTE table at K3 (mm_init). Changes to380the level count, page size, PTE bit layout, or PAGE_OFFSET after381v0.1.0 are breaking changes to the arch ABI and require a382corresponding bump of the ``wasm.linux.exec`` ``abi_version``.383 384--------------------------------------------------------------------------------3853. Per-CPU data layout386--------------------------------------------------------------------------------387 388Linux's per-CPU machinery uses two abstractions that this port has to389reproduce against ``wasm-ld``'s linker-script-less layout:390 3911. **Static per-CPU image:** a contiguous block of bytes in the linked392 binary, bracketed by linker symbols ``__per_cpu_start`` and393 ``__per_cpu_end``, containing the initial values of every variable394 declared with ``DEFINE_PER_CPU``. ``__per_cpu_load`` is the address395 from which ``pcpu_embed_first_chunk`` reads when copying the image396 into each CPU's runtime slot — on every other arch, the linker397 script aliases ``__per_cpu_load`` onto ``__per_cpu_start``.398 3992. **Dynamic per-CPU allocator:** ``__alloc_percpu`` requests are400 served out of a runtime chunk allocated by401 ``setup_per_cpu_areas`` at boot. The dynamic area starts402 immediately after the static area within each CPU's slot, sized403 per CPU as404 ``unit_size = roundup(static_size + reserved_size + dyn_size,405 PAGE_SIZE)``.406 407The generic ``mm/percpu.c::pcpu_build_alloc_info()`` derives408``static_size = __per_cpu_end - __per_cpu_start``. On wasm32, that409subtraction does NOT yield the static extent — see §17.7 EIGHTH410HISTORICAL NOTE for the empirical refutation and the cross-tool411layout interaction that breaks it.412 413Quick summary of the mechanism:414 415* ``wasm-ld`` with ``--no-merge-data-segments`` emits data segments416 in input order (across input objects, and within an object in417 source declaration order). It does NOT sort by section name.418 419* ``arch/wasm32/built-in.a`` is extracted alphabetically first from420 ``vmlinux.a``; within it, ``percpu_anchors.o`` is the first421 member to define ``.data..percpu*`` segments.422 423* ``percpu_anchors.c`` declares ``__per_cpu_start`` (section424 ``.data..percpu..aa_start``) and ``__per_cpu_end`` (section425 ``.data..percpu..zz_end``). Both emit FIRST and adjacent, BEFORE426 any other compilation unit's ``.data..percpu`` data. So427 ``&__per_cpu_end - &__per_cpu_start`` is one byte (or padded428 thereto by alignment), wildly understating the actual extent.429 4303.1 Wasm32-specific solution (K4)431~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~432 433Two pieces, each slotted against an upstream-documented extension434point:435 436(a) ``__per_cpu_load`` aliased to ``__per_cpu_start`` via437 ``__attribute__((alias("__per_cpu_start")))`` in438 ``percpu_anchors.c``. This matches what every other arch's439 linker script does with the ``__per_cpu_load = .`` line just440 before ``__per_cpu_start = .``.441 442(b) ``CONFIG_HAVE_SETUP_PER_CPU_AREA=y`` (set in443 ``arch/wasm32/Kconfig``) replaces the generic444 ``mm/percpu.c::setup_per_cpu_areas`` with a wasm32-private445 implementation in ``arch/wasm32/kernel/percpu.c``. The446 wasm32 version constructs ``pcpu_alloc_info`` directly via447 ``pcpu_alloc_alloc_info()`` and hardcodes ``static_size`` to448 ``WASM32_STATIC_PCPU_SIZE = 128 KiB``. The actual defconfig449 extent is on the order of a few KiB; 128 KiB is generous450 head-room. ``__per_cpu_offset[]`` is defined locally in this451 same file (the generic definition in ``mm/percpu.c`` is452 suppressed when ``HAVE_SETUP_PER_CPU_AREA=y``).453 454The two pieces compose: ``__per_cpu_load`` aliasing fixes the455memcpy source address; the custom ``setup_per_cpu_areas`` fixes456the static-size and offset computation.457 4583.2 Cost459~~~~~~~~460 461Each per-CPU slot has ``128 KiB - actual_extent`` of unused462memory in its static-image area. With ``NR_CPUS=8`` and the463defconfig static extent (~few KiB), worst case is roughly 1 MiB464of unused-but-reserved RAM. We have 64 MiB of kernel RAM at K4.465Acceptable.466 4673.3 Future cleanup468~~~~~~~~~~~~~~~~~~469 470The proper fix is a post-link Python pass that walks the linked471wasm binary's ``.data..percpu*`` data segments, computes the472actual highest address reached, and patches ``__per_cpu_end``'s473exported value (or rewrites ``setup_per_cpu_areas`` to read the474computed size from a wasm global emitted by the pass). At that475point:476 477* ``percpu_anchors.c`` can drop its custom anchor sections and478 the ``__per_cpu_load`` alias.479 480* ``arch/wasm32/kernel/percpu.c`` can be deleted; the generic481 ``mm/percpu.c::setup_per_cpu_areas`` takes over.482 483* ``CONFIG_HAVE_SETUP_PER_CPU_AREA`` can be unset.484 485This is captured as a §15 row in the same family as the486``vmlinux.lds.S`` post-link Python pass that covers487``__setup_start`` / ``__initcall_start`` / etc. — same root488cause (wasm-ld doesn't consume linker scripts) and the same489fix retires multiple symptomatic rows simultaneously.490 491--------------------------------------------------------------------------------492Appendix A — Per-module index tables (derived, non-normative)493--------------------------------------------------------------------------------494 495The names in §1 are the ABI. The indices below are what each module496ends up with after ``wasm-ld`` + the ``wasm-add-memory.py`` post-link497splice; they are reviewable here as a debugging aid but they are NOT498the spec — if a future toolchain or splice order changes them, the499spec is unchanged so long as the import NAMES still match §1.500 501Userspace ``.wasm`` (e.g. ``/init``, ``/bin/bash``, ``/bin/ls``)::502 503 memory 0 ← env.user_memory (wasm-ld default, declared by --import-memory=env,user_memory)504 memory 1 ← env.kernel_memory (spliced in by wackywasm-tools/bin/wasmld505 via scripts/wasm-add-memory.py)506 507Kernel ``vmlinux.wasm`` (one binary, instantiated per Worker)::508 509 memory 0 ← env.kernel_memory (wasm-ld default, declared by --import-memory=env,kernel_memory510 in scripts/link-vmlinux-wasm.sh)511 memory 1 ← env.user_memory (spliced in by scripts/wasm-add-memory.py during512 the kernel link; same import name in every513 Worker. HardwareJS binds this name at Worker514 instantiation time: process Workers get the515 process's own user pages; the vmlinux Worker516 gets the 1-page sentinel.)517 518Changing the splice order — for example, splicing the second memory519*before* wasm-ld's default in some future refactor — flips the520indices but does not change the ABI. ``uaccess.S`` would need to be521updated to match, the doc would not.522