fcntl(fd, F_DUPFD, n) (and F_DUPFD_CLOEXEC) returns ENOSYS (-38) for a file descriptor that refers to a non-console kernel device — e.g. a /dev/pts/* slave or /dev/zero — while dup(fd) and dup2(fd, n) on the same descriptor work. F_DUPFD works only when fd is the console.
In stock Linux F_DUPFD is handled generically in do_fcntl() (f_dupfd() → alloc_fd()), independent of the backing file, so any real fd should support it.
Reproduction
A small wasm32-musl probe (built with the platform toolchain), run from the interactive init shell:
fcntl(2, F_DUPFD, 7) = 7 errno=0 (fd 2 = console)
fcntl(0, F_DUPFD, 7) = 7 errno=0 (fd 0 = console)
fcntl(0, F_DUPFD, 1) = 4 errno=0 (fd 0 = console)
open("/dev/zero") = 0
fcntl(0, F_DUPFD, 7) [zero fd] = -1 errno=38 (ENOSYS) <-- BUG
fcntl(0, F_DUPFD, 1) [zero fd] = -1 errno=38 (ENOSYS) <-- BUG
dup2(0, 1) [zero fd] = 1 errno=0
dup(0) [zero fd] = 1 errno=0
The same holds for a /dev/pts/* slave opened onto a low fd: fcntl(F_DUPFD) → ENOSYS, dup/dup2 → OK.
Root cause
The worker's fcntl routing (worker.mjs, the "D2-FCNTL" block) routes fcntl kernel-side only for fd >= 3 or a low fd already recorded in kernelRoutedLowFds (populated when a kernel fd is dup2'd onto a low fd). A descriptor obtained by a fresh open() that lands on a low fd (0/1/2) — e.g. after a process closes 0/1/2 and opens a pts slave, which lands on fd 0 — is a real kernel fd but is not in kernelRoutedLowFds, so its fcntl falls to the SAB-RPC fallback, which has no fcntl handler and returns -ENOSYS. dup/dup2 are separate syscalls that are routed correctly, which is why they work on the identical fd.
Impact
This breaks Expect (spawn/expect) on musl. Expect's child (exp_getptyslave) closes fds 0/1/2, open()s the pts slave (which lands on fd 0), then wires it onto stdout/stderr with fcntl(0, F_DUPFD, 1) / fcntl(0, F_DUPFD, 2). Both return ENOSYS, so the exec'd child's fd 1 is never wired to the pty and it dies immediately (cat: standard output: Bad file descriptor). More generally, any program that fcntl(fd, F_DUPFD, n)s a freshly-opened non-console fd is affected.
Suggested fix
Route the generic fcntl commands (at least F_DUPFD / F_DUPFD_CLOEXEC, and arguably F_GETFL/F_SETFL/F_GETFD/F_SETFD) kernel-side whenever perProcessTaskAddr !== 0, regardless of the fd number — the kernel sys_fcntl ([72], HWJS_NATIVE) handles all fds including the console tty. That removes the dependency on kernelRoutedLowFds tracking a fresh open() and mirrors how dup/dup2 already behave. (I did not attach a worker.mjs branch — I don't have a pushable host checkout, and the routing heuristic exists for reasons I did not want to change untested.)
Workaround (client-side)
For Expect specifically, rewriting the two child-side pty-slave dups from fcntl(0, F_DUPFD, 1/2) to dup2(0, 1) / dup2(0, 2) (semantically identical, since the target fds were just closed) fully unblocks spawn/expect; dup2 routes kernel-side. This is a viable per-application workaround but does not fix the underlying F_DUPFD gap.