brintos

brintos / hardwarejs public Read only

0
0

NaaS L3 uplink: auto-reconnect after a dropped relay WebSocket (one drop = dead guest networking until relaunch) #8

Closed jdbrinton opened this issue · 2 comments
jdbrinton jdbrinton commented

[filed by the maintainer's AI agent while debugging linux-in-the-browser#94 fallout]

WebSocketPacketNaasUplink (src/devices/web_socket_packet_uplink.ts) tears down on WS close and never reconnects — after any relay-side drop (deploy roll of naas-l3-production has a documented brief gap; network blips; and, until today's relay fix, any full-size guest TCP segment) the guest keeps its interface but every packet goes to sDrops forever. The user-visible shape: "[naas] VPN disconnected: relay connection closed", then every later network command (ssh, wget) silently hangs until the machine is relaunched.

Witnessed live 2026-07-15: the relay's old WS read limit (mtu+64) 1009-failed the session on the first ~1500-byte upload segment (openssl 3.5 TLS ClientHello via the new ports wget); the uplink then stayed dead, which is why ssh -T git@github.com hung on machines where wget had already killed the VPN. The relay-side trigger is fixed (brintos-web 1d3ebcb: 64 KiB read limit + bidirectional TCP MSS clamping to mtu-40, RED-flip regression test in 8e57f0e), but the uplink should still self-heal.

Sketch: on non-closedByUs close, reconnect with expo backoff + jitter, re-HELLO with a fresh token (the mint callback already exists — opts.token is a function), resume forwarding on HELLO_OK. The assigned egress IP will change (existing flows RST/timeout end-to-end — acceptable, same as a real VPN rebind). Emit "[naas] L3 uplink: reconnecting (attempt N)" so the state is visible. A REFRESH-style session resume that keeps the assigned IP would be a nicer follow-on but needs relay-side session affinity.

jdbrinton jdbrinton commented

Triage (agent for joel, 2026-07-15): accepted, in progress. Plan: reconnect loop in WebSocketPacketNaasUplink with exponential backoff + jitter (0.5s → 30s cap, unlimited while the device is attached), full session re-handshake on each attempt, guest tx dropped (logged once per outage) while down, [naas] console lines for reconnecting/reconnected, and deliberate teardown distinguished from unexpected close so machine shutdown doesn't spawn a zombie reconnector.

jdbrinton jdbrinton commented

Implemented (agent for joel, 2026-07-15), 65c0930 on joel-dev, pushed. As triaged: exponential backoff + jitter (0.5s→30s cap, unlimited attempts while attached), fresh token mint per re-HELLO via the opts.token callback, forwarding resumes on HELLO_OK with the attempt counter reset. Extras: relay ERROR frames force-close our half so those outages self-heal with a fresh token too; a socket-generation counter fences stale events from superseded sockets; deliberate close() cancels pending retries (including mid-backoff) and never reconnects; tx-drop + disconnect notices are once-per-outage; token-mint failure on open no longer strands an open never-HELLOed socket. 10 new unit tests against a fake WS with fake timers cover the handshake, backoff ladder, fencing, ERROR self-heal, and stale-close immunity.

Known follow-on (noted in the commit): a REFRESH-style session resume that keeps the assigned egress IP across reconnects needs relay-side affinity — new session per reconnect for now, which restores connectivity but not existing TCP flows. Goes live with the next site deploy. Closing; reopen if a live relay roll still strands a guest after the deploy.

jdbrinton closed this as completed