chez-gc-findings: collect-safe FFI doesn't isolate co-tenant heaps
ober
df55de05480b222e445edb078f7b2e97ac25e376
--- a/docs/chez-gc-findings.md +++ b/docs/chez-gc-findings.md @@ -279,3 +279,50 @@ multi-threaded process, declare it `__collect_safe`. are more reliable as plain scripts. - Sub-millisecond `(sleep ...)` "yields" do not reliably deactivate the thread the way a real blocking/collect-safe call does, so they are not a fix. + +## Follow-up: collect-safe is necessary but not sufficient — heavy-allocation co-tenants + +A later finding from `jerboa-wormhole`'s transit relay. Every blocking syscall +in play was already `__collect_safe` (sockets via `wormhole/net.ss`, ring crypto +via `wormhole/crypto-native.ss`), and `(sleep ...)` is collect-safe — yet a +*second* server sharing the process still destabilised the first. + +The mailbox relay (a latency-sensitive WebSocket request/response server) and +the transit relay (a throughput server that pairs two sockets and pipes the bulk +file bytes between them) were started in the **same process** by `wormhole +relay`. Measured byte-identical file-transfer success over a localhost relay: + +| configuration | success | +|------------------------------------------------|---------| +| direct transit, no relay involved | 20/20 | +| relay path, relay **co-hosted** with mailbox | ~6/10 | +| relay path, relay in a **separate** process | ~9/10 | +| relay-only (whole file piped), **co-hosted** | ~7/10 | + +The failure is always the *mailbox* connection dropping ("connection closed +waiting for phase …") at an unrelated moment — not the transit connection. The +mechanism: the transit relay's pipe loop allocates a fresh bytevector per chunk +(`get-bytevector-some`), so piping a multi-hundred-KB file is a burst of +allocation. That allocation triggers GC *in the shared heap*, and a mailbox +reader thread momentarily between collect-safe points (e.g. parsing a frame, +mid-`ws-recv`) makes the stop-the-world pause long enough that a peer read +returns short; `ws-recv` treats any read glitch as EOF and closes the socket. + +So: **collect-safe FFI keeps a single subsystem from deadlocking, but it does +not isolate two subsystems that share a heap.** A high-allocation-rate worker +(a byte pump, a compressor, a bulk hasher) co-resident with a +latency-sensitive request/response server will inject GC pauses into the latter. +Two robust options: + +1. **Process isolation** — run the throughput server as its own OS process so it + has its own heap and its GC pauses can't touch the latency-sensitive one. + This is what `jerboa-wormhole` settled on: `wormhole relay` is mailbox-only, + the transit relay is a separate `wormhole transit-relay`, and it is opt-in. +2. **Allocation discipline in the hot path** — reuse a single buffer for the + pipe instead of allocating per chunk, to cut the GC trigger rate. Helps, but + does not fully decouple the two subsystems' GC the way (1) does. + +Forking the throughput child *early* (before any threads exist) is the cheap way +to get (1) from one launcher; forking a process that already has worker threads +is unsafe (only the forking thread survives, and any lock another thread held is +frozen in the child).