docs: Chez stop-the-world GC vs. threads — freeze investigation
ober
09402d1d34283a45e131bab43c884eea3980a3a6
new file mode 100644 --- /dev/null +++ b/docs/chez-gc-findings.md @@ -0,0 +1,281 @@ +# Chez GC vs. threads vs. `(std net tcp)` — a freeze investigation + +Status: empirical findings + mechanism, written while building `jerboa-wormhole` +(a threaded WebSocket mailbox client/server that also does heavy bignum work +for a SPAKE2 PAKE). Reproduced repeatedly on macOS (Darwin 25.5, arm64, +`jerbuild`-bundled Chez). The conclusion is concrete and actionable; one part +of the *exact* internal sequence is labelled a hypothesis. + +## TL;DR + +`(std net tcp)` declares its `read`/`write`/`accept`/`connect` FFI **without** +`__collect_safe`, and works around blocking by putting the socket in +non-blocking mode and sleeping between `EAGAIN` retries. That is fine in +isolation, but it is **not safe to run concurrently with another thread that +allocates heavily** (i.e. triggers GC often). When a process contains both: + +1. one or more `(std net tcp)` reader threads sitting in their `c-read` + + `sleep` poll loop, and +2. one or more threads doing heavy allocation (here: 2048-bit modular + exponentiation for SPAKE2), + +the whole process **deadlocks** — every thread freezes, indefinitely (observed +> 60 s, i.e. forever). The fix is to declare the blocking socket syscalls +`__collect_safe` (and then they can even be plain blocking calls, no +non-blocking + sleep dance), exactly like `(std net thread-httpd)`, +`(std os aproc)` and `(std net tls-rustls)` already do. + +## The mechanism (what `__collect_safe` actually controls) + +This is documented in the stdlib itself. From `std/os/aproc.ss`: + +> Chez's built-in `(system cmd)` and `(open-process-ports ...)`'s subsequent +> I/O acquire the **TC mutex**, which means while one thread is parked in those +> calls every other green thread in the process is suspended. … blocking +> syscalls are declared `__collect_safe`, so **the TC mutex is released while +> the kernel does the work** — everything else keeps running. + +And from `std/net/tcp.ss` itself (lines 80-90), describing why it uses +non-blocking sockets: + +> a blocking `c-read` (a plain, non-`__collect_safe` foreign-procedure) then +> pins Chez's collector for the whole read, freezing every thread. + +So the model is: + +- Pure Scheme code on different OS threads **does** run in parallel (verified: + two threads each doing SPAKE2 `mod-expt` in a loop make simultaneous + progress). +- A **non-`__collect_safe` foreign call** keeps the thread "active" and holds + the **TC mutex** for the duration of the call. While it is held: + - the collector cannot complete a stop-the-world rendezvous (it needs that + thread to reach a safepoint / deactivate), and + - other threads that need to coordinate with the collector stall. +- A **`__collect_safe` foreign call** *deactivates* the thread first (releases + the TC mutex), lets the kernel block as long as it likes, and reactivates on + return. The collector treats a deactivated thread as already "parked", so GC + proceeds without it. + +`(std net tcp)` tries to dodge the "pins the collector for the whole read" +problem by never letting `c-read` block — the fd is `O_NONBLOCK`, so `c-read` +returns `EAGAIN` immediately and the thread then `(sleep ...)`s (sleep *is* +collect-safe). The intent is that each `c-read` only holds the TC mutex for a +few microseconds. + +That intent holds when GC is infrequent. It breaks when GC is frequent. + +## Why "brief, non-blocking `c-read`" is still not enough + +Under heavy allocation the collector runs *constantly*. Now consider the +reader thread's hot loop (`std/net/tcp.ss`, the custom-port `read!`): + +``` +(let retry () + (let ([n (c-read fd buf count)]) ; non-__collect_safe, holds TC mutex + (cond + [(> n 0) ...] + [(EAGAIN/EINTR) (sleep *retry-delay*) (retry)] ; sleep is collect-safe + [else 0]))) +``` + +Every ~10 ms the reader leaves the collect-safe `sleep`, **re-acquires the TC +mutex to run `c-read`**, then goes back to sleep. A heavily-allocating worker +thread, meanwhile, is asking for stop-the-world collections many times per +10 ms window. The two contend on the same TC mutex / collector handshake, and +the result is a circular wait that does not resolve: + +- the collecting thread holds the collection state and is waiting for the + reader to be deactivated; +- the reader is trying to take the TC mutex (or transition between + active `c-read` and deactivated `sleep`) and cannot, because the collector + side owns the contended state. + +(The precise instruction-level interleaving inside Chez's `S_collect` / +thread-deactivation path is a **hypothesis** — I did not read the Chez C +runtime. What is *certain* is the externally observable behaviour below and +that flipping the FFI to `__collect_safe` eliminates it.) + +The frequency matters, which is why this looked like a Heisenbug for a long +time: anything that injected extra collect-safe yields (an unbuffered file +`write()` per server command, a `(sleep)` per loop turn, even the `(std test)` +framework's own bookkeeping) changed the timing enough to *sometimes* dodge +the window. None of those are real fixes — they just lower the probability. + +## Reproductions (all on the same machine, all deterministic) + +1. **Heavy bignum, single thread** — fine. + SPAKE2 `mod-expt` (2048-bit modulus, ~2047-bit exponent) in the main + thread: completes in ~10 ms, never hangs. + +2. **Heavy bignum, two threads, NO sockets** — fine. + Two `fork-thread` workers each calling `spake2-start` 5× while the main + thread busy-waits on a shared counter: completes, exit 0. *Concurrent heavy + allocation by itself is not the problem.* + +3. **Two idle `(std net tcp)` reader threads + two heavy-bignum threads, same + process** — **freezes 5/5**. + Start the mailbox server, open two `rv-open` clients (each spawns a reader + thread that polls its idle socket), then have two threads each call + `spake2-start`. Both threads print a "starting" marker and never reach the + line after `spake2-start`. Killed at 15 s and at 60 s — a true deadlock, + not slowness. + +4. **Same as #3 but pure `mod-expt` with constant inputs (no crypto FFI at + all)** — **freezes**. + Rules out the ring crypto FFI (`jerboa_sha256`, `jerboa_random_bytes`); the + trigger is plain Chez bignum allocation, i.e. GC pressure, colliding with + the tcp reader threads. + +5. **Reducing the SPAKE2 exponent 2047→256 bits (≈8× less garbage)** — still + **freezes 5/5**. Lower GC frequency was not enough; the deadlock is a race, + not a function of total allocation. + +Marker traces from #3/#4 consistently show both worker threads stuck *inside* +`mod-expt`/`spake2-start` while the reader threads exist, confirming the two +ingredients. + +## Contrast: code that gets it right + +- `std/net/thread-httpd.ss` — thread-per-connection HTTP server. Declares + `accept`, `read`, `write` as `(foreign-procedure __collect_safe ...)` and + uses **plain blocking sockets**. Its own comment (lines 48-50): "mark them + `__collect_safe` so the calling thread is deactivated, letting other Scheme + threads run (including the accept loop while a worker is busy in read)." +- `std/os/aproc.ss` — `read`/`write`/`waitpid`/`system` all `__collect_safe`. +- `std/net/tls-rustls.ss` — `jerboa_tls_read`/`_write`/`_accept`/`_connect` + all `__collect_safe`. + +`(std net tcp)` and `std/os/fcntl.ss` are the outliers that rely on +non-blocking + sleep instead. + +## Recommended fix for `(std net tcp)` + +Declare the blocking syscalls `__collect_safe` and (optionally but +preferably) drop the non-blocking + `*retry-delay*` sleep machinery in favour +of plain blocking calls: + +```scheme +(def c-accept (foreign-procedure __collect_safe "accept" (int void* void*) int)) +(def c-connect (foreign-procedure __collect_safe "connect" (int void* int) int)) +(def c-read (foreign-procedure __collect_safe "read" (int u8* size_t) ssize_t)) +(def c-write (foreign-procedure __collect_safe "write" (int u8* size_t) ssize_t)) +``` + +Benefits: + +- No deadlock under concurrent allocation — a thread blocked in `read()` is + deactivated, so the collector never waits on it. +- Lower latency and lower CPU than the 10 ms poll loop (no busy `EAGAIN` + spinning, no minimum 10 ms wakeup granularity). +- Simpler code (the `O_NONBLOCK`, `EAGAIN`, `*retry-delay*` paths can go). + +Caveat: with blocking `accept`/`read`, `tcp-close` from another thread must +actually interrupt a blocked `read()` (closing the fd does on Linux/BSD/macOS; +the half-open / fd-reuse window already noted in `tcp.ss` still applies and is +worth a follow-up). + +## Workarounds for callers who can't patch the stdlib + +If you must keep `(std net tcp)`: + +- Don't run heavy-allocation Scheme on one thread while another thread holds a + `(std net tcp)` reader in the same process. In practice that means: do the + expensive crypto **before** spawning the socket reader thread, or run the + socket peer in a **separate process**. +- Or ship a private GC-safe TCP shim (a ~120-line copy of `tcp.ss`'s FFI with + `__collect_safe` on read/write/accept/connect). This is what `jerboa-wormhole` + does in `wormhole/net.ss`. + +## Second front: `condition-wait` / `mutex-acquire` are not collect-safe either + +Fixing the socket layer was necessary but **not sufficient**. With GC-safe +sockets, a second freeze remained whenever **two threads in one process both +drove socket I/O while sharing mutex-guarded state**. Concretely: a rendezvous +client with a background reader thread (filling a queue under a mutex) plus a +caller thread waiting on that queue. Two such clients in one process froze +~deterministically, even with *zero* heavy allocation (no crypto, no bignum) — +the JSON churn of the relay handshake was enough GC pressure. + +The same TC-mutex rule applies to Chez's own blocking thread primitives: + +- A thread parked in **`condition-wait`** holds/▸needs the TC coordination + state for the whole wait; if another thread triggers GC it cannot complete + the stop-the-world rendezvous → freeze. +- A thread blocked in **`mutex-acquire`** on a *contended* mutex is "active" + and not at a safepoint; a GC during that block waits for it, and if the + holder is itself waiting on the collector you get a circular wait. + +`(sleep ...)` poll loops were tried as a "GC-safe" substitute for +`condition-wait`. They lowered the rate but did **not** eliminate the freeze, +because the brief `with-mutex` critical sections (reader vs. waiter on the same +queue mutex) are still non-collect-safe windows, and two concurrent clients hit +them often enough. + +What actually fixed it: **remove the cross-thread coordination entirely.** The +mailbox protocol is request/response, so the client was rewritten to be +single-threaded — each "receive" is an inline blocking read on the (now +collect-safe) socket, buffering any unrelated frames in a thread-local list. +No reader thread, no queue mutex, no condition variable. Each connection is +owned by exactly one thread. With that plus the GC-safe socket layer, two +peers doing a full SPAKE2 exchange concurrently in one process run 6/6, 10/10, +20/20. + +Design rule that falls out of this: + +> In threaded Chez with this GC, treat **every** non-`__collect_safe` blocking +> primitive — foreign syscalls, `condition-wait`, contended `mutex-acquire` — +> as something that can wedge a stop-the-world collection. Prefer designs where +> each socket/resource is owned by a single thread and threads do **not** block +> on shared synchronization while other threads allocate. When you must block, +> block in a `__collect_safe` call. + +For the relay the same rule shows up as a *write* problem: a broadcast writes +to other peers' sockets, so two handler threads can contend on one socket's +write mutex — and if the holder is parked in the `__collect_safe` send while the +other waits on the mutex, GC deadlocks. The fix is the dual of the client's: +give each connection a single **writer thread** fed by a (wormhole queue), so +each socket's write mutex is only ever taken by one thread and never contended. +(It hid for a while because the relay allocates lightly and runs in its own +process; merely importing another module shifted the timing enough to make an +all-in-one-process test deadlock every run.) + +## Third front: even FAST foreign calls — `(std crypto native-rust)` + +The freeze is not specific to *blocking* syscalls. `(std crypto native-rust)`'s +ring bindings (`jerboa_sha256`, `jerboa_hmac_sha256`, `jerboa_chacha20_*`, +`jerboa_random_bytes`, …) are short, non-blocking CPU functions — and they are +declared **without** `__collect_safe`. They still wedge GC: each call holds the +TC mutex for its (brief) duration, and when two threads make *many* such calls +while allocating (a SPAKE2 PAKE: random → sha256 → bignum → chacha20/hmac for +the key-confirmation phase, ×2 peers), the collector keeps trying to stop the +world and keeps finding a thread mid-FFI. Empirically: a single `random-bytes` +per peer was fine; the full crypto sequence froze ~deterministically. + +Marking those FFIs `__collect_safe` fixes it (a private GC-safe copy of the +needed bindings lives in `jerboa-wormhole/wormhole/crypto-native.ss`). The +deactivate/reactivate cost is negligible for short calls. + +Recommendation: `(std crypto native-rust)` should declare its +`foreign-procedure`s `__collect_safe` (or ship a `*-safe` variant). The ring +calls are pure and operate only on the passed bytevectors, so they are safe to +run while the collector moves *other* objects; Chez locks the `u8*` bytevector +arguments for the duration of the call (the same reason `(std net thread-httpd)` +can pass bytevectors straight to a `__collect_safe` `read`/`write`). + +Generalised rule: in threaded Chez, **any** non-`__collect_safe` foreign call — +blocking or not — that runs concurrently with allocating threads is a latent +freeze. If a foreign binding will ever be called from a worker thread in a +multi-threaded process, declare it `__collect_safe`. + +## Why this was painful to localise + +- Output is lost on `SIGTERM`/timeout kill because Chez block-buffers ports + when stdout/stderr is a pipe; only **unbuffered file ports** + (`(buffer-mode none)`) survived the kill and revealed where each thread + stopped. +- The `(std test)` framework wraps each case body in nested `guard`s and does + its own allocation; that perturbed GC timing enough to flip the bug on/off, + which falsely implicated the test framework. Networked, multi-thread tests + are more reliable as plain scripts. +- Sub-millisecond `(sleep ...)` "yields" do not reliably deactivate the thread + the way a real blocking/collect-safe call does, so they are not a fix.