docs/aproc-roadmap: plan future iterations on (std os aproc)
ober
db6d1c98972b7d1e546cd30b8b5a8b3bf7ab5039
new file mode 100644 --- /dev/null +++ b/docs/aproc-roadmap.md @@ -0,0 +1,405 @@ +# `(std os aproc)` — Roadmap + +`(std os aproc)` ships in v1 as a small async-subprocess facility built +on `__collect_safe` libc primitives. It solves one concrete problem: +Chez's built-in `(system cmd)` and port I/O pin the TC mutex for the +entire subprocess lifetime, freezing every other green thread. v1 +releases the mutex while parked in the kernel so TUIs, watchdogs, and +streaming loops keep running. + +This roadmap covers everything else. + +## v1 state (today) + +API surface (exported): + +``` +aproc-spawn cmd → handle +aproc-handle? x → bool +aproc-pid h | aproc-stdin-fd h | … → int +aproc-exit-code h → int | #f +aproc-read h fd [count] → bytevector | #!eof +aproc-read-stdout h [count] → bytevector | #!eof +aproc-read-stderr h [count] → bytevector | #!eof +aproc-write h bv → int (bytes written) +aproc-close-stdin! h → void +aproc-close! h → void +aproc-wait h → exit-code +aproc-poll h → exit-code | #f +aproc-kill h [sig] → void +aproc-collect h → (values out err code) +aproc-run cmd [dir [check?]] → stdout-string +aproc-run/status cmd [dir] → (values out err code) +aproc-system cmd → exit-code +``` + +Implementation: Chez `open-process-ports` for fork+exec (we never use +the resulting transcoded ports for I/O — only retain them so the GC +doesn't close their fds). All reads/writes/waits go through libc +declared `__collect_safe`, which releases the TC mutex for the call. + +Known limits documented below. + +--- + +## P0 — blocks adopting aproc as the default subprocess facility + +### P0.1 Timeouts on wait / collect + +`aproc-wait` blocks indefinitely. `bash.ss` accepts a `timeout` +parameter today and silently ignores it because aproc has no +deadline. This is a correctness regression vs. what bash.ss promised +its callers. + +``` +aproc-wait/timeout h ms → exit-code | 'timeout +aproc-collect/timeout h ms → (values out err code-or-timeout) +aproc-run/status cmd [dir [timeout-ms]] → as today, fourth value = 'timeout-or-ok +``` + +Implementation: spawn a watchdog thread that `aproc-kill`s on +deadline, then waitpid as normal. On timeout, return after killing +and reaping so no zombie is left. A `kill -TERM` first, then `-KILL` +after a grace period (~500ms) — matches `subprocess.run(timeout=…)` +in Python. + +Open question: do we want a hard "abort" return that resignals the +caller, or a soft sentinel value? I'd lean soft (`'timeout`) so +existing call sites don't need exception handling. + +### P0.2 EINTR / errno handling + +`aproc-wait` retries on any `rc < 0`, treating every failure as +EINTR. With `__collect_safe`, the kernel auto-restarts most calls, +so in practice this is fine — but a real `ECHILD` (no such PID) or +`EINVAL` (bad flags) becomes an infinite loop. + +Need: `errno`-aware retry. Borrow the macOS / glibc / Bionic +detection from `core/debug-repl.ss`: + +``` +(def c-errno-location + (let loop ((ns '("__error" "__errno_location" "__errno"))) + …)) +``` + +Promote into a shared `(std os errno)` module so aproc, debug-repl, +and future code don't each rebuild it. + +### P0.3 Argv-style spawn (skip the shell) + +`aproc-spawn` always goes through `/bin/sh -c`, which means: + +- Callers escape every argument themselves (we have `sh-quote` but + it's internal). +- A typo in escaping is a shell injection. +- Extra process in the tree (`sh` → real cmd). +- Can't run binaries that aren't on `$PATH` from `sh`'s view. + +Want: + +``` +aproc-spawn* argv [opts] → handle + ;; argv: (list "git" "log" "--oneline") + ;; opts: env, dir, stdin/stdout/stderr redirection +``` + +Implementation: `posix_spawn(3)` is the cleanest — single libc call, +no fork-then-exec race, no manual signal-mask wrangling. `posix_spawn` +takes opaque `posix_spawn_file_actions_t` and `posix_spawnattr_t` +sized differently on macOS vs Linux vs BSDs. Two options: + +(a) Vendor a tiny C shim (`jerboa-aproc.c`) that exposes + `aproc_spawn(argv, env, fa)` returning pid + pipe fds. The shim + knows the struct sizes. Compiled into the jerboa-native library. + +(b) Use `fork` + `execvp` directly. Simpler but unsafe in + multithreaded programs (POSIX only allows async-signal-safe calls + between fork and exec, and Chez Scheme code is not + async-signal-safe — a GC or allocation in the child window can + deadlock). The window is microseconds but it's there. + +Recommend (a). Existing native shim in jerboa-native-rs can host the +function — it's already linked into the binary. + +### P0.4 Environment variables + +`aproc-run/status cmd` runs with the parent's full environment. +Currently the only way to set a var is prefix the shell string with +`FOO=bar`, which mixes argv and env in a leaky way. + +``` +aproc-run/status cmd [opts] + opts: env: '(("FOO" . "bar") …) ; merged on top of parent env + env-pure: '(("FOO" . "bar") …) ; replace parent env entirely + dir: "/path" + timeout-ms: 30000 + stdin: bv | string | port | #f ; data to write then close + stdout: 'capture | port | path | 'inherit + stderr: 'capture | port | path | 'inherit | 'merge-stdout +``` + +`opts` is a hashtable or keyword-arg style — pick one and stick to it. +I'd lean on Jerboa's keyword-arg convention (`keyword:` form) for +ergonomics: `(aproc-run/status "git status" dir: "/tmp" timeout-ms: 5000)`. + +--- + +## P1 — should-have + +### P1.1 Streaming callbacks / channels + +Today, getting partial output as it streams requires the caller to +spawn a reader thread that loops on `aproc-read-stdout`. Every +streaming consumer (the TUI, log tailers, build watchers) reimplements +this. + +Wanted (any one of): + +``` +;; Callback style +(aproc-stream cmd + on-stdout: (lambda (bv) …) + on-stderr: (lambda (bv) …) + on-exit: (lambda (code) …)) + +;; Channel style — preferred, integrates with (std misc channel) +(let-values (((out-ch err-ch exit-ch) (aproc-channels h))) + …) + +;; Port-like — read from a channel-backed input port +(aproc-stdout-port h) ; non-blocking input-port wrapping the fd +``` + +The channel form fits Jerboa's structured-concurrency story best. +Implementation: aproc internally spawns one reader thread per fd that +loops `c-read` + `channel-put`. Closed on EOF, drained on exit. + +### P1.2 Large-output drain performance + +`drain-fd` does `(reverse chunks) → concat`. For a 100MB output split +across 25k 4KB chunks, that's 25k pointer reversals + a single +`bytevector-copy!` across the whole buffer. The copy is O(n), the +reversal is O(chunks). Real problem for `cat big-file` or `find /`. + +Fix: pre-grow a single buffer geometrically (double on full). One +allocation per doubling instead of one per chunk. Or stream into a +`bytevector-output-port` (Chez supports this) and `get-output-string` +at the end. + +### P1.3 Binary vs UTF-8 output + +`aproc-collect` calls `utf8->string` on captured bytes. If the +subprocess outputs binary or invalid UTF-8 (e.g. `xxd`, image-magick, +a coredump), `utf8->string` raises. + +``` +aproc-collect h [encoding:] + encoding: 'utf8 (default, current) + 'utf8-lossy ; replace invalid sequences + 'bytes ; return bytevectors, don't decode +``` + +`'utf8-lossy` is what most callers actually want — the TUI doesn't +care if one byte of `make` output is garbled, it cares about not +crashing. + +### P1.4 Process groups & signal propagation + +Today: `aproc-kill` sends a signal to the immediate child pid. If +the child is `sh -c "long-pipeline | foo | bar"`, killing `sh` +doesn't kill the pipeline members — they're orphaned and keep +running. + +Want: spawn each subprocess in its own process group (`setpgid`), +kill the whole group. Standard pattern. Requires the argv-spawn path +(P0.3) so we can call `setpgid` between fork and exec. + +``` +aproc-kill-group h sig +aproc-spawn* argv (new-process-group: #t …) +``` + +### P1.5 Stdin from a value + +Common pattern: "run cmd, send this string as stdin, capture stdout." +Today: spawn, write, close, collect — four calls. Should be one: + +``` +(aproc-run/status "jq ." stdin: json-str) +``` + +Internally: spawn, write, close-stdin, collect. Trivial wrapper but +removes a class of bugs (forgetting to close-stdin and deadlocking). + +### P1.6 Buffer cross-talk guard + +We retain the Chez stdin port in the handle (so the GC doesn't close +the fd) but write via raw `c-write`. If a caller does port I/O on +`aproc-stdin-port` AND raw writes via `aproc-write`, Chez's buffer +and the raw writes interleave silently. + +Fix options: +- (a) Don't expose the ports at all. `aproc-stdin-fd` only. +- (b) Drop the ports entirely; `dup` the fds in `aproc-spawn` so we + own them outright, then `close-port` the originals immediately. + Simpler invariant: handles own fds, full stop. + +I'd take (b). One extra `dup` per fd at spawn, no GC subtleties. + +--- + +## P2 — nice-to-have + +### P2.1 PTY / pseudo-terminal support + +Some CLI tools (claude, gemini in some modes, ssh, anything that +checks `isatty(0)`) refuse to operate in line mode when fd 0 isn't a +TTY. Today we redirect to pipes and they degrade. + +``` +aproc-spawn* argv (pty: #t …) + ;; allocate openpty(), child sees a TTY, parent reads/writes the master fd +``` + +Useful for: interactive REPLs, anything that hides progress bars +without a TTY, sandboxed shells. + +### P2.2 Non-blocking reads / `aproc-read-available` + +`aproc-read` blocks until at least one byte arrives. Sometimes you +want "give me whatever's there right now, return empty if nothing." + +``` +aproc-read-available h fd [count] → bytevector (possibly empty) +``` + +Implementation: set `O_NONBLOCK` on the fd or use `poll(2)` / +`select(2)` with timeout 0. + +### P2.3 Multi-process operations + +``` +aproc-pipeline (list argv1 argv2 argv3) ; argv1 | argv2 | argv3 +aproc-wait-any (list h1 h2 h3) → handle that exited first +aproc-wait-all (list h1 h2 h3) → list of exit codes +``` + +`wait-any` needs `waitpid(-1, …)` and pid→handle dispatch. Useful +for `fan-out then wait for first failure` patterns. + +### P2.4 Inheritable file descriptors + +``` +aproc-spawn* argv (inherit-fd: '((3 . input-port-or-fd) (4 . …))) +``` + +Lets you pass arbitrary fds into the child — useful for self-pipe +tricks, passing sockets to subprocesses, etc. + +### P2.5 Resource accounting + +``` +aproc-rusage h → record with cpu-user cpu-sys maxrss … +``` + +Wraps `wait4()` or `getrusage(RUSAGE_CHILDREN)`. Useful for +benchmarks, the existing `mcp__jerboa__jerboa_benchmark` tool, build +profilers. + +--- + +## P3 — long-tail polish + +- **OS-thread tracking**: when an aproc handle is GC'd without + `aproc-close!`, we leak fds + zombies. Add a guardian that reaps + on finalize (with a warning). +- **Cancellation propagation**: integrate with `(std async)` so an + Async task that's killed also kills its aproc handles. Currently + they outlive their parent task. +- **Verbose mode**: env var `APROC_TRACE=1` to log every spawn / wait + / kill to stderr. Aids debugging deadlocks. +- **Windows**: not on the table. If it ever is, the `CreateProcess` + story replaces `posix_spawn` and almost everything else stays. + +--- + +## Cross-cutting + +### Documentation + +`docs/aproc.md` doesn't exist yet. Should pair this roadmap with a +user-facing reference covering: + +- The TC-mutex story (why this exists at all). +- Migration from `(system cmd)` / `(std os shell)`. +- Common patterns: capture, stream, time-bound, pipeline. +- When NOT to use aproc (very short subprocesses where `system`'s + overhead is fine and you don't have concurrent threads). + +### Test coverage + +v1 has 13 smoke tests. The matrix for full coverage: + +| dimension | values to exercise | +|--------------------|-------------------------------------------------| +| spawn shape | shell-string, argv, with env, with cwd, with pty | +| exit | clean, non-zero, killed by signal, timed out | +| I/O size | empty, small, 1MB, 100MB | +| streaming | callback, channel, port | +| concurrency | N=1, N=10, N=100 simultaneous handles | +| cancellation | wait timeout, async-task kill, parent exit | +| edge cases | EINTR, EAGAIN on read, SIGPIPE on write | +| portability | macOS, Linux, FreeBSD, OpenBSD | + +Target: separate test file `tests/test-aproc.ss` with a stress +section gated behind `APROC_STRESS=1` so CI doesn't pay the cost. + +### Performance baseline + +Establish numbers, track regressions: + +- `aproc-system "true"` vs Chez `(system "true")` — overhead per call +- `aproc-run "yes | head -n 100000"` — throughput on pipe drain +- 100 concurrent `aproc-run "sleep 1"` — should finish in ~1s, not + 100s (proves __collect_safe is doing its job at scale) + +### Integration audit + +Places in the wider tree that today use `(system …)`, `shell/status`, +or `open-process-ports` and should migrate once their feature deps +are in place: + +| consumer | needs | +|-------------------------------------------|-------------| +| jerboa-code: `bash.ss` (already on aproc) | P0.1 timeout, P1.5 stdin | +| jerboa-code: `external-llm.ss` | P0.1 timeout, then drop sandbox-run/system path | +| jerboa-code: `git.ss`, `mentions.ss`, `hooks.ss`, `checkpoints.ss`, `tui-memstats.ss` | P0.1 timeout | +| jerboa: `std/os/shell.sls` | optional rewrite-internals once aproc has parity | +| jerboa: `lsp.ss` / language servers | P2.1 PTY for stdio-protocol servers that need it | + +`std/os/shell` is the interesting one — if aproc reaches feature +parity, `shell.sls` could become a thin compatibility shim. Don't +force it; let consumers migrate when they hit a need aproc solves. + +--- + +## Suggested sequencing + +A reasonable order of attack, each phase a self-contained PR: + +1. **P0.2 + P0.1** — errno module, then timeouts. Unblocks bash.ss + honoring its `timeout` arg. +2. **P0.3 + P0.4** — argv-spawn via native shim, env/dir/stdin opts. + Pulls in P1.5 (stdin-from-value) for free. Pulls in P1.6 (drop + ports, dup fds) because we're rewriting spawn anyway. +3. **P1.1 + P1.3** — streaming channels, lossy UTF-8 decoder. + Unblocks TUI showing partial output. +4. **P1.2** — drain perf rewrite. Cheap, do it once we have stress + tests from the v1.5 work. +5. **P1.4** — process groups. Requires argv-spawn (done in step 2). +6. **P2.1 onward** — opportunistic, driven by concrete needs. + +Each step ships with its slice of the test matrix and an update to +`docs/aproc.md`. The roadmap itself stays here and gets pruned as +items land.