std/os/aproc: argv spawn, env, PTY, pipeline, inherit-fd, timeouts
ober
a555ea61f0db27fdfe34c00550925addf63fb257
--- a/docs/aproc-roadmap.md +++ b/docs/aproc-roadmap.md @@ -1,405 +1,110 @@ # `(std os aproc)` — Roadmap -`(std os aproc)` ships in v1 as a small async-subprocess facility built -on `__collect_safe` libc primitives. It solves one concrete problem: -Chez's built-in `(system cmd)` and port I/O pin the TC mutex for the -entire subprocess lifetime, freezing every other green thread. v1 -releases the mutex while parked in the kernel so TUIs, watchdogs, and -streaming loops keep running. - -This roadmap covers everything else. - -## v1 state (today) - -API surface (exported): - -``` -aproc-spawn cmd → handle -aproc-handle? x → bool -aproc-pid h | aproc-stdin-fd h | … → int -aproc-exit-code h → int | #f -aproc-read h fd [count] → bytevector | #!eof -aproc-read-stdout h [count] → bytevector | #!eof -aproc-read-stderr h [count] → bytevector | #!eof -aproc-write h bv → int (bytes written) -aproc-close-stdin! h → void -aproc-close! h → void -aproc-wait h → exit-code -aproc-poll h → exit-code | #f -aproc-kill h [sig] → void -aproc-collect h → (values out err code) -aproc-run cmd [dir [check?]] → stdout-string -aproc-run/status cmd [dir] → (values out err code) -aproc-system cmd → exit-code -``` - -Implementation: Chez `open-process-ports` for fork+exec (we never use -the resulting transcoded ports for I/O — only retain them so the GC -doesn't close their fds). All reads/writes/waits go through libc -declared `__collect_safe`, which releases the TC mutex for the call. - -Known limits documented below. - ---- - -## P0 — blocks adopting aproc as the default subprocess facility - -### P0.1 Timeouts on wait / collect - -`aproc-wait` blocks indefinitely. `bash.ss` accepts a `timeout` -parameter today and silently ignores it because aproc has no -deadline. This is a correctness regression vs. what bash.ss promised -its callers. - -``` -aproc-wait/timeout h ms → exit-code | 'timeout -aproc-collect/timeout h ms → (values out err code-or-timeout) -aproc-run/status cmd [dir [timeout-ms]] → as today, fourth value = 'timeout-or-ok -``` - -Implementation: spawn a watchdog thread that `aproc-kill`s on -deadline, then waitpid as normal. On timeout, return after killing -and reaping so no zombie is left. A `kill -TERM` first, then `-KILL` -after a grace period (~500ms) — matches `subprocess.run(timeout=…)` -in Python. - -Open question: do we want a hard "abort" return that resignals the -caller, or a soft sentinel value? I'd lean soft (`'timeout`) so -existing call sites don't need exception handling. - -### P0.2 EINTR / errno handling - -`aproc-wait` retries on any `rc < 0`, treating every failure as -EINTR. With `__collect_safe`, the kernel auto-restarts most calls, -so in practice this is fine — but a real `ECHILD` (no such PID) or -`EINVAL` (bad flags) becomes an infinite loop. - -Need: `errno`-aware retry. Borrow the macOS / glibc / Bionic -detection from `core/debug-repl.ss`: - -``` -(def c-errno-location - (let loop ((ns '("__error" "__errno_location" "__errno"))) - …)) -``` - -Promote into a shared `(std os errno)` module so aproc, debug-repl, -and future code don't each rebuild it. - -### P0.3 Argv-style spawn (skip the shell) - -`aproc-spawn` always goes through `/bin/sh -c`, which means: - -- Callers escape every argument themselves (we have `sh-quote` but - it's internal). -- A typo in escaping is a shell injection. -- Extra process in the tree (`sh` → real cmd). -- Can't run binaries that aren't on `$PATH` from `sh`'s view. - -Want: - -``` -aproc-spawn* argv [opts] → handle - ;; argv: (list "git" "log" "--oneline") - ;; opts: env, dir, stdin/stdout/stderr redirection -``` - -Implementation: `posix_spawn(3)` is the cleanest — single libc call, -no fork-then-exec race, no manual signal-mask wrangling. `posix_spawn` -takes opaque `posix_spawn_file_actions_t` and `posix_spawnattr_t` -sized differently on macOS vs Linux vs BSDs. Two options: - -(a) Vendor a tiny C shim (`jerboa-aproc.c`) that exposes - `aproc_spawn(argv, env, fa)` returning pid + pipe fds. The shim - knows the struct sizes. Compiled into the jerboa-native library. - -(b) Use `fork` + `execvp` directly. Simpler but unsafe in - multithreaded programs (POSIX only allows async-signal-safe calls - between fork and exec, and Chez Scheme code is not - async-signal-safe — a GC or allocation in the child window can - deadlock). The window is microseconds but it's there. - -Recommend (a). Existing native shim in jerboa-native-rs can host the -function — it's already linked into the binary. - -### P0.4 Environment variables - -`aproc-run/status cmd` runs with the parent's full environment. -Currently the only way to set a var is prefix the shell string with -`FOO=bar`, which mixes argv and env in a leaky way. - -``` -aproc-run/status cmd [opts] - opts: env: '(("FOO" . "bar") …) ; merged on top of parent env - env-pure: '(("FOO" . "bar") …) ; replace parent env entirely - dir: "/path" - timeout-ms: 30000 - stdin: bv | string | port | #f ; data to write then close - stdout: 'capture | port | path | 'inherit - stderr: 'capture | port | path | 'inherit | 'merge-stdout -``` - -`opts` is a hashtable or keyword-arg style — pick one and stick to it. -I'd lean on Jerboa's keyword-arg convention (`keyword:` form) for -ergonomics: `(aproc-run/status "git status" dir: "/tmp" timeout-ms: 5000)`. - ---- - -## P1 — should-have - -### P1.1 Streaming callbacks / channels - -Today, getting partial output as it streams requires the caller to -spawn a reader thread that loops on `aproc-read-stdout`. Every -streaming consumer (the TUI, log tailers, build watchers) reimplements -this. - -Wanted (any one of): - -``` -;; Callback style -(aproc-stream cmd - on-stdout: (lambda (bv) …) - on-stderr: (lambda (bv) …) - on-exit: (lambda (code) …)) - -;; Channel style — preferred, integrates with (std misc channel) -(let-values (((out-ch err-ch exit-ch) (aproc-channels h))) - …) - -;; Port-like — read from a channel-backed input port -(aproc-stdout-port h) ; non-blocking input-port wrapping the fd -``` - -The channel form fits Jerboa's structured-concurrency story best. -Implementation: aproc internally spawns one reader thread per fd that -loops `c-read` + `channel-put`. Closed on EOF, drained on exit. - -### P1.2 Large-output drain performance - -`drain-fd` does `(reverse chunks) → concat`. For a 100MB output split -across 25k 4KB chunks, that's 25k pointer reversals + a single -`bytevector-copy!` across the whole buffer. The copy is O(n), the -reversal is O(chunks). Real problem for `cat big-file` or `find /`. - -Fix: pre-grow a single buffer geometrically (double on full). One -allocation per doubling instead of one per chunk. Or stream into a -`bytevector-output-port` (Chez supports this) and `get-output-string` -at the end. - -### P1.3 Binary vs UTF-8 output - -`aproc-collect` calls `utf8->string` on captured bytes. If the -subprocess outputs binary or invalid UTF-8 (e.g. `xxd`, image-magick, -a coredump), `utf8->string` raises. - -``` -aproc-collect h [encoding:] - encoding: 'utf8 (default, current) - 'utf8-lossy ; replace invalid sequences - 'bytes ; return bytevectors, don't decode -``` - -`'utf8-lossy` is what most callers actually want — the TUI doesn't -care if one byte of `make` output is garbled, it cares about not -crashing. - -### P1.4 Process groups & signal propagation - -Today: `aproc-kill` sends a signal to the immediate child pid. If -the child is `sh -c "long-pipeline | foo | bar"`, killing `sh` -doesn't kill the pipeline members — they're orphaned and keep +`(std os aproc)` is an async-subprocess facility built on `__collect_safe` +libc primitives. It solves the TC-mutex problem: Chez's built-in +`(system cmd)` and port I/O pin the mutex for the entire subprocess +lifetime, freezing every other green thread. aproc releases the mutex +while parked in the kernel so TUIs, watchdogs, and streaming loops keep running. -Want: spawn each subprocess in its own process group (`setpgid`), -kill the whole group. Standard pattern. Requires the argv-spawn path -(P0.3) so we can call `setpgid` between fork and exec. - -``` -aproc-kill-group h sig -aproc-spawn* argv (new-process-group: #t …) -``` - -### P1.5 Stdin from a value - -Common pattern: "run cmd, send this string as stdin, capture stdout." -Today: spawn, write, close, collect — four calls. Should be one: - -``` -(aproc-run/status "jq ." stdin: json-str) -``` - -Internally: spawn, write, close-stdin, collect. Trivial wrapper but -removes a class of bugs (forgetting to close-stdin and deadlocking). - -### P1.6 Buffer cross-talk guard - -We retain the Chez stdin port in the handle (so the GC doesn't close -the fd) but write via raw `c-write`. If a caller does port I/O on -`aproc-stdin-port` AND raw writes via `aproc-write`, Chez's buffer -and the raw writes interleave silently. - -Fix options: -- (a) Don't expose the ports at all. `aproc-stdin-fd` only. -- (b) Drop the ports entirely; `dup` the fds in `aproc-spawn` so we - own them outright, then `close-port` the originals immediately. - Simpler invariant: handles own fds, full stop. +For the user-facing reference see [`aproc.md`](aproc.md). -I'd take (b). One extra `dup` per fd at spawn, no GC subtleties. +## What shipped ---- +P0 (timeouts, errno module, argv spawn, env/dir/stdin keyword opts), +P1 (streaming channels, geometric drain buffer, encoding modes, +process groups, stdin-from-value, dup'd fds for buffer-crosstalk +guard), P2 (PTY, non-blocking reads, pipeline / wait-any / wait-all, +inheritable file descriptors, getrusage), and the P3 polish items that +made sense on Unix (GC guardian for stray handles, `APROC_TRACE=1` +verbose mode). -## P2 — nice-to-have +Test coverage: `tests/test-aproc.ss`, 32 tests, stress section gated +behind `APROC_STRESS=1`. -### P2.1 PTY / pseudo-terminal support +## What didn't ship — and why -Some CLI tools (claude, gemini in some modes, ssh, anything that -checks `isatty(0)`) refuse to operate in line mode when fd 0 isn't a -TTY. Today we redirect to pipes and they degrade. +- **Cancellation propagation from `(std async)`** — `(std async)` has no + task-cancellation primitive to hook into. When that lands, aproc + handles should register a guardian that kills/reaps on task cancel. +- **Windows** — `posix_spawn` and the whole errno story are POSIX-only. + If Windows ever comes back on the table, the Rust shim's spawn path + would route through `CreateProcessW` and most everything else stays. -``` -aproc-spawn* argv (pty: #t …) - ;; allocate openpty(), child sees a TTY, parent reads/writes the master fd -``` +## Future iterations -Useful for: interactive REPLs, anything that hides progress bars -without a TTY, sandboxed shells. +### Native pipeline (replace userspace pump threads) -### P2.2 Non-blocking reads / `aproc-read-available` +`aproc-pipeline` today pumps bytes between stages with one userspace +thread per junction (`read` from upstream, `write` downstream). This +works but adds two context switches per chunk vs. a direct +kernel-pipe handoff. -`aproc-read` blocks until at least one byte arrives. Sometimes you -want "give me whatever's there right now, return empty if nothing." +A native variant would have the Rust shim build the whole pipeline in +one call: `pipe()` between stages, `dup2` in each child's pre_exec, no +parent involvement on the byte path. The shim already does dup2 for +inherit-fd; this is a generalization. Worth it if anyone hits a +throughput ceiling on a hot pipeline; not worth it for the common case +of human-scale data volumes. -``` -aproc-read-available h fd [count] → bytevector (possibly empty) -``` +### `posix_spawn` fast path -Implementation: set `O_NONBLOCK` on the fd or use `poll(2)` / -`select(2)` with timeout 0. +Rust's `std::process::Command` falls back to `fork+exec` whenever +`pre_exec` is used. We use `pre_exec` for every spawn (to handle +process groups, fd inheritance, stderr merge, PTY setup). The +fork+exec path costs a full address-space copy on Linux (CoW, then +discarded by exec) — `posix_spawn` skips that. -### P2.3 Multi-process operations +Bypassing `pre_exec` for the common no-customization case +(stdin/stdout/stderr pipes, no pgroup, no inherit-fd, no merge) would +let `std::process::Command` pick `posix_spawn` automatically. The +saving is microseconds per spawn — only meaningful for code that +spawns thousands of subprocesses (test runners, find-style traversals). -``` -aproc-pipeline (list argv1 argv2 argv3) ; argv1 | argv2 | argv3 -aproc-wait-any (list h1 h2 h3) → handle that exited first -aproc-wait-all (list h1 h2 h3) → list of exit codes -``` +### Per-stream encoding -`wait-any` needs `waitpid(-1, …)` and pid→handle dispatch. Useful -for `fan-out then wait for first failure` patterns. +`encoding:` currently applies to both stdout and stderr. A program +that emits binary on stdout and human text on stderr (`tar`, `dd` +verbose) would benefit from `stdout-encoding: 'bytes` + +`stderr-encoding: 'utf8-lossy`. Cheap addition once a caller wants it. -### P2.4 Inheritable file descriptors +### Sandbox integration -``` -aproc-spawn* argv (inherit-fd: '((3 . input-port-or-fd) (4 . …))) -``` +aproc's `new-pgroup: #t` is the foundation but doesn't yet wire into +the sandbox seccomp / landlock / namespace plumbing. A +`sandbox:` keyword that hands off to `lib/std/os/sandbox.sls` would +let scripts spawn subprocesses that inherit their parent's sandbox +profile. -Lets you pass arbitrary fds into the child — useful for self-pipe -tricks, passing sockets to subprocesses, etc. +### Async-friendly readers -### P2.5 Resource accounting +`aproc-channels` exposes raw byte chunks; many callers want +line-buffered reading. A `aproc-lines` that wraps the output channel +with a line-splitting transducer would be a tiny addition. -``` -aproc-rusage h → record with cpu-user cpu-sys maxrss … -``` +### Integration audit (consumers to migrate) -Wraps `wait4()` or `getrusage(RUSAGE_CHILDREN)`. Useful for -benchmarks, the existing `mcp__jerboa__jerboa_benchmark` tool, build -profilers. +Places in the wider tree that still use `(system …)`, +`shell/status`, or `open-process-ports` and could migrate now that +aproc has feature parity: ---- +| consumer | current state | needs | +|---|---|---| +| `jerboa-code: bash.ss` | already on aproc | done; pass `timeout-ms:` from caller | +| `jerboa-code: external-llm.ss` | sandbox-run/system | drop sandbox path, use `aproc-run/status*` directly | +| `jerboa-code: git.ss`, `mentions.ss`, `hooks.ss`, `checkpoints.ss`, `tui-memstats.ss` | `(system …)` | mechanical migration with `timeout-ms:` | +| `jerboa: std/os/shell.sls` | open-process-ports | could become a compatibility shim over aproc | +| `jerboa: lsp.ss` | `(system …)` for language servers | use `pty: #t` for servers that detect stdio mode | -## P3 — long-tail polish +Don't force migrations; let consumers pull aproc in when they hit a +need the old code can't serve (concurrent execution, timeouts, etc.). -- **OS-thread tracking**: when an aproc handle is GC'd without - `aproc-close!`, we leak fds + zombies. Add a guardian that reaps - on finalize (with a warning). -- **Cancellation propagation**: integrate with `(std async)` so an - Async task that's killed also kills its aproc handles. Currently - they outlive their parent task. -- **Verbose mode**: env var `APROC_TRACE=1` to log every spawn / wait - / kill to stderr. Aids debugging deadlocks. -- **Windows**: not on the table. If it ever is, the `CreateProcess` - story replaces `posix_spawn` and almost everything else stays. +### Performance baselines ---- - -## Cross-cutting - -### Documentation - -`docs/aproc.md` doesn't exist yet. Should pair this roadmap with a -user-facing reference covering: - -- The TC-mutex story (why this exists at all). -- Migration from `(system cmd)` / `(std os shell)`. -- Common patterns: capture, stream, time-bound, pipeline. -- When NOT to use aproc (very short subprocesses where `system`'s - overhead is fine and you don't have concurrent threads). - -### Test coverage - -v1 has 13 smoke tests. The matrix for full coverage: - -| dimension | values to exercise | -|--------------------|-------------------------------------------------| -| spawn shape | shell-string, argv, with env, with cwd, with pty | -| exit | clean, non-zero, killed by signal, timed out | -| I/O size | empty, small, 1MB, 100MB | -| streaming | callback, channel, port | -| concurrency | N=1, N=10, N=100 simultaneous handles | -| cancellation | wait timeout, async-task kill, parent exit | -| edge cases | EINTR, EAGAIN on read, SIGPIPE on write | -| portability | macOS, Linux, FreeBSD, OpenBSD | - -Target: separate test file `tests/test-aproc.ss` with a stress -section gated behind `APROC_STRESS=1` so CI doesn't pay the cost. - -### Performance baseline - -Establish numbers, track regressions: +Should establish numbers and watch for regressions: - `aproc-system "true"` vs Chez `(system "true")` — overhead per call -- `aproc-run "yes | head -n 100000"` — throughput on pipe drain -- 100 concurrent `aproc-run "sleep 1"` — should finish in ~1s, not - 100s (proves __collect_safe is doing its job at scale) - -### Integration audit - -Places in the wider tree that today use `(system …)`, `shell/status`, -or `open-process-ports` and should migrate once their feature deps -are in place: - -| consumer | needs | -|-------------------------------------------|-------------| -| jerboa-code: `bash.ss` (already on aproc) | P0.1 timeout, P1.5 stdin | -| jerboa-code: `external-llm.ss` | P0.1 timeout, then drop sandbox-run/system path | -| jerboa-code: `git.ss`, `mentions.ss`, `hooks.ss`, `checkpoints.ss`, `tui-memstats.ss` | P0.1 timeout | -| jerboa: `std/os/shell.sls` | optional rewrite-internals once aproc has parity | -| jerboa: `lsp.ss` / language servers | P2.1 PTY for stdio-protocol servers that need it | - -`std/os/shell` is the interesting one — if aproc reaches feature -parity, `shell.sls` could become a thin compatibility shim. Don't -force it; let consumers migrate when they hit a need aproc solves. - ---- - -## Suggested sequencing - -A reasonable order of attack, each phase a self-contained PR: - -1. **P0.2 + P0.1** — errno module, then timeouts. Unblocks bash.ss - honoring its `timeout` arg. -2. **P0.3 + P0.4** — argv-spawn via native shim, env/dir/stdin opts. - Pulls in P1.5 (stdin-from-value) for free. Pulls in P1.6 (drop - ports, dup fds) because we're rewriting spawn anyway. -3. **P1.1 + P1.3** — streaming channels, lossy UTF-8 decoder. - Unblocks TUI showing partial output. -4. **P1.2** — drain perf rewrite. Cheap, do it once we have stress - tests from the v1.5 work. -5. **P1.4** — process groups. Requires argv-spawn (done in step 2). -6. **P2.1 onward** — opportunistic, driven by concrete needs. - -Each step ships with its slice of the test matrix and an update to -`docs/aproc.md`. The roadmap itself stays here and gets pruned as -items land. +- `aproc-run "yes | head -n 100000"` — pipe-drain throughput +- 100 concurrent `aproc-run "sleep 1"` — must finish in ~1s, not 100s + (the `APROC_STRESS=1` stress test asserts this — needs a CI + invocation) new file mode 100644 --- /dev/null +++ b/docs/aproc.md @@ -0,0 +1,270 @@ +# `(std os aproc)` — Async subprocess facility + +Run external commands without freezing the rest of the Scheme process. + +## Why this exists + +Chez's built-in `(system cmd)` and the ports returned by +`(open-process-ports …)` acquire the **TC mutex** for the entire +subprocess lifetime. While one Scheme thread is parked in those calls, +every other green thread in the process is suspended. TUIs hang. +Watchdogs stop ticking. Async tasks stall. Debug REPLs freeze. + +`(std os aproc)` solves this by routing every blocking syscall through +`__collect_safe` foreign procedures. Chez releases the mutex before the +kernel call and re-acquires it on return, so the rest of the runtime +keeps running while the kernel does its work. + +If your program is single-threaded and never starts a green thread, you +do not need this module — `(system cmd)` is fine. If anything else in +your image needs to make progress while a subprocess runs, you want +aproc. + +## Quick reference + +```scheme +;; Capture stdout, raise on non-zero exit +(aproc-run "echo hello") ; => "hello\n" + +;; Capture all three; never raises on exit code +(aproc-run/status "make all" 'timeout-ms: 30000) ; => (values out err code|'timeout) + +;; Just the exit code, no capture +(aproc-system "true") ; => 0 + +;; argv form, no shell — args go straight to argv +(aproc-run/status* '("git" "log" "--oneline") + 'dir: "/repo" + 'env: '(("GIT_PAGER" . "cat"))) + +;; Send stdin then collect +(aproc-run/status "jq ." 'stdin: json-string) +``` + +## API + +### High-level + +| API | What it does | +|---|---| +| `(aproc-run cmd)` | shell, returns stdout string; raises on non-zero exit | +| `(aproc-run/status cmd [opts])` | shell, returns `(values out err code)` | +| `(aproc-run/status* argv [opts])` | argv, returns `(values out err code)` | +| `(aproc-system cmd)` | shell, returns exit code only | + +Options (keyword args): + +``` +env: '(("FOO" . "bar") …) ; overlay on parent env +env-pure: '(("FOO" . "bar") …) ; replace parent env entirely +dir: "/path" ; working directory +timeout-ms: 5000 ; kill after N ms; code becomes 'timeout +stdin: bv | string ; write then close stdin +stdout: 'capture | 'inherit | 'devnull | "/path/to/file" +stderr: 'capture | 'inherit | 'devnull | "/path" | 'merge-stdout +encoding: 'utf8 | 'utf8-lossy | 'bytes +new-pgroup: #t ; child becomes its own process-group leader +pty: #t ; allocate a pseudo-TTY (stdin/stdout/stderr) +inherit-fd: '((3 . src-fd) …) ; dup arbitrary fds into the child +``` + +### Low-level handle + +``` +(aproc-spawn "shell command") ; legacy shell-string form +(aproc-spawn* '("argv" "…") [opts]) ; argv form, with kwargs + +(aproc-pid h) (aproc-pgrp h) +(aproc-stdin-fd h) (aproc-stdout-fd h) (aproc-stderr-fd h) +(aproc-pty-master-fd h) ; #f if not a PTY +(aproc-exit-code h) ; #f until waited + +(aproc-write h bv) +(aproc-read-stdout h [count]) +(aproc-read-stderr h [count]) +(aproc-read-available h fd [count]) ; non-blocking; empty bv if no data + +(aproc-close-stdin! h) +(aproc-close! h) ; close all retained fds + +(aproc-wait h) ; blocks; returns exit code +(aproc-wait/timeout h ms) ; exit code | 'timeout +(aproc-poll h) ; exit code | #f +(aproc-kill h [sig]) ; default SIGTERM +(aproc-kill-group h [sig]) ; needs new-pgroup: #t + +(aproc-collect h [encoding]) ; (values out err code), drains all +(aproc-collect/timeout h ms [encoding]) + +(aproc-rusage h) ; getrusage record +``` + +### Streaming via channels + +```scheme +(let ([h (aproc-spawn* '("tail" "-F" "log"))]) + (let-values (((out-ch err-ch exit-ch) (aproc-channels h))) + (let loop () + (let ([c (channel-get out-ch)]) + (cond + [(eof-object? c) (channel-get exit-ch)] + [else (display (utf8->string c)) (loop)]))))) +``` + +`aproc-channels` spawns one reader thread per fd and pushes raw +bytevector chunks into a channel. `exit-ch` receives the exit code +exactly once. + +### Multi-process + +```scheme +;; Build a pipeline; returns the LAST stage as a handle whose stdout +;; you collect. +(let ([h (aproc-pipeline '("printf" "hi") '("wc" "-c"))]) + (aproc-collect h)) + +;; Wait for any one of N handles to finish. +(aproc-wait-any (list h1 h2 h3)) ; returns the winning handle + +;; Wait for all; returns list of exit codes (in input order). +(aproc-wait-all (list h1 h2 h3)) +``` + +## Patterns + +### Capture and check + +```scheme +(let-values (((out err code) (aproc-run/status "rustc --version"))) + (when (zero? code) out)) +``` + +### Time-bound + +```scheme +(let-values (((out err code) + (aproc-run/status* '("make" "-j8" "all") + 'timeout-ms: 60000))) + (case code + [(timeout) (printf "build hung; killed~%")] + [(0) (printf "ok~%")] + [else (printf "failed: ~a~%~a~%" code err)])) +``` + +### Stream output as it arrives + +```scheme +(let ([h (aproc-spawn* '("cargo" "build") 'new-pgroup: #t)]) + (let-values (((out-ch err-ch exit-ch) (aproc-channels h))) + (let loop () + (let ([c (channel-get out-ch)]) + (cond + [(eof-object? c) (channel-get exit-ch)] + [else (display (utf8->string c)) (flush-output-port (current-output-port)) (loop)]))))) +``` + +### Time-bound a whole pipeline + +```scheme +(let ([h (aproc-pipeline + '("find" "." "-type" "f") + '("xargs" "wc" "-l"))]) + ;; Kill the entire pipeline if it takes > 30 s + (let-values (((out err code) (aproc-collect/timeout h 30000))) + out)) +``` + +### Cancellation: kill the whole pgroup + +```scheme +;; new-pgroup: #t puts the child (and everything it forks) in its own +;; group. aproc-kill-group sends to the group, not just pid. +(let ([h (aproc-spawn* '("sh" "-c" "long-pipeline & wait") 'new-pgroup: #t)]) + (start-watchdog! h) + (aproc-collect h)) +``` + +### Pass an open fd to the child + +```scheme +;; e.g. a socket pre-bound by the parent, handed to the child as fd 3. +(let ([sock-fd (open-listen-socket! "0.0.0.0:8080")]) + (aproc-run/status* '("./worker") + 'inherit-fd: `((3 . ,sock-fd)))) +``` + +The Rust shim `dup2`s `source` onto `target` and clears `FD_CLOEXEC` so +the fd survives `execve(2)`. + +## Migration from `(system cmd)` + +```scheme +;; Before +(system "make all") +;; After — same return value, releases TC mutex while make runs +(aproc-system "make all") + +;; Before — silently freezes the rest of the image +(define output (with-output-to-string (lambda () (system "git log")))) +;; After +(define output (aproc-run "git log")) + +;; Before — leaks ports on exception, no timeout +(call-with-values (lambda () (open-process-ports "curl example.com")) + (lambda (in out err pid) + (let ([body (get-bytevector-all out)]) + (close-port in) (close-port out) (close-port err) + body))) +;; After +(let-values (((out err code) (aproc-run/status "curl example.com"))) + (when (zero? code) out)) +``` + +## Tracing + +Set `APROC_TRACE=1` in the environment to log every spawn, wait, kill, +and reap to stderr — useful when chasing deadlocks. The trace lines are +prefixed with `[aproc]` and include pid, argv, and elapsed time. + +## When NOT to use aproc + +- The program is single-threaded and stays that way. `(system cmd)`'s + TC-mutex hold is harmless if nothing else is trying to run. +- The subprocess is so short (single-digit ms) that the extra fork+exec + overhead of going through the native shim isn't worth it. +- You're inside a `#!chezscheme` library that can't take a dependency + on `(std os aproc)` for layering reasons. + +## TC-mutex story + +Chez Scheme is a managed runtime with a thread coordinator (TC). The TC +mutex serializes GC, signal delivery, allocations crossing thread +boundaries, and a few other shared-state operations. Every Scheme +operation acquires it briefly; long-running calls hold it for their +entire duration unless the foreign procedure is marked `__collect_safe`. + +`__collect_safe` declares: "this call may block in the kernel; release +the mutex before entering, reacquire on return; do not touch the Scheme +heap while the mutex is dropped." aproc uses `__collect_safe` on every +foreign call that can block: `read`, `write`, `waitpid`, `kill`, +`close`, `fcntl(F_GETFL/F_SETFL)`, etc. The Rust shim's `spawn` call is +NOT `__collect_safe` because it runs in user space and is fast enough +not to matter; the long-running waits and I/O — which is where Chez +ports would block — all are. + +The result: a `(aproc-collect h)` on a 5-minute `make` call lets the +rest of the image run normally for the full 5 minutes. + +## Implementation surface + +- `lib/std/os/aproc.sls` — Scheme module, ~1000 lines. +- `lib/std/os/errno.sls` — cross-platform errno + constants. +- `jerboa-native-rs/src/aproc.rs` — Rust shim providing + `jerboa_aproc_spawn`, `jerboa_aproc_spawn_pty`, + `jerboa_aproc_set_nonblock`, `jerboa_aproc_killpg`, + `jerboa_aproc_wait4`, `jerboa_aproc_dup`, `jerboa_aproc_close`, + `jerboa_aproc_last_error`. + +The shim is optional. If `libjerboa_native` isn't loaded the module +falls back to `(open-process-ports …)` for `aproc-spawn` (shell form +only); argv-spawn, PTY, killpg, and wait4 features become unavailable. --- a/jerboa-native-rs/Cargo.lock +++ b/jerboa-native-rs/Cargo.lock @@ -242,6 +242,45 @@ dependencies = [ ] [[package]] +name = "asn1-rs" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5493c3bedbacf7fd7382c6346bbd66687d12bbaad3a89a2d2c303ee6cf20b048" +dependencies = [ + "asn1-rs-derive", + "asn1-rs-impl", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror 1.0.69", + "time", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "965c2d33e53cb6b267e148a4cb0760bc01f4904c1cd4bb4002a085bb016d1490" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "synstructure", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] name = "async-trait" version = "0.1.89" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -578,6 +617,12 @@ dependencies = [ ] [[package]] +name = "data-encoding" +version = "2.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7a1e2f27636f116493b8b860f5546edb47c8d8f8ea73e1d2a20be88e28d1fea" + +[[package]] name = "der" version = "0.7.10" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -588,6 +633,20 @@ dependencies = [ ] [[package]] +name = "der-parser" +version = "9.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5cd0a5c643689626bec213c4d8bd4d96acc8ffdb4ad4bb6bc16abf27d5f4b553" +dependencies = [ + "asn1-rs", + "displaydoc", + "nom", + "num-bigint", + "num-traits", + "rusticata-macros", +] + +[[package]] name = "deranged" version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -1325,6 +1384,12 @@ dependencies = [ ] [[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] name = "lexical-core" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -1640,6 +1705,15 @@ dependencies = [ ] [[package]] +name = "oid-registry" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a8d8034d9489cdaf79228eb9f6a3b8d7bb32ba00d6645ebd48eef4077ceb5bd9" +dependencies = [ + "asn1-rs", +] + +[[package]] name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -1877,7 +1951,7 @@ dependencies = [ "rustc-hash", "rustls", "socket2", - "thiserror", + "thiserror 2.0.18", "tokio", "tracing", "web-time", @@ -1898,7 +1972,7 @@ dependencies = [ "rustls", "rustls-pki-types", "slab", - "thiserror", + "thiserror 2.0.18", "tinyvec", "tracing", "web-time", @@ -2008,6 +2082,7 @@ dependencies = [ "ring", "rustls-pki-types", "time", + "x509-parser", "yasna", ] @@ -2210,6 +2285,15 @@ dependencies = [ ] [[package]] +name = "rusticata-macros" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" +dependencies = [ + "nom", +] + +[[package]] name = "rustix" version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -2579,11 +2663,31 @@ dependencies = [ [[package]] name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" dependencies = [ - "thiserror-impl", + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", ] [[package]] @@ -2604,10 +2708,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" dependencies = [ "deranged", + "itoa", "num-conv", "powerfmt", "serde_core", "time-core", + "time-macros", ] [[package]] @@ -2617,6 +2723,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" [[package]] +name = "time-macros" +version = "0.2.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] name = "tiny-keccak" version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -3323,6 +3439,24 @@ dependencies = [ ] [[package]] +name = "x509-parser" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fcbc162f30700d6f3f82a24bf7cc62ffe7caea42c0b2cba8bf7f3ae50cf51f69" +dependencies = [ + "asn1-rs", + "data-encoding", + "der-parser", + "lazy_static", + "nom", + "oid-registry", + "ring", + "rusticata-macros", + "thiserror 1.0.69", + "time", +] + +[[package]] name = "xattr" version = "1.6.1" source = "registry+https://github.com/rust-lang/crates.io-index"