Update Jerboa security readiness docs
ober
68a443b5a5136889e029c57f22188084a686f467
--- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ Every Jerboa file looks like this: (displayln (my-function 1 2)) ``` -Run with: `scheme --libdirs lib --script file.ss` +Run with: `jerboa run file.ss` **NEVER** write `(library ...)` forms — that's `.sls` internal syntax. @@ -405,7 +405,7 @@ Jerboa is a niche Scheme dialect with limited training data. **Never guess — a | Find where something is defined | `jerboa_find_definition` — source file, module, kind, arity | | Search for symbol by substring | `jerboa_apropos` or `jerboa_smart_complete` | | Build the project | `jerboa_build_and_report` or `jerboa_make` — prefer over bash `make` | -| Run tests | `jerboa_run_tests` — prefer over bash `scheme --script` | +| Run tests | `jerboa_run_tests` — prefer over ad hoc shell test invocations | | Check for stale .so artifacts | `jerboa_stale_static` — common cause of "edit has no effect" | | Macro expansion | `jerboa_expand_macro` / `jerboa_trace_macro` | | Inspect struct/class types | `jerboa_class_info` — fields, inheritance, constructor signature | --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ All user-facing Jerboa code is written in `.ss` files: Run a file with: ```bash -scheme --libdirs lib --script hello.ss +jerboa run hello.ss ``` Do not write `(library ...)` forms in user code. The `.sls` files in @@ -60,7 +60,7 @@ literal list. ## What Jerboa Includes -At this snapshot, `lib/std` contains 614 `.ss` modules. The prelude +At this snapshot, `lib/std` plus `lib/jerboa` contain 647 `.ss` modules. The prelude exports the everyday language; specialized libraries are imported from `(std ...)`, `(jerboa ...)`, and related module trees. @@ -234,7 +234,7 @@ make jmcp-portable ## Requirements and Platforms -- Chez Scheme 10.x. +- `jerboa` on `PATH`, or `bin/jerboa` from a source checkout. - Optional Rust toolchain for `libjerboa_native` features. - Optional legacy `chez-*` C libraries for older wrappers that remain available while the Rust backend supersedes them. @@ -251,12 +251,12 @@ support on macOS. lib/ jerboa/ core reader, macros, runtime, prelude, package/build glue, typed wrappers, WASM support - std/ 614 standard-library .ss modules + std/ standard-library .ss modules jerboa-native-rs/ optional Rust shared library backend mcp/ active Jerboa MCP server data/ MCP cookbook, API signatures, feature and changelog data docs/ language, library, build, security, WASM, typed, and ops docs -tests/ Scheme test suites +tests/ Jerboa test suites benchmarks/ benchmark harnesses tools/ generators, linters, build and audit helpers vendor/ vendored Chez Scheme tree @@ -274,7 +274,7 @@ Start with these docs: - [`docs/concurrency.md`](docs/concurrency.md), [`docs/fiber.md`](docs/fiber.md), [`docs/async.md`](docs/async.md), [`docs/stm.md`](docs/stm.md), and - [`docs/core-async.md`](docs/core-async.md) for concurrency. + [`docs/lightweight-concurrency.md`](docs/lightweight-concurrency.md) for concurrency. - [`docs/security-reference.md`](docs/security-reference.md) and [`docs/safety-guide.md`](docs/safety-guide.md) for hardening. - [`docs/release-security.md`](docs/release-security.md) for production --- a/data/anti-patterns.sexp +++ b/data/anti-patterns.sexp @@ -2732,4 +2732,18 @@ "jerboa_security_scan" "make build" "git clean -xfd && make binary" - "cc -c -D*_NO_OPENSSL"))) + "cc -c -D*_NO_OPENSSL")) + (("advice" + . + "Use POSIX awk constructs such as assigning to a temporary variable plus gsub, or require and invoke gawk explicitly with a clear prerequisite check. After changing doc/test tooling, run the relevant Make target on the platform awk available in the current environment.") + ("avoid" + . + "Do not use GNU awk-only functions such as gensub in repository tooling that must run on macOS, FreeBSD, or other BSD awk environments.") + ("id" . "gnu-awk-only-doc-tooling") ("kinds" "test" "docs") + ("pattern" . "gensub\\(|awk .*gensub|gawk-only") + ("severity" . "medium") + ("tags" "awk" "macos" "bsd" "docs" "check-docs") + ("title" + . + "Do Not Use GNU Awk-Only Features In Portable Doc/Test Tools") + ("tools" "rg" "make check-docs" "git diff --check"))) deleted file mode 100644 --- a/docs/Future-proofing.md +++ /dev/null @@ -1,380 +0,0 @@ -# Future-Proofing Jerboa Against AI-Era Adversaries - -Forward-looking design strategy for keeping Jerboa robust as AI-driven bug -hunting and exploit construction scale. This document is the strategic / -language-design layer; see the cross-references at the end for the -current-state assessment and the per-subsystem detail. - ---- - -## 1. Threat Model - -What changed in the AI era: - -- **Scale of variant analysis.** A CVE in OpenSSL becomes a systematic - search across all C codebases within hours, not months. -- **Semantic fuzzing.** Coverage-guided fuzzers primed with - LLM-generated test inputs that understand grammars and protocol state - machines, not just byte-level mutation. -- **Automated exploit-chain construction.** Public PoCs assembled from - a CVE description plus a bug-class library, with minimal human time. -- **Variant hunting across forks and history.** Regression detection - across git history at scale; the same bug introduced years apart in - different forks is now findable in a single pass. -- **Reverse engineering at scale.** Binary analysis that previously - required expert manual work is now routine for any tool. - -What did not change: - -- **Logic bugs in unfamiliar domains** still benefit from human - reasoning more often than AI bug-hunters expect — though the gap is - closing. -- **Stated invariants and types are easier to verify than to evade.** - Every spec you write is a wall AI must climb, not just speculate - around. Refutation requires a counterexample; speculation generalises. -- **Defense via small, well-defined trust boundaries scales linearly; - "harden everything everywhere" does not.** - -The strategic implication: the language posture that wins is not "fewer -bugs" (a losing race at scale) but **bugs that exist cannot compose -into exploits, cannot reach interesting capabilities, and cannot evade -audit.** - ---- - -## 2. Strategic Posture — Three Axes - -Every design choice should ladder up to at least one of: - -**A. Eliminate bug classes by construction.** A bug the language makes -structurally impossible cannot be found by any analysis. This compounds: -each eliminated class is a permanent reduction in the adversary's -effective search space. - -**B. Break the exploit chain.** Even when individual bugs exist, an -exploit needs them to compose into a chain that reaches an interesting -capability. Capability discipline, effects in types, and structural CFI -all attack the chain rather than the bugs. - -**C. Shrink the trust boundary.** A small, well-defined TCB that is -heavily verified outperforms a large, casually-trusted runtime. Finding -a bug in 5 KLOC of audited interpreter is hard; finding a bug in 5 MLOC -of pragmatic runtime is easy. - ---- - -## 3. Core Principles - -Ranked roughly by force, each with Jerboa's current status. - -### 3.1 Eliminate, Don't Detect - -Memory safety, no null, no unchecked integer overflow, no raw pointers, -no implicit type conversions, no buffer-of-bytes-pretending-to-be-typed-data. - -A bug the language makes impossible cannot be found. Memory safety -alone eliminates 60-70% of historical C CVEs. - -**Jerboa today:** Memory safety via Chez Scheme — strong. `#f` instead -of null where idiomatic, but null-equivalents still creep into some -APIs. Integer overflow is checked in fixnum arithmetic; bignum -promotion is automatic. - -**Gap:** Some standard idioms (untyped hash tables, association lists, -raw bytevectors threaded through layers) re-introduce shapes that -bypass static reasoning. Encouraging `defstruct` / `defrecord` over -ad-hoc hash maps tightens this without language changes. - -### 3.2 Capabilities, Not Ambient Authority - -A function with no file-system capability in scope must be **unable** -to open files, not merely discouraged from doing so. Object-capability -discipline (Pony, E, Caja) is the strongest single multiplier: an RCE -in module X stops being equivalent to an RCE everywhere. - -**Jerboa today:** `(std capability)` implements the model, with -attenuation and revocation. See `capability.md`. - -**Gap:** Adoption is opt-in. The default prelude exposes ambient -authority (`open-output-file`, `system`, etc.). Until the **default** -is capability-required and ambient authority is **opt-in**, the model -doesn't bite. - -**Convergence path:** Promote `(jerboa prelude safe)` (the existing -113-binding allowlist documented in `ai-threat.md`) to the production -default for new projects. Make ambient-authority imports explicit, -greppable, and audited. Add a static check that production code does -not import the unsafe prelude. - -### 3.3 Effects in the Type System - -Function signatures should declare what side effects are performed -(fs-read, fs-write, network, exec, alloc). AI-generated code then -cannot sneak in effects without changing types that reviewers see. - -**Jerboa today:** `(std effect)` provides algebraic effects with -handlers. See `effects.md`. - -**Gap:** Like capabilities, effect declarations are opt-in rather than -required. A function that performs IO without declaring it is currently -legal. - -**Convergence path:** Move toward effect inference at compile time, -with declared signatures required on module boundaries. Initially a -lint, eventually a build-time error in strict mode. - -### 3.4 Structural Control-Flow Integrity via WASM - -WebAssembly's separated code/data spaces, typed indirect calls, and -absence of `ret` make ROP **structurally impossible** — not merely -mitigated. Compiling security-critical components to WASM gives this -for free without per-platform hardware work. - -**Jerboa today:** The wasm runtime infrastructure exists — -`(std wasm)`, `jerboa-native-rs` embeds `wasmi`. `secure.md` lays out -a multi-phase rollout. `jerboa-websearch` has shipped wasm-sandboxed -HTML parsing (`src/jerbsearch/sandbox/html-parse.ss`) — the canonical -working example. - -**Gap:** Adoption is per-project and incomplete. `jerboa-dns` currently -parses untrusted UDP wire bytes in native Scheme despite `secure.md` -listing the DNS-parser-in-wasm migration as Phase 1b. This is the -**canonical "case-by-case drift" failure**: the pattern exists, the -default doesn't, and the maintainer's mental model drifts ahead of the -code. - -**Convergence path:** Push wasm-sandboxed parsing into the standard -library as the default for untrusted-byte parsers. A `(std parse -sandboxed)` API that takes bytes and returns a typed parse result, -with the wasm wrapping invisible to the caller. - -### 3.5 First-Class Parser Primitives with Proven Bounds - -WUFFS is the prototype: a domain-specific language for byte parsers -where the compiler proves array bounds, integer bounds, and termination. -Ad-hoc `string-split + index-lookup` parsing is exactly where AI -variant-hunters score easy hits — every protocol parser written this -way has the same bug class. A standard parser story eliminates the -variance. - -**Jerboa today:** No equivalent. Protocol parsers in the ecosystem are -hand-rolled bytevector manipulation (see -`~/mine/jerboa-dns/lib/jerboa-dns/protocol.sls`). - -**Convergence path:** Adopt or build a parser-combinator library with -explicit bounds tracking. Long-term, a Jerboa-flavored WUFFS — a -subset language that compiles to safe bytevector operations with -compile-time proofs. - -### 3.6 Refinement Types / Contracts Default-On - -Function pre/post-conditions and refinement predicates raise AI's bar -significantly: instead of "this might be wrong, let me speculate," it -must produce a counterexample to a stated invariant. Speculation -generalises; counterexamples don't. - -**Jerboa today:** `define/ct` and `lambda/ct` provide contract-typing -per-function. Used inconsistently across stdlib and downstream code. - -**Convergence path:** Make contract-typing on by default for new code -in `(std ...)`. Surface uncovered-boundary warnings in the linter. -Long-term: SMT-backed refinement checking for numerically-precise -invariants. - -### 3.7 Linear / Affine Types for Resources - -Resources (file handles, sockets, capability tokens, cryptographic -keys) consumed-on-use eliminate whole categories: use-after-close, -double-spend, TOCTOU on capability checks. - -**Jerboa today:** No linearity in the type system. `with-resource` -provides scoped cleanup but does not prevent the handle escaping the -scope. - -**Convergence path:** Introduce affine annotations in `define/ct` for -resource-bearing values. Mark capability tokens and file handles as -affine in the standard library. - -### 3.8 Small, Formally Specified TCB - -The smaller and more deliberately scoped the trust base, the more -tractably it can be audited and the less surface there is for runtime -bugs. Large ambient runtimes (Scheme + libkernel + Rust runtime) have -many gadgets, many idiom bugs, and many attack-relevant primitives. - -**Jerboa today:** Chez Scheme + libkernel is ~1 MB of native code. Per -`secure.md`, this is the primary ROP gadget source. - -**Convergence path:** `secure.md` enumerates the options — harden Chez -with CET/SHSTK and hardening CFLAGS, switch security-critical paths to -WASM so the TCB they depend on shrinks to the wasm runtime, or in the -extreme, target a minimal interpreter (s7) for high-assurance tools. - ---- - -## 4. Anti-Patterns - -Things that look like security wins but aren't, or that quietly defeat -the principles above. - -### 4.1 Obscurity as Defense - -"Jerboa is niche, so AI is bad at it." True today, false in 18 months. -Worse: your own LLM tooling suffers symmetrically — every advantage to -the adversary is a disadvantage to you. `jerboa-lora` is the workaround, -but it's a tax, not a structural defense. - -### 4.2 Liberal FFI - -Every `foreign-procedure` is a hole through which the safety model -leaks. The "Fearless FFI" doc (`ffi.md`) introduces a DSL that is much -safer than raw bindings, but the **existence** of the unsafe primitives -in the unrestricted prelude means a future contributor or LLM can reach -for them without noticing the cost. FFI should require an `unsafe` -capability gated and visible at the call site. - -### 4.3 Big Language Surface - -Every feature is a future idiom bug. Macros, reflection, dynamic -loading, `eval` — each is a place where AI variant-hunters will look -for bypass patterns. Add features grudgingly; remove or restrict where -possible. - -### 4.4 Convenience Features That Shadow Safer Alternatives - -If the safe way is verbose, no one uses it under deadline pressure — -and LLMs trained on existing code generate the convenient idiom, not -the safe one. **The easy default must be the safe default.** If -`string-split` is one line and `parse-with-bounds` is six, parsers will -be written with `string-split`. - -### 4.5 Runtime-Only Safety - -A runtime check is something AI can fuzz until it triggers, then write -an exploit around. A compile-time impossibility is something AI cannot -interact with at all. Prefer types, refinements, and capabilities over -runtime assertions wherever feasible. - -### 4.6 Ad-Hoc Per-Project Security Architecture - -The `jerboa-dns` case study makes this concrete: the project's mental -model said "wasm-sandboxed parsing," the code said "native Scheme on -raw UDP bytes." Case-by-case decisions about which security primitives -to deploy drift under time pressure and LLM-assisted feature work. -**The architecture has to encode the security decision, not the -maintainer's intent.** - ---- - -## 5. Where Jerboa Stands — Honest Inventory - -| Principle | Status | Notes | -|------------------------|:--------:|--------------------------------------------------| -| Memory safety | ✓ Strong | Chez Scheme foundation | -| Capabilities | ◐ Partial| `(std capability)` exists; adoption opt-in | -| Effects | ◐ Partial| `(std effect)` exists; declarations opt-in | -| WASM CFI | ◐ Partial| Runtime exists; deployment per-project | -| Parser proofs | ✗ Missing| Hand-rolled bytevector parsing dominant | -| Refinement / contracts | ◐ Weak | `define/ct` exists; not the default | -| Linear / affine | ✗ Missing| `with-resource` is scoped cleanup, not linearity | -| Small TCB | ◐ Partial| Chez + libkernel large; `secure.md` plans work | -| Safe-by-default prelude| ◐ Partial| `(jerboa prelude safe)` exists, not yet default | - -The pattern is consistent: many of the right primitives exist; they -are not the default. **The work for the next era is less "build new -mechanisms" than "make the existing safe ones the path of least -resistance."** - ---- - -## 6. Realistic Convergence Path - -In rough priority order: - -1. **Finish the in-flight wasm migration.** `secure.md` Phases 1a/1b — - embed wasmi properly, port the DNS parser to Rust→WASM. Take - `jerboa-dns` off the "claims wasm, isn't" list. - -2. **Make wasm-sandboxed parsing the stdlib default.** A - `(std parse sandboxed)` API where wasm wrapping is invisible to the - caller. Hand-rolled parsers become the explicit opt-out, not the - implicit default. - -3. **Gate FFI behind an explicit unsafe-tagged surface.** - `(jerboa prelude)` does not export `foreign-procedure`. Code wanting - FFI imports `(jerboa prelude unsafe)` or equivalent — auditable via - `grep`. - -4. **Promote the safe prelude to the production default.** - `(jerboa prelude safe)` exists already; make it the default for new - projects via `jerbuild` templates. Existing projects opt in as they - migrate. - -5. **Add a contract / spec coverage pass.** Extend `define/ct` to be on - by default in stdlib modules. Surface contract coverage in - `jerboa_security_audit`. Long-term: SMT-backed refinement checking - on numeric invariants. - -6. **Build the linear-resource discipline.** Mark capability tokens, - file handles, and crypto keys as affine in stdlib types. Lint for - escape via `define/ct` annotations. - -7. **Audit `secure.md` for other "claimed but not shipped" items.** - The `jerboa-dns` gap suggests other Phase-1 work may also be - deferred without the gap being visible from the top of the project. - -Each step is a refactor of an already-working system, not a research -project. The cumulative effect over a year of incremental work is -substantial; the cost of any single step is bounded. - ---- - -## 7. Reference Language Comparison - -| Language | Mem safety | Capabilities | Effects in types | Structural CFI | Parser proofs | Refinement | Linear | -|-------------------|:---------:|:------------:|:----------------:|:--------------:|:-------------:|:----------:|:--------:| -| C / C++ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | -| Rust | ✓ | ✗ | ✗ | partial | ✗ | ✗ | affine | -| Haskell | ✓ | ✗ | partial (IO) | ✗ | ✗ | partial (LH)| ✗ | -| Pony | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ref-caps | -| Koka / Eff | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | -| WUFFS | ✓ | ✗ | ✗ | ✗ | ✓ | partial | ✗ | -| Idris / Lean | ✓ | partial | ✓ | ✗ | partial | ✓ | ✓ | -| WebAssembly | ✓ | partial | ✗ | ✓ | ✗ | ✗ | ✗ | -| **Jerboa today** | ✓ | opt-in | opt-in | via wasm (per-project) | ✗ | opt-in | ✗ | -| **Jerboa target** | ✓ | default | default | default | ✓ | default | ✓ | - -No shipping language combines all of the principles. The Jerboa -opportunity is to converge the combination over time as a series of -refactors, not a research project. Each row of the "Jerboa target" line -maps to a concrete step in §6. - ---- - -## 8. Cross-References - -- `ai-threat.md` — Current-state security assessment; complements this - document's forward-looking framing -- `capability.md` — `(std capability)` API and patterns for the - object-capability model -- `effects.md` — `(std effect)` API for algebraic effects -- `ffi.md` — Current FFI surface and the "Fearless FFI" DSL -- `../secure.md` — Concrete binary-hardening roadmap (Chez / musl / - wasm-runtime work) -- `architecture-split.md` — Chez / Rust / wasm split architecture - ---- - -## 9. Maintenance - -This document is forward-looking. As convergence work in §6 lands, -update §5's status table and shorten or remove the gap notes in §3. -When the "Jerboa today" row of the §7 comparison table catches up to -"Jerboa target," collapse them into a single row and write a new -target reflecting the next horizon. - -Drift between this document and the code is itself a smell — if the -status table claims a default that the code doesn't enforce (as -`jerboa-dns` did with wasm sandboxing), that's the same case-by-case -drift §4.6 warns about, applied to documentation. Fix the code or fix -the doc. --- a/docs/JERBOA-LANG.md +++ b/docs/JERBOA-LANG.md @@ -18,8 +18,8 @@ written in `.ss` files. The `.sls` files are implementation internals. ``` Run with: -``` -scheme --libdirs lib --script hello.ss +```bash +jerboa run hello.ss ``` One import gives you the entire language: @@ -125,7 +125,7 @@ Both forms work in `import`: ### Heredoc Strings -```scheme +```scheme-skip #<<END This is a multi-line string literal. --- a/docs/actor-model.md +++ b/docs/actor-model.md @@ -1,3827 +1,183 @@ -# Jerboa Actor Model: Complete Implementation Guide +# Actor Model -This document is a step-by-step implementation guide for building a production-quality -actor system on Chez Scheme 10.4+ (threaded build, `ta6le`). Each layer is independently -implementable and testable. A lesser model can implement this by following the layers in -order — do not skip ahead. +Jerboa ships a production actor stack under `(std actor)`. The facade exports +the core actor API, ask/reply protocols, supervision trees, registry helpers, +work-stealing scheduler controls, and distributed actor references. -**Validated against**: Chez Scheme 10.4.0, threaded build, Linux x86_64. - ---- - -## Design Philosophy - -**Goals** (what makes this better than Gerbil's `:std/actor`): - -1. **Clean layer separation** — each layer is independently importable and testable. - Gerbil mixes local spawn, remote RPC, filesystem deployment, and admin auth into - a single 400-symbol namespace. Here each layer is a separate library. - -2. **No shimming** — built directly on Chez's native OS threads, not green threads. - Every primitive maps directly to a Chez or OS concept. - -3. **Native serialization** — use Chez's built-in `fasl-write`/`fasl-read` for - distributed transport. Any Scheme value (records, vectors, bytevectors, symbols, - numbers, booleans, pairs) is automatically serializable. No separate serialization - library needed. - -4. **OTP-style supervision** — Erlang-proven restart strategies (one-for-one, - one-for-all, rest-for-one) with max-intensity/period restart limiting. - -5. **Location transparency** — `(send actor-ref msg)` works whether `actor-ref` is - local or remote. The caller does not need to know. - -6. **Typed protocols via macros** — `defprotocol` generates message structs and - typed dispatch. Less boilerplate than Gerbil's `defmessage` + `defcall-actor`. - -7. **Gradual complexity** — Layers 3-6 (local actors + supervision) are useful - without Layer 2 (work-stealing) or Layer 7 (distributed). Build and ship each - layer independently. - -**Non-goals**: -- Full Gerbil compatibility (we implement what real programs use, not every symbol) -- Green threads / continuations (real OS threads are simpler and SMP-safe) -- Hot code loading (out of scope) - ---- - -## Quick Start (TL;DR) - -To get a working actor system in 5 steps: - -```scheme -(import (chezscheme) (jerboa core) - (std actor core) - (std actor protocol) - (std actor supervisor) - (std actor registry) - (std actor scheduler)) - -;; 1. Start the thread pool -(define sched (scheduler-start! (make-scheduler (cpu-count)))) -(set-actor-scheduler! (lambda (thunk) (scheduler-submit! sched thunk))) - -;; 2. Start the name registry -(start-registry!) - -;; 3. Define a protocol -(defprotocol counter - (increment n) - (get-value -> value) - (reset)) - -;; 4. Implement the actor -(define (make-counter initial) - (let ([n initial]) - (spawn-actor - (lambda (msg) - (with-ask-context msg - (lambda (actual) - (cond - [(counter:increment? actual) - (set! n (+ n (counter:increment-n actual)))] - [(counter:get-value? actual) - (reply n)] - [(counter:reset? actual) - (set! n initial)]))))))) - -;; 5. Supervise it -(define app - (start-supervisor 'one-for-one - (list (make-child-spec 'counter - (lambda () (make-counter 0)) - 'permanent 5.0 'worker)) - 10 5)) - -;; Use it -(let ([ref (caddr (car (supervisor-which-children app)))]) - (register! 'counter ref)) - -(counter:increment! (whereis 'counter) 42) -(display (counter:get-value?! (whereis 'counter))) ;; => 42 - -;; Shutdown -(scheduler-stop! sched) -``` - ---- - -## Chez Scheme Primitives Reference - -Every primitive used in this guide. Know these before implementing. - -### Threading (Chez 10, threaded build) -| Primitive | Description | -|------------------------------------|------------------------------------------------------------------------------| -| `(fork-thread thunk)` | Starts a new OS thread immediately, returns thread-id | -| `(get-thread-id)` | Returns current thread's integer id | -| `(make-mutex)` | Creates a new mutex | -| `(mutex-acquire mutex)` | Blocks until mutex acquired | -| `(mutex-release mutex)` | Releases a mutex | -| `(with-mutex mutex body ...)` | Acquires, evaluates body, releases (even on exception) | -| `(make-condition)` | Creates a condition variable | -| `(condition-wait cond mutex)` | Atomically releases mutex and blocks on condition; re-acquires mutex on wake | -| `(condition-wait cond mutex time)` | Same but with timeout; returns `#f` on timeout, `#t` if signaled | -| `(condition-signal cond)` | Wakes one waiting thread | -| `(condition-broadcast cond)` | Wakes all waiting threads | -| `(make-thread-parameter default)` | Creates a thread-local parameter (SMP-safe, no global lock) | - -### Data Structures -| Primitive | Description | -|----------------------------------|--------------------------------------------| -| `(make-vector n init)` | Creates a vector of size n | -| `(make-eq-hashtable)` | Creates a hashtable with `eq?` comparison | -| `(make-hashtable hash equiv)` | Creates a hashtable with custom hash/equiv | -| `(hashtable-set! ht key val)` | Insert/update | -| `(hashtable-ref ht key default)` | Lookup with default | -| `(hashtable-delete! ht key)` | Remove | -| `(hashtable-keys ht)` | Returns vector of keys | - -### Serialization -| Primitive | Description | -|-----------|-------------| -| `(fasl-write obj port)` | Serializes any Scheme value (records, vectors, etc.) to binary port | -| `(fasl-read port)` | Deserializes from binary port | -| `(open-bytevector-output-port)` | Returns `(values port get-bytevector-proc)` | -| `(open-bytevector-input-port bv)` | Creates input port from bytevector | - -### Time -| Primitive | Description | -|-----------|-------------| -| `(make-time type nanoseconds seconds)` | Creates a time object. Use `'time-duration` for durations | -| `(current-time)` | Returns `time-utc` object | -| `(time-second t)` | Extracts seconds from a time object | -| `(time-nanosecond t)` | Extracts nanoseconds from a time object | - -**WARNING**: `time->seconds` does NOT exist in Chez. It is in `(std srfi srfi-19)`. -To convert a time object to a float without SRFI-19: -```scheme -(define (time->float t) - (+ (time-second t) (/ (time-nanosecond t) 1000000000.0))) -``` - -### Other -| Primitive | Description | -|-----------|-------------| -| `(random n)` | Returns random integer in [0, n) | -| `(filter pred lst)` | Standard R6RS filter | -| `(define-record-type ...)` | R6RS record types with protocol, sealed, etc. | -| `(guard (var [test expr] ...) body ...)` | R6RS exception handling | - -**WARNING**: `match` is NOT built-in Chez. Import from `(jerboa core)`. -**WARNING**: `cpu-count` does NOT exist in Chez. Read from `/proc` or use a constant: -```scheme -(define (cpu-count) - (or (let ([p (open-input-file "/proc/cpuinfo")]) - (let loop ([n 0]) - (let ([line (get-line p)]) - (cond - [(eof-object? line) (close-port p) n] - [(and (>= (string-length line) 9) - (string=? (substring line 0 9) "processor")) - (loop (fx+ n 1))] - [else (loop n)])))) - 4)) ;; fallback -``` - ---- - -## Architecture Overview - -``` -┌──────────────────────────────────────────────────────┐ -│ Layer 7: Distributed Transport │ -│ lib/std/actor/transport.sls │ -│ TCP+TLS, fasl serialization, location transparency │ -├──────────────────────────────────────────────────────┤ -│ Layer 6: Registry │ -│ lib/std/actor/registry.sls │ -│ Named actors, whereis, register, unregister │ -├──────────────────────────────────────────────────────┤ -│ Layer 5: Supervision Trees │ -│ lib/std/actor/supervisor.sls │ -│ OTP strategies, restart intensity, child specs │ -├──────────────────────────────────────────────────────┤ -│ Layer 4: Protocol System │ -│ lib/std/actor/protocol.sls │ -│ defprotocol, ask, tell, call, pattern dispatch │ -├──────────────────────────────────────────────────────┤ -│ Layer 3: Actor Core │ -│ lib/std/actor/core.sls │ -│ spawn-actor, send, receive, self, dead letters │ -├──────────────────────────────────────────────────────┤ -│ Layer 2: Scheduler │ -│ lib/std/actor/scheduler.sls │ -│ Work-stealing thread pool, lightweight tasks │ -├──────────────────────────────────────────────────────┤ -│ Layer 1: Data Structures │ -│ lib/std/actor/mpsc.sls — MPSC queue (mailbox) │ -│ lib/std/actor/deque.sls — Work-stealing deque │ -├──────────────────────────────────────────────────────┤ -│ Foundation (already exists in Jerboa) │ -│ (std misc channel) — bounded channels + select │ -│ (std misc thread) — Gambit thread API │ -│ (std task) — task groups + futures │ -│ (std net ssl) — TCP+TLS via jerboa-ssl (legacy) │ -│ (jerboa core) — match, def, defstruct │ -└──────────────────────────────────────────────────────┘ -``` - -**Implementation order**: Layer 1 → Layer 3 → Layer 4 → Layer 5 → Layer 6 → Layer 2 → Layer 7. -Layer 2 (scheduler) can be deferred — Layers 3-6 work fine on 1:1 OS threads initially. - ---- - -## Layer 1A: MPSC Queue (`lib/std/actor/mpsc.sls`) - -### Purpose - -Each actor has a mailbox. Multiple threads (producers) can send messages to it -concurrently. Only the actor's own thread (consumer) reads from it. -This is the Multi-Producer Single-Consumer (MPSC) pattern. - -### Data Structure: Two-Lock Linked List - -We use a Michael-Scott style two-lock linked list: one lock for the tail (producers) -and one lock for the head (consumer). This minimizes contention because producers -never block the consumer and vice versa. - -This is simpler and more practical for Chez than a lock-free Michael-Scott queue -(which would require `compare-and-swap` via FFI C shims — Chez does not expose CAS natively). - -### Critical Design: Signaling Without Deadlock - -The original design had a subtle deadlock risk: signaling `not-empty` while holding -`tail-mutex`. If the consumer holds `head-mutex` and tries to signal or if the -producer tries to acquire `head-mutex` while holding `tail-mutex`, you can deadlock -when another thread does the reverse. - -**Solution**: Use a single condition variable protected by `head-mutex` only. The -producer signals by acquiring `head-mutex` briefly AFTER releasing `tail-mutex`. -This ensures no nested lock acquisition. - -```scheme -#!chezscheme -(library (std actor mpsc) - (export - make-mpsc-queue - mpsc-queue? - mpsc-enqueue! ;; producer: add to tail - mpsc-dequeue! ;; consumer: remove from head (blocks if empty) - mpsc-try-dequeue! ;; consumer: remove or return (values #f #f) immediately - mpsc-empty? ;; peek (approximate — only safe from consumer thread) - mpsc-close! ;; signal no more messages - mpsc-closed?) - (import (chezscheme)) - - ;; Node in the linked list - (define-record-type mpsc-node - (fields - (mutable value) ;; the message, or 'sentinel for dummy head - (mutable next)) ;; next node or #f - (protocol - (lambda (new) - (lambda (val) (new val #f)))) - (sealed #t)) - - (define-record-type mpsc-queue - (fields - (mutable head) ;; points to dummy node; consumer reads head.next - (mutable tail) ;; points to last real node (or dummy if empty) - (immutable head-mutex) ;; consumer lock (also protects condition variable) - (immutable tail-mutex) ;; producer lock - (immutable not-empty) ;; condition: signaled when item enqueued - (mutable closed?)) - (protocol - (lambda (new) - (lambda () - (let ([dummy (make-mpsc-node 'sentinel)]) - (new dummy dummy - (make-mutex) (make-mutex) - (make-condition) - #f))))) - (sealed #t)) - - ;; Producer: enqueue a value - ;; Lock only the tail — does not interfere with consumer reading head. - ;; Signal the consumer AFTER releasing tail-mutex to avoid nested locking. - (define (mpsc-enqueue! q val) - (let ([node (make-mpsc-node val)]) - (with-mutex (mpsc-queue-tail-mutex q) - (when (mpsc-queue-closed? q) - (error 'mpsc-enqueue! "queue is closed")) - (mpsc-node-next-set! (mpsc-queue-tail q) node) - (mpsc-queue-tail-set! q node)) - ;; Signal consumer OUTSIDE tail-lock (head-mutex acquired briefly) - (with-mutex (mpsc-queue-head-mutex q) - (condition-signal (mpsc-queue-not-empty q))))) - - ;; Consumer: dequeue, blocking if empty - (define (mpsc-dequeue! q) - (with-mutex (mpsc-queue-head-mutex q) - (let loop () - (let ([next (mpsc-node-next (mpsc-queue-head q))]) - (cond - [next - ;; Advance dummy head to the first real node - ;; The old head is discarded; the real node becomes the new dummy - (let ([val (mpsc-node-value next)]) - (mpsc-queue-head-set! q next) - (mpsc-node-value-set! next 'sentinel) ;; help GC - val)] - [(mpsc-queue-closed? q) - (error 'mpsc-dequeue! "queue closed and empty")] - [else - (condition-wait (mpsc-queue-not-empty q) - (mpsc-queue-head-mutex q)) - (loop)]))))) - - ;; Consumer: try dequeue without blocking - ;; Returns (values val #t) if successful, (values #f #f) if empty - (define (mpsc-try-dequeue! q) - (with-mutex (mpsc-queue-head-mutex q) - (let ([next (mpsc-node-next (mpsc-queue-head q))]) - (cond - [next - (let ([val (mpsc-node-value next)]) - (mpsc-queue-head-set! q next) - (mpsc-node-value-set! next 'sentinel) - (values val #t))] - [else - (values #f #f)])))) - - (define (mpsc-empty? q) - ;; Approximate: safe only from the consumer thread. - ;; Reads head.next without lock — may see stale data from producers. - (not (mpsc-node-next (mpsc-queue-head q)))) - - (define (mpsc-close! q) - (with-mutex (mpsc-queue-tail-mutex q) - (mpsc-queue-closed?-set! q #t)) - ;; Wake all blocked consumers - (with-mutex (mpsc-queue-head-mutex q) - (condition-broadcast (mpsc-queue-not-empty q)))) - - ) ;; end library -``` - -**Why `with-mutex` instead of manual acquire/release**: `with-mutex` is a Chez -built-in that uses `dynamic-wind` to guarantee the mutex is released even if an -exception occurs inside the body. Manual acquire/release leaks the lock on exception. - -**Test file**: `tests/test-actor-mpsc.ss` -- Enqueue from 10 threads simultaneously, dequeue from 1 thread — verify all messages received -- `try-dequeue` on empty queue returns `(values #f #f)` -- Close while consumer is blocked — consumer gets error -- Ordering: messages from a single producer arrive in FIFO order - ---- - -## Layer 1B: Work-Stealing Deque (`lib/std/actor/deque.sls`) - -### Purpose - -The scheduler (Layer 2) gives each worker thread its own double-ended queue of tasks. -The owner thread pushes/pops from the bottom. Idle workers steal from the top of -other workers' deques. This is the Chase-Lev work-stealing deque. - -### Implementation Note - -A fully lock-free Chase-Lev deque requires `compare-and-swap` (CAS) on memory -words — an operation Chez does not expose natively. Two options: - -**Option A (Recommended for initial implementation)**: Use a single mutex per deque. -The deque is mostly uncontended (owner push/pop), and stealing is rare. A mutex -is fast enough when not under heavy contention. - -**Option B (For high-throughput production)**: Add a C shim: -```c -// support/atomic.c -#include <stdatomic.h> -#include <stdint.h> - -// Returns 1 if swap succeeded, 0 if not -int jerboa_cas_int64(int64_t *ptr, int64_t expected, int64_t desired) { - return atomic_compare_exchange_strong( - (_Atomic int64_t*)ptr, &expected, desired); -} - -void jerboa_atomic_store_int64(int64_t *ptr, int64_t val) { - atomic_store((_Atomic int64_t*)ptr, val); -} - -int64_t jerboa_atomic_load_int64(int64_t *ptr) { - return atomic_load((_Atomic int64_t*)ptr); -} -``` -Compile: `gcc -shared -fPIC -O2 -o libjerboa-atomic.so support/atomic.c` -Load: `(load-shared-object "./libjerboa-atomic.so")` -Use: `(define cas-int64 (foreign-procedure "jerboa_cas_int64" (void* integer-64 integer-64) int))` - -The document describes the mutex-based version. Upgrading to lock-free is a drop-in -replacement at the deque level — the scheduler above does not change. - -```scheme -#!chezscheme -(library (std actor deque) - (export - make-work-deque - work-deque? - deque-push-bottom! ;; owner pushes a task - deque-pop-bottom! ;; owner pops (LIFO — locality of reference) - deque-steal-top! ;; thief steals (FIFO — oldest tasks first) - deque-empty? - deque-size) - (import (chezscheme)) - - ;; Circular buffer that grows as needed - (define-record-type work-deque - (fields - (mutable buf) ;; vector of tasks - (mutable bottom) ;; owner's end (push/pop here) - (mutable top) ;; thief's end (steal from here) - (immutable mutex)) - (protocol - (lambda (new) - (lambda () - (new (make-vector 64 #f) 0 0 (make-mutex))))) - (sealed #t)) - - (define (deque-capacity d) (vector-length (work-deque-buf d))) - - (define (deque-size d) - (with-mutex (work-deque-mutex d) - (let ([b (work-deque-bottom d)] - [t (work-deque-top d)]) - (if (fx>= b t) (fx- b t) 0)))) - - (define (deque-empty? d) - (with-mutex (work-deque-mutex d) - (fx<= (work-deque-bottom d) (work-deque-top d)))) - - ;; Grow buffer when full (called under lock) - (define (deque-grow! d) - (let* ([old (work-deque-buf d)] - [old-cap (vector-length old)] - [new-cap (fx* old-cap 2)] - [new-buf (make-vector new-cap #f)] - [top (work-deque-top d)] - [bottom (work-deque-bottom d)]) - (do ([i top (fx+ i 1)]) - ((fx= i bottom)) - (vector-set! new-buf (fxmod i new-cap) - (vector-ref old (fxmod i old-cap)))) - (work-deque-buf-set! d new-buf))) - - ;; Owner pushes a task to the bottom - (define (deque-push-bottom! d task) - (with-mutex (work-deque-mutex d) - (let ([b (work-deque-bottom d)]) - (when (fx>= (fx- b (work-deque-top d)) (fx- (deque-capacity d) 1)) - (deque-grow! d)) - (vector-set! (work-deque-buf d) (fxmod b (deque-capacity d)) task) - (work-deque-bottom-set! d (fx+ b 1))))) - - ;; Owner pops from the bottom (LIFO — most recently pushed task first) - ;; Returns the task or #f if empty - (define (deque-pop-bottom! d) - (with-mutex (work-deque-mutex d) - (let ([b (work-deque-bottom d)] - [t (work-deque-top d)]) - (if (fx> b t) - (let ([new-b (fx- b 1)]) - (work-deque-bottom-set! d new-b) - (let ([task (vector-ref (work-deque-buf d) - (fxmod new-b (deque-capacity d)))]) - (vector-set! (work-deque-buf d) (fxmod new-b (deque-capacity d)) #f) - task)) - #f))))