Phase 4 implementation plan: The Definitive Scheme
ober
6dd401dbd5d4586237ef07b57e5f0fd09c15d24e
--- a/docs/implement.md +++ b/docs/implement.md @@ -1,1168 +1,1271 @@ -# Jerboa Implementation Plan: Phase 3 — Production Excellence +# Jerboa Implementation Plan: Phase 4 — The Definitive Scheme -## Status: COMPLETE ✓ +## Status: Phase 3 Complete, Phase 4 Proposed -All Phase 3 sub-phases have been implemented and pushed (2026-03-11): +Phases 1-3 established Jerboa as a compelling Gerbil-on-Chez implementation with 138+ modules and 1,524+ tests. Phase 4 pushes Jerboa beyond any existing Scheme into territory occupied by Rust, Go, Haskell, and OCaml — while retaining the macro system that none of them have. -| Phase | Libraries | Tests | Commit | -|-------|-----------|-------|--------| -| 3a: Observability | 5 | 131 | `cf41c35` | -| 3b: Advanced Networking | 5 | 129 | `197ba30` | -| 3c: Build & Package Tooling | 5 | 119 | `4e513a9` | -| 3d: Language Extensions | 5 | 158 | `644a501` | -| 3e: WASM Target | 3 | 100 | `9b8b16a` | -| **Total** | **23** | **637** | | - ---- - -## Phase 2 (Previous) — The Superior Scheme: COMPLETE ✓ +### Where We Stand -All Phase 2 sub-phases were implemented and pushed (2026-03-11): - -| Phase | Libraries | Tests | Commit | +| Phase | Libraries | Tests | Status | |-------|-----------|-------|--------| -| 2a: Foundations | 7 | 111 | `316cb5e` | -| 2b: Performance | 6 | 101 | `691e709` | -| 2c: Type System | 4 | 111 | `4e99988` | -| 2d: Systems & Distributed | 6 | 105 | `53794bb` | -| 2e: Ecosystem | 5 | 113 | `e1a0a5e` | -| **Total** | **28** | **541** | | +| 1: Core | 51 | 289 | Complete | +| 2: Advanced | 28 | 541 | Complete | +| 3: Production | 23 | 637 | Complete | +| **4: Definitive** | **~65** | **~1,200** | **Proposed** | --- -## Where We Are (After Phase 3) - -Jerboa now has 110+ modules, 1,178+ tests, and covers: +## Design Philosophy -**Phase 2 additions**: PGO, devirtualization, compile-time regex, continuation mark optimization, GADTs, type classes, linear types, effect typing, M:N scheduler, async streams, Raft consensus, zero-copy networking, process supervision, connection pooling, property-based testing, doc generator, S-Expr config, gRPC, sorted maps, persistent vectors, persistent hash maps, channel select, error messages, derive system, memory-mapped I/O, REPL enhancements. +Jerboa's architectural advantage is that `defstruct` maps to native Chez records, methods dispatch via `eq-hashtable`, and macros compile to idiomatic Chez code that cp0 can fully optimize. Phase 4 leverages this foundation to build features no other Scheme has, drawing from the best ideas across programming languages: -**Phase 3 additions**: -- **Observability**: structured logging, Prometheus metrics, distributed tracing, health checks, circuit breakers -- **Advanced Networking**: WebSocket (RFC 6455), HTTP/2 framing + HPACK, DNS wire format, rate limiting, HTTP router -- **Build & Package**: semantic versioning + dep resolver, lockfiles, hot code reload, sandboxed eval, cross-compilation config -- **Language Extensions**: SQL-like query DSL, data schema validation, data pipeline DSL, term rewriting, source linting -- **WASM Target**: binary format (LEB128, IEEE 754), Scheme→WASM compiler (i32 subset), stack-based interpreter +- **From Rust**: ownership tracking, borrow checking (via linear types), fearless concurrency +- **From Haskell**: type classes with coherence, kind system, deriving via generics +- **From OCaml 5**: direct-style effects with deep handlers, multicore domains +- **From Erlang/BEAM**: hot code swapping, distribution, process isolation +- **From Go**: channels with select, fast compilation, single-binary deployment +- **From Clojure**: persistent data structures, transducers, spec/schema +- **From Zig**: comptime execution, no hidden allocations in systems code +- **From Unison**: content-addressed code, structural editing -The original Phase 2 plan identified 25 additions; those are now complete. Phase 3 added 23 more libraries to cover the "production excellence" gap — the tooling, observability, and interoperability needed to deploy Jerboa in real systems. +The key insight: Chez Scheme's cp0 optimizer is so good that macro-generated code performs like hand-written C. Every feature compiles down through macros to native Chez — the macro system *is* the compiler, and it's user-extensible. --- -## Phase 2 Plan Details +## Track 1: Multicore Runtime — Beyond Erlang, Beyond Go -The following tracks were the Phase 2 design document (now fully implemented): - ---- +Jerboa already has actors, channels, STM, and a work-stealing scheduler. Phase 4 makes this the most sophisticated concurrency runtime in any Scheme. -## Track 1: Compiler Infrastructure — The Performance Moat +### 1.1 Engine-Based Preemptive Actor Scheduling -Chez Scheme has the best optimizing compiler in the Lisp world. Jerboa should be the language that gives users direct access to that power. No other Scheme exposes compile-time optimization hooks this way. - -### 1.1 Profile-Guided Optimization (PGO) - -Record type feedback from production runs, feed it back to the compiler. - -**What Chez gives us**: `cp0` (copy propagation pass 0) already does aggressive inlining and constant folding. With type profiles, we can tell it which branches to favor. +Chez Scheme has a built-in `make-engine` mechanism — preemptible computations with fuel-based scheduling. No other Scheme exposes this. Use it to build a true preemptive actor runtime where no actor can monopolize a worker thread. ```scheme -;; Instrument: record which types flow through each call site -(jerboa build myapp.ss --profile) -./myapp --workload production < data.txt -;; Produces myapp.profile — type histograms per call site - -;; Optimize: use the profile to specialize -(jerboa build myapp.ss --pgo myapp.profile -o myapp-fast) -;; Now (+ x y) at line 47 emits fx+ because the profile shows x,y are always fixnums +;; Spawn 1,000,000 actors — each gets fair CPU time slices +(define pool (make-actor-pool #:workers (cpu-count) #:fuel 10000)) + +(for-each + (lambda (i) + (spawn-actor pool + (lambda (self) + (actor-receive self + [('ping sender) (actor-send sender 'pong)] + [('compute n) (fib n)])))) ;; even long-running fib gets preempted + (iota 1000000)) ``` **Implementation**: -- `lib/std/dev/pgo.sls` — instrumentation macros that wrap call sites with type counters -- A `define-syntax` transformer that reads profile data at compile time and emits specialized code paths -- Integration with `(jerboa build)`: `--profile` flag instruments, `--pgo` flag applies -- Store profiles as FASL files (native Chez serialization, fast to read) +- Wrap each actor's message handler in `make-engine` with a configurable fuel count +- When fuel exhausts, engine suspends → actor goes back on the run queue +- Worker threads pop actors from work-stealing deques, run engine for one quantum +- Actor state = engine continuation + mailbox ref (~200 bytes per actor) +- Use Chez's `set-timer` + `timer-interrupt-handler` as the fuel mechanism -**Why this is unique**: No Scheme, no Lisp, no ML has PGO. Only C/C++ (GCC/LLVM), Go, and Rust (via LLVM) have it. Jerboa would be the first functional language with PGO. +**Why unique**: Go's goroutines aren't preemptible (they yield at function calls). Erlang's reduction counting is similar but tied to BEAM. Jerboa gets preemption via Chez engines on native threads — the best of both worlds. -**LOC**: ~500 +**Files**: `lib/std/actor/engine.sls` (~400 LOC) +**Tests**: 20 tests (preemption fairness, fuel exhaustion, million-actor spawn) -### 1.2 Whole-Program Devirtualization +### 1.2 Affinity-Based Scheduling and NUMA Awareness -When the compiler can see all implementations of a method, replace dynamic dispatch with a `cond` on the type. +For high-performance servers, schedule actors to the same core that owns their data. ```scheme -;; Before: runtime hashtable lookup -({area} shape) ;; → find-method → eq-hashtable-ref → call - -;; After (when only circle, rect, triangle implement area): -(cond - [(circle? shape) (circle-area shape)] ;; native record predicate, inlineable - [(rect? shape) (rect-area shape)] - [(triangle? shape) (triangle-area shape)] - [else (error 'area "no method" shape)]) +(define-actor-group db-actors + #:affinity 'core-pinned + #:workers 4) + +(define-actor-group compute-actors + #:affinity 'numa-local ;; keep on same NUMA node + #:workers (numa-node-count)) + +;; Actor-to-actor messages within same group avoid cross-core cache invalidation +(spawn-in-group db-actors (lambda (self) ...)) ``` **Implementation**: -- Collect all `bind-method!` calls during WPO's whole-program analysis -- For each method, if the set of implementing types is closed, emit a `cond` dispatch -- Chez's cp0 can then inline the accessor bodies if they're small -- Result: method call → record predicate check → inlined body. Two instructions. - -**Why this matters**: This is the optimization that makes Java's HotSpot fast (speculative devirtualization). Jerboa can do it statically at compile time because WPO sees the whole program. +- FFI to `sched_setaffinity` / `pthread_setaffinity_np` for core pinning +- NUMA topology discovery via `/sys/devices/system/node/` +- Per-group work-stealing deques — stealing prefers same NUMA node +- Thread-local allocator hints for NUMA-aware memory placement -**LOC**: ~400 +**Files**: `lib/std/actor/affinity.sls` (~300 LOC), `lib/std/os/numa.sls` (~200 LOC) +**Tests**: 15 tests -### 1.3 Compile-Time Partial Evaluation +### 1.3 Continuations as Serializable Values -Go beyond macros — let the compiler evaluate any pure expression at compile time. +Chez's one-shot continuations (`call/1cc`) can be serialized to bytevectors via `fasl-write`. This enables actor migration, distributed checkpointing, and time-travel debugging. ```scheme -(define-ct (fib n) - (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) - -(define answer (fib 30)) -;; At compile time: evaluates to 832040 -;; At runtime: (define answer 832040) — zero-cost constant +;; Checkpoint an actor's state +(define (checkpoint-actor actor) + (let ([cont (actor-continuation actor)] + [mailbox (actor-mailbox-snapshot actor)]) + (fasl-write (list cont mailbox) "actor-checkpoint.fasl"))) + +;; Restore on a different machine +(define (restore-actor path) + (let ([state (fasl-read path)]) + (spawn-actor-from-checkpoint (car state) (cadr state)))) ``` **Implementation**: -- Extend `(std staging)` with a binding-time analysis: classify expressions as static (known at compile time) or dynamic -- Static expressions are evaluated by the macro expander via `eval` -- Hybrid: partially evaluate a function, leaving dynamic parts as residual code -- Guard: pure functions only (no mutation, no I/O) - -**What this enables**: -- Compile-time regex → DFA conversion (specialized matcher, no regex engine at runtime) -- Compile-time JSON schema → specialized parser -- Compile-time SQL query → optimized access plan -- Compile-time protocol buffer → serializer/deserializer +- Use `fasl-write` / `fasl-read` for continuation serialization +- Strip non-serializable closures (FFI pointers, ports) with a pre-serialization walk +- Integration with `(std actor transport)` for live actor migration -**LOC**: ~600 +**Files**: `lib/std/actor/checkpoint.sls` (~250 LOC) +**Tests**: 15 tests -### 1.4 Continuation Mark Optimization +### 1.4 Structured Concurrency with Deadlock Detection -Chez's `call/1cc` is fast but not free. For the common case where an effect handler resumes exactly once (async/await, state, exceptions), eliminate the continuation capture entirely. - -**Implementation**: -- Extend `with-handler` with a static analysis: if the handler's `resume` is called exactly once and is in tail position, the handler is "linear" -- Linear handlers compile to a direct call (no continuation capture) -- This makes algebraic effects zero-cost for the 90% case +Extend `with-task-group` with automatic deadlock detection using a wait-for graph. ```scheme -;; The State effect handler is linear — get/put always resume once -(with-handler ([State - (get (k) (resume k current-state)) - (put (v k) (set! current-state v) (resume k (void)))]) - body) - -;; Compiles to (approximately): -(let ([current-state init]) - (fluid-let ([*state-get* (lambda () current-state)] - [*state-put* (lambda (v) (set! current-state v))]) - body)) -;; No call/1cc at all! +(with-task-group (lambda (tg) + (let ([ch1 (make-channel 0)] + [ch2 (make-channel 0)]) + ;; Deadlock: task A waits on ch2, task B waits on ch1 + (task-group-spawn tg (lambda () (channel-get ch2) (channel-put ch1 'a))) + (task-group-spawn tg (lambda () (channel-get ch1) (channel-put ch2 'b))) + ;; Runtime detects the cycle and raises &deadlock with the wait-for graph + ))) ``` -**LOC**: ~350 +**Implementation**: +- Global wait-for graph (hash table: thread-id → waiting-on-resource) +- Updated at every blocking operation (channel-get, mutex-lock, condition-wait) +- Background detector thread runs DFS cycle detection periodically +- On cycle detection: raise `&deadlock` condition with the cycle path +- Opt-in via `(parameterize ([*detect-deadlocks* #t]) ...)` + +**Files**: `lib/std/concur/deadlock.sls` (~300 LOC) +**Tests**: 15 tests --- -## Track 2: Type System — From Gradual to Powerful +## Track 2: Type System — Toward Dependent Types -The existing type system is annotations-as-assertions. That's step 1. Step 2 is making the type system powerful enough that typed Jerboa code runs measurably faster than untyped code. +Phase 2 gave us gradual typing, GADTs, type classes, and linear types. Phase 4 pushes toward a type system competitive with Haskell, Scala 3, and Idris — but always optional. -### 2.1 Algebraic Data Types with GADT Patterns +### 2.1 Compile-Time Type Checking (Not Just Runtime Assertions) -Combine sealed struct hierarchies with type-indexed pattern matching. This is the feature that makes Haskell, OCaml, and Rust's type systems so expressive. +Currently `define/t` emits runtime assertions. Phase 4 adds a static checker that runs at macro-expansion time, reporting errors before the program runs. ```scheme -;; Typed expression language — the type parameter tracks the result type -(deftype (Expr a) - (IntLit [val : Fixnum] : (Expr Fixnum)) - (BoolLit [val : Boolean] : (Expr Boolean)) - (Add [l : (Expr Fixnum)] - [r : (Expr Fixnum)] : (Expr Fixnum)) - (If [test : (Expr Boolean)] - [then : (Expr a)] - [else : (Expr a)] : (Expr a)) - (Equal [l : (Expr Fixnum)] - [r : (Expr Fixnum)] : (Expr Boolean))) - -;; Type-safe evaluator — the return type matches the GADT index -(define/t (eval-expr [e : (Expr a)]) : a - (match-type e - [(IntLit v) v] ;; v : Fixnum, return : Fixnum ✓ - [(BoolLit v) v] ;; v : Boolean, return : Boolean ✓ - [(Add l r) (fx+ (eval-expr l) (eval-expr r))] - [(If test then else) - (if (eval-expr test) (eval-expr then) (eval-expr else))] - [(Equal l r) (fx= (eval-expr l) (eval-expr r))])) +(define/t (add [x : fixnum] [y : fixnum]) : fixnum + (string-append x y)) +;; COMPILE ERROR: string-append expects (string, string), got (fixnum, fixnum) +;; at lib/myapp.sls:5:3 ``` **Implementation**: -- `deftype` macro generates sealed record types with an extra phantom type parameter tracked at compile time -- `match-type` refines the phantom type in each branch, enabling type-directed code generation -- Runtime representation: standard records (no type parameter at runtime — fully erased) -- This is *the* feature for writing interpreters, compilers, and DSLs in Jerboa +- Type environment threaded through macro expansion via syntax properties +- Bidirectional type checking: annotations flow down, inferred types flow up +- Constraint-based local inference within function bodies +- Error messages with source location, expected vs actual, and suggestions +- `define/t` becomes dual-mode: static check at expand time + optional runtime assert -**LOC**: ~700 +**Why this matters**: Typed Racket proved that a Scheme can have serious static typing. But TR's approach (separate #lang) fractures the ecosystem. Jerboa's approach (opt-in per-function via `define/t`) is gradual without fragmentation. -### 2.2 Type Classes / Protocols +**Files**: `lib/std/typed/check.sls` (~800 LOC), `lib/std/typed/infer.sls` (~600 LOC), `lib/std/typed/env.sls` (~300 LOC) +**Tests**: 40 tests -Haskell's type classes, but simpler. Define a set of operations a type must support, then write generic code against the protocol. - -```scheme -(defprotocol Printable - (to-string [self] : String)) +### 2.2 Higher-Kinded Types and Functor/Monad/Applicative -(defprotocol Hashable - (hash-code [self] : Fixnum)) +Type classes that abstract over type constructors, not just types. +```scheme (defprotocol (Functor f) (fmap [fn : (-> a b)] [fa : (f a)] : (f b))) -;; Implement for specific types -(implement Printable point - (to-string [self] (format "(~a, ~a)" (point-x self) (point-y self)))) - -(implement (Functor list) - (fmap [fn lst] (map fn lst))) - -;; Generic code — works for any Printable -(define/t (show-all [xs : (List (Printable a))]) : (List String) - (map to-string xs)) +(defprotocol (Monad m) #:extends (Functor m) + (return [a] : (m a)) + (bind [ma : (m a)] [f : (-> a (m b))] : (m b))) + +;; Implement for Option type +(deftype (Option a) (Some [val : a]) (None)) + +(implement (Functor Option) + (fmap [fn fa] + (match fa + [(Some v) (Some (fn v))] + [(None) (None)]))) + +(implement (Monad Option) + (return [a] (Some a)) + (bind [ma f] + (match ma + [(Some v) (f v)] + [(None) (None)]))) + +;; do-notation via macro +(do/m Option + [x <- (Some 3)] + [y <- (Some 4)] + (return (+ x y))) +;; => (Some 7) ``` **Implementation**: -- `defprotocol` generates a vtable struct per protocol -- `implement` registers a vtable instance in a compile-time registry -- At call sites where the concrete type is known → direct call (no vtable indirection) -- At call sites where the type is abstract → vtable dispatch (one pointer chase) -- Chez cp0 can inline the direct-call case entirely +- Extend `defprotocol` to accept type-constructor parameters `(f)`, `(m)` +- `#:extends` clause for protocol inheritance +- `do/m` macro desugars to nested `bind` calls +- Instance resolution at macro-expansion time when concrete type is known +- Fallback to vtable dispatch when type is abstract -**Why not just methods?** Methods dispatch on a single argument. Protocols can dispatch on multiple type parameters (`Functor f` abstracts over the container type). This is what makes generic programming work. +**Files**: `lib/std/typed/hkt.sls` (~500 LOC), `lib/std/typed/monad.sls` (~400 LOC) +**Tests**: 30 tests -**LOC**: ~600 +### 2.3 Refinement Types with SMT-Backed Verification -### 2.3 Linear Types for Resource Safety - -Mark values that must be used exactly once. Prevents resource leaks at compile time. +Go beyond runtime `assert-refined` — verify refinements statically using an embedded decision procedure. ```scheme -(define/t (open-file [path : String]) : (Linear Port) - (open-input-file path)) - -(define/t (read-all [p : (Linear Port)]) : (Values String (Linear Port)) - ;; Must return the port — can't drop it - (let ([data (get-string-all p)]) - (values data p))) - -(define/t (close [p : (Linear Port)]) : Void - ;; Consumes the port — type system ensures it's not used again - (close-port p)) - -;; This is a compile-time error: -(define/t (leak [path : String]) : String - (let ([p (open-file path)]) - (let-values ([(data _p) (read-all p)]) - data))) ;; ERROR: linear value _p not consumed +(define/t (safe-divide [n : number] [d : (Refine number (not zero?))]) : number + (/ n d)) + +(safe-divide 10 0) +;; COMPILE ERROR: refinement violation +;; d must satisfy (not zero?) +;; but literal 0 is always zero +;; at lib/math.sls:3:18 + +;; Flow-sensitive refinement +(define/t (safe-head [lst : (Refine list (not null?))]) : any + (car lst)) + +(define/t (process [xs : list]) + (if (null? xs) + 'empty + (safe-head xs))) ;; OK — refinement satisfied by the (not null?) branch ``` **Implementation**: -- Linear types tracked at compile time via the macro expander's environment -- Each linear binding has a "consumed" flag; checked at scope exit -- `values` and `let-values` thread linear bindings through -- In release mode: checks erased (the type system proved correctness) +- Lightweight constraint solver for linear arithmetic and boolean predicates +- Integration with occurrence typing — branch conditions refine types +- Only checks what it can prove; unresolvable refinements fall back to runtime checks +- Special rules for `null?`, `zero?`, `negative?`, `positive?`, comparison operators -**LOC**: ~500 +**Files**: `lib/std/typed/refine.sls` (~500 LOC), `lib/std/typed/solver.sls` (~400 LOC) +**Tests**: 30 tests -### 2.4 Effect Typing — Know What Your Code Does +### 2.4 Type-Safe Extensible Records (Row Polymorphism Done Right) -Annotate functions with the effects they may perform. The compiler warns on unhandled effects. +Go beyond structural row checks to full row-polymorphic record operations. ```scheme -(define/t (fetch-user [id : Fixnum]) : (Eff [IO, Async] User) - (let ([response (perform (Async await (http-get (format "/users/~a" id))))]) - (json->user (perform (IO read-body response))))) - -;; Pure function — the type proves it -(define/t (validate [user : User]) : (Eff [] Boolean) - (and (string? (user-name user)) - (> (user-age user) 0))) - -;; Compiler warns: fetch-user performs IO, Async — but no handler installed -(define (main) - (fetch-user 42)) ;; WARNING: unhandled effects [IO, Async] +;; Function works on any record with at least 'name' and 'age' fields +(define/t (greet [person : (Row name: string age: fixnum | r)]) : string + (format "Hello ~a, age ~a" (name person) (age person))) + +;; Works with any record that has those fields +(defstruct employee (name age department salary)) +(defstruct student (name age university gpa)) + +(greet (make-employee "Alice" 30 "Engineering" 100000)) ;; OK +(greet (make-student "Bob" 20 "MIT" 3.9)) ;; OK + +;; Record extension +(define/t (with-id [rec : (Row | r)] [id : fixnum]) : (Row id: fixnum | r) + (record-extend rec 'id id)) ``` **Implementation**: -- Extend the type syntax with `(Eff [effects...] result-type)` -- Effect inference: scan function bodies for `perform` calls, accumulate effect sets -- Handler checking: `with-handler` discharges effects from the body's effect set -- Polymorphic effects: `(define/t (map-eff [f : (-> a (Eff e b))] [xs : (List a)]) : (Eff e (List b)))` +- Row type variables (`| r`) represent "and possibly more fields" +- Unification of row types during type checking +- `record-extend` / `record-restrict` as primitive operations +- Compiles to Chez `define-record-type` with dynamic field tables for open rows -**LOC**: ~500 +**Files**: `lib/std/typed/row.sls` (~500 LOC) +**Tests**: 25 tests --- -## Track 3: Concurrency — Beyond Erlang +## Track 3: Effects System — Deep Handlers and Multishot -Jerboa already has actors, STM, and structured concurrency. Now make them industrial-strength. +Phase 2's effects use one-shot continuations. Phase 4 adds deep handlers (that re-install themselves) and limited multishot support for backtracking. -### 3.1 M:N Runtime with Preemptive Scheduling +### 3.1 Deep Effect Handlers -Currently actors run on OS threads. For 100,000+ concurrent actors, we need lightweight green threads multiplexed onto OS threads — but unlike Gerbil's approach, do it on top of Chez's native thread support. +Currently handlers are shallow — after resuming, the handler is no longer installed. Deep handlers automatically re-install for the remainder of the computation. ```scheme -;; Spawn 1,000,000 actors on 8 OS threads -(define pool (make-scheduler #:workers 8)) +;; Shallow (current): must manually re-install +(with-handler ([State (get (k) (resume k cell))]) + (State get) ;; handled + (State get)) ;; NOT handled — handler consumed by first resume + +;; Deep (new): handler persists +(with-deep-handler ([State (get (k) (resume k cell))]) + (State get) ;; handled + (State get) ;; also handled — handler re-installed after resume + (State get)) ;; still handled +``` -(for-each - (lambda (i) - (spawn-actor pool - (lambda (msg) - (match msg - [('ping sender) (send sender 'pong)])))) - (iota 1000000)) +**Implementation**: +- `with-deep-handler` wraps each `resume` call to re-install the handler frame before continuing +- Uses `dynamic-wind` to ensure handler is on the stack during the resumed computation +- Negligible overhead: one parameter mutation per resume + +**Files**: `lib/std/effect/deep.sls` (~200 LOC) +**Tests**: 20 tests + +### 3.2 Multishot Continuations via Delimited Prompts -;; Each actor is ~200 bytes (continuation + mailbox pointer) -;; Total: ~200 MB for 1M actors +For backtracking search, nondeterminism, and probabilistic programming, support continuations that can be invoked more than once. + +```scheme +(defeffect Choose (choose options)) + +;; Backtracking search — explore all choices +(define (all-solutions thunk) + (let ([results '()]) + (with-multishot-handler + ([Choose + (choose (k options) + (for-each (lambda (opt) + (let ([r (resume k opt)]) ;; k invoked multiple times! + (set! results (cons r results)))) + options))]) + (thunk)) + (reverse results))) + +(all-solutions (lambda () + (let ([x (Choose choose '(1 2 3))] + [y (Choose choose '(a b))]) + (list x y)))) +;; => ((1 a) (1 b) (2 a) (2 b) (3 a) (3 b)) ``` **Implementation**: -- Extend `(std actor scheduler)` with a timer-interrupt based preemption mechanism -- Use Chez's `timer-interrupt-handler` to yield the current actor after a time slice -- Actor state = saved one-shot continuation (from `call/1cc`) + mailbox ref -- The scheduler dequeues the next ready actor and resumes its continuation -- Work-stealing between worker threads for load balancing +- Use Chez's full `call/cc` (not `call/1cc`) for multishot handlers +- Stack copying cost is real — document performance implications +- `with-multishot-handler` as explicit opt-in (don't slow down one-shot handlers) +- Useful for: SAT solvers, probabilistic programming, parser combinators, logic programming + +**Files**: `lib/std/effect/multishot.sls` (~350 LOC) +**Tests**: 20 tests -**Key Chez primitives**: `timer-interrupt-handler`, `set-timer`, `call/1cc`, `engine` (Chez's built-in coroutine mechanism — engines are preemptible computations!) +### 3.3 Effect Polymorphism in the Type System -**Chez engines**: Chez has a built-in `make-engine` / `engine-return` / `engine-block` mechanism that provides preemptive, timed evaluation. Each engine gets a fuel count (ticks); when fuel runs out, the engine suspends and returns its continuation. This is *exactly* what we need for actor scheduling: +Connect the effect system to the type system so the compiler knows which effects a function may perform. ```scheme -(define (run-actor actor fuel) - (let ([eng (make-engine (lambda () (actor-body actor)))]) - (eng fuel - ;; Completed within fuel - (lambda (remaining-fuel value) (actor-complete! actor value)) - ;; Ran out of fuel — preempted - (lambda (remaining-engine) (reschedule! actor remaining-engine))))) +(define/te (pure-add [x : fixnum] [y : fixnum]) : (Eff [] fixnum) + (fx+ x y)) + +(define/te (stateful-add [x : fixnum]) : (Eff [State] fixnum) + (let ([current (perform (State get))]) + (perform (State put (fx+ current x))) + (perform (State get)))) + +;; Compiler ensures all effects are handled +(run-state 0 (lambda () (stateful-add 5))) ;; OK — State handled + +(stateful-add 5) +;; WARNING: unhandled effect [State] at lib/app.sls:10 ``` -**Why this is better than goroutines**: Goroutines can't be inspected or migrated. Jerboa actors have typed mailboxes, supervision trees, and can be transparently distributed across nodes. +**Implementation**: +- Extend `define/te` with effect set inference +- `with-handler` discharges named effects from the inferred set +- Effect polymorphism: `(define/te (map-eff [f : (-> a (Eff e b))] [xs : list]) : (Eff e list) ...)` +- Warning (not error) for unhandled effects — gradual adoption + +**Files**: `lib/std/typed/effects.sls` (~400 LOC) +**Tests**: 25 tests + +--- -**LOC**: ~800 +## Track 4: Metaprogramming — The Unfair Advantage -### 3.2 Channel Select with Priority and Default +### 4.1 Multi-Stage Programming (Staging à la MetaOCaml) -Go's `select` is one of its best features. Jerboa should have it, but better. +Go beyond `define-ct` to proper staged computation with code quotation and splicing. ```scheme -(select - ;; Receive from channels with priority (first match wins on tie) - [(recv ch1) => (lambda (msg) (handle-request msg))] - [(recv ch2) => (lambda (msg) (handle-event msg))] - ;; Send to a channel (blocks if full) - [(send result-ch answer) => (lambda () (log "sent"))] - ;; Timer - [(after 5000) => (lambda () (log "timeout"))] - ;; Default — non-blocking poll - [default => (lambda () (log "nothing ready"))]) +;; Stage 0: generate optimized code at compile time +(define-staged (make-power n) + (lambda/staged (x) + (let loop ([i n]) + (if (= i 0) + (quote-stage 1) + (quote-stage (* x ~(loop (- i 1)))))))) + +;; At compile time: (make-power 5) generates: +;; (lambda (x) (* x (* x (* x (* x (* x 1)))))) +;; Chez cp0 then folds the (* ... 1) away + +(define power5 (make-power 5)) +(power5 3) ;; => 243, computed with 4 multiplications, no loop ``` **Implementation**: -- `select` macro compiles to a wait on multiple condition variables with a shared "claimed" flag -- When any channel becomes ready, it signals the select's condition variable -- Priority: check channels in order; first ready one wins -- `default`: if no channel is ready, execute immediately (non-blocking) -- `after`: register a timer with the event loop; fires if no channel fires first +- `quote-stage` / `~` (splice) for code quotation and antiquotation +- Type-safe: spliced expressions must have the right type +- Cross-stage persistence for values that survive from stage 0 to stage 1 +- Integration with Chez's `eval` for compile-time evaluation -**Integration with actors**: `receive` in an actor body becomes syntactic sugar for `select` on the actor's mailbox. +**Why unique**: MetaOCaml has this. BER MetaScheme has a prototype. Nobody has it integrated with a macro system AND algebraic effects. -**LOC**: ~400 +**Files**: `lib/std/staging/multi.sls` (~500 LOC) +**Tests**: 25 tests -### 3.3 Async Streams +### 4.2 Syntax-Level Pattern Matching (Match on AST) -Lazy sequences that produce values asynchronously. The marriage of `(std seq)` and `(std async)`. +Pattern matching on syntax objects for writing macros more naturally. ```scheme -;; An async stream of lines from a network connection -(define (line-stream conn) - (async-generate - (lambda (yield) - (let loop () - (let ([line (await (tcp-read-line conn))]) - (unless (eof-object? line) - (yield line) - (loop))))))) - -;; Process with familiar sequence operations — but each step may suspend -(async-for-each - (lambda (line) - (await (process-line line))) - (async-filter - (lambda (line) (string-prefix? "DATA:" line)) - (line-stream connection))) +(define-syntax my-let + (syntax-match () + [(_ ([var expr] ...) body ...) + #'((lambda (var ...) body ...) expr ...)] + [(_ loop ([var init] ...) body ...) + #'(letrec ([loop (lambda (var ...) body ...)]) + (loop init ...))])) ``` **Implementation**: -- `async-generate` creates a producer that yields values via one-shot continuation -- `async-for-each`, `async-map`, `async-filter` — standard operations that `await` between elements -- Back-pressure: the producer suspends when the consumer isn't ready -- Cancellation: dropping the stream reference triggers cleanup via guardian +- `syntax-match` — pattern matching on syntax objects with template output +- Integrates with `match2` patterns (guards, nested patterns, `or` patterns) +- Syntax patterns: `...` for repetition, `_` for wildcard, `#:keyword` for keywords +- Much more readable than nested `syntax-case` with `with-syntax` -**LOC**: ~450 +**Files**: `lib/std/staging/syntax-match.sls` (~350 LOC) +**Tests**: 20 tests -### 3.4 Distributed Consensus (Raft) +### 4.3 Compile-Time Code Contracts -CRDTs give eventual consistency. For strong consistency, implement Raft. +Verify properties of generated code at macro-expansion time. ```scheme -(define cluster (raft-cluster - #:nodes '("node1:9001" "node2:9001" "node3:9001") - #:state-machine (lambda (state command) - (match command - [('set key val) (hash-set state key val)] - [('get key) (values state (hash-ref state key))])))) - -;; Strongly consistent reads and writes -(raft-apply! cluster '(set "user:1" "Alice")) ;; replicated to majority -(raft-query cluster '(get "user:1")) ;; reads from leader → "Alice" +(define-syntax/contract (safe-vector-ref vec idx) + #:pre (and (identifier? #'vec) (integer? (syntax->datum #'idx))) + #:post (lambda (expanded) (not (contains-unsafe? expanded))) + #'(let ([v vec] [i idx]) + (assert (< i (vector-length v))) + (vector-ref v i))) ``` **Implementation**: -- Build on `(std actor transport)` for RPC and `(std actor cluster)` for node management -- Leader election, log replication, and safety per the Raft paper -- State machine interface: user provides a pure function `(state, command) → (state, response)` -- Snapshotting for log compaction -- Joint consensus for cluster membership changes +- `define-syntax/contract` wraps a transformer with pre/post checks +- Pre-conditions: validated on the input syntax +- Post-conditions: validated on the expanded output +- Violations are compile-time errors with source locations +- Useful for macro libraries that must guarantee safety properties -**LOC**: ~1200 +**Files**: `lib/std/staging/contract.sls` (~250 LOC) +**Tests**: 15 tests --- -## Track 4: Metaprogramming — The Unfair Advantage - -Scheme's macro system is its superpower. Jerboa should push it further than any language has gone. +## Track 5: Systems Programming — The Rust Alternative -### 4.1 Syntax-Level Computation (Typed Macros) +### 5.1 Async I/O Runtime with io_uring Backend -Macros that carry type information through the expansion. The macro system becomes a type-level programming language. +Build a proper async runtime that uses Linux io_uring for zero-copy, zero-syscall I/O. ```scheme -;; Type-level natural numbers -(define-type-syntax Zero) -(define-type-syntax (Succ n)) - -;; Type-safe heterogeneous list indexed by length -(define-syntax HList - (syntax-rules () - [(_ ()) '()] - [(_ (t . ts)) (cons t (HList ts))])) - -;; Type-safe vector access — out-of-bounds is a compile-time error -(define-syntax vec-ref/safe - (lambda (stx) - (syntax-case stx () - [(_ vec idx) - (let ([len (syntax-local-value #'vec 'vector-length)] - [i (syntax->datum #'idx)]) - (when (>= i len) - (syntax-error stx "index out of bounds")) - #'(vector-ref vec idx))]))) +(define runtime (make-io-runtime #:backend 'io-uring #:workers (cpu-count))) + +(run-io runtime (lambda () + ;; All I/O is non-blocking, submitted to io_uring in batches + (let ([listener (tcp-listen "0.0.0.0" 8080)]) + (let loop () + (let ([conn (await (tcp-accept listener))]) + (spawn-task (lambda () + (let ([request (await (read-http-request conn))]) + (let ([response (handle-request request)]) + (await (write-http-response conn response)) + (await (close conn)))))) + (loop)))))) ``` **Implementation**: -- Extend the macro expander environment with compile-time value bindings -- `syntax-local-value` retrieves compile-time metadata for a binding -- `define-for-syntax` binds values available during macro expansion -- Type-level computation happens entirely at compile time — zero runtime cost +- `(std os iouring)` already exists — extend with high-level async wrappers +- Submission queue batching: collect I/O requests, submit in one syscall +- Completion queue polling in a dedicated thread, dispatching to actor/task callbacks +- Fallback to epoll on older kernels (< 5.1) +- File I/O, socket I/O, timer, and fsync all via io_uring -**LOC**: ~500 +**Files**: `lib/std/async/ioruntime.sls` (~600 LOC), `lib/std/async/iouring-ops.sls` (~400 LOC) +**Tests**: 25 tests -### 4.2 Declarative Derive System +### 5.2 Arena Allocators for Zero-GC Hot Paths -Automatically generate implementations from struct definitions. Like Rust's `#[derive]` or Haskell's `deriving`. +For latency-sensitive code, allocate from a fixed arena that gets bulk-freed, avoiding GC pauses entirely. ```scheme -(defstruct point (x y) - #:derive (equal hash print json serializable)) - -;; Auto-generates: -;; - (point=? a b) — structural equality -;; - (point-hash p) — consistent hash code -;; - Custom print method: #<point x: 3 y: 4> -;; - (point->json p) → {"x": 3, "y": 4} -;; - (json->point j) → (make-point 3 4) -;; - (point->bytes p) / (bytes->point bv) — binary serialization - -;; Users can define custom derivations: -(define-derivation comparable - (lambda (struct-info) - (let ([fields (struct-info-fields struct-info)]) - #`(define (#,(format-id 'compare (symbol->string (struct-info-name struct-info))) - a b) - (let loop ([fs '#,fields]) - (if (null? fs) 0 - (let ([cmp (compare (field-ref a (car fs)) - (field-ref b (car fs)))]) - (if (zero? cmp) (loop (cdr fs)) cmp)))))))) +(define arena (make-arena (* 1024 1024))) ;; 1 MB arena + +(with-arena arena + ;; All allocations in this scope use the arena + (let ([buf (arena-alloc-bytevector 4096)]) + (read-into! fd buf) + (process buf))) +;; Arena reset — all memory freed in O(1), no GC involvement + +;; For request handlers: +(define (handle-request req) + (with-arena (request-arena req) + ;; Temporary allocations for parsing/serialization live in per-request arena + ;; Arena freed when request completes + (let ([body (parse-json (request-body req))]) + (json->bytevector (process body))))) ``` **Implementation**: -- Extend `defstruct` to accept `#:derive` clause -- Each derivation is a macro that receives struct metadata (name, fields, types, parent) and produces definitions -- Built-in derivations: `equal`, `hash`, `print`, `json`, `serializable`, `comparable`, `copy`, `builder` -- User-extensible via `define-derivation` +- `make-arena` allocates a contiguous block via `foreign-alloc` +- `arena-alloc-bytevector` returns bytevectors backed by arena memory +- `with-arena` resets the bump pointer on scope exit +- Guardian integration: arena itself is GC-managed, but contents are not +- Thread-local arena parameter for implicit arena selection + +**Why this matters**: Java's ZGC and Go's GC still have tail latencies. Arena allocation gives deterministic latency for request-processing hot paths. Zig popularized this pattern; Jerboa brings it to a GC'd language. -**LOC**: ~700 +**Files**: `lib/std/mem/arena.sls` (~350 LOC) +**Tests**: 20 tests -### 4.3 Compile-Time Regular Expressions +### 5.3 Structured Binary Data (Like Rust's `repr(C)`) -Compile regex patterns to DFA state machines at compile time. No regex engine overhead at runtime. +Define packed binary layouts that map directly to C structs, network packets, and file formats. ```scheme -(define-regex email-pattern - "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$") +(define-binary-struct ip-header + #:endian 'big + (version uint4) + (ihl uint4) + (dscp uint6) + (ecn uint2) + (total-length uint16) + (id uint16) + (flags uint3) + (frag-offset uint13) + (ttl uint8) + (protocol uint8) + (checksum uint16) + (src-addr uint32) + (dst-addr uint32)) + +;; Zero-copy parsing from a bytevector +(define header (bytevector->ip-header packet 0)) +(ip-header-src-addr header) ;; => #x0A000001 + +;; Zero-copy serialization +(define bv (ip-header->bytevector header)) +``` + +**Implementation**: +- `define-binary-struct` generates `foreign-ref` / `foreign-set!` accessors at computed offsets +- Bit-field support via shift-and-mask operations +- Endianness specified per-struct or per-field +- Nested structs and fixed-size arrays +- Validation: field values checked against bit-width at write time -;; At compile time: parses regex, builds NFA, converts to DFA, generates code -;; At runtime: a direct state machine — no interpretation +**Files**: `lib/std/binary.sls` (~500 LOC) +**Tests**: 25 tests -(email-pattern "user@example.com") ;; => #t -(email-pattern "invalid") ;; => #f +### 5.4 Safe Memory-Mapped Persistent Data Structures -;; With capture groups -(define-regex url-pattern - "^(https?)://([^/]+)(/.*)?$" - #:captures (scheme host path)) +Combine `(std os mmap)` with persistent data structures for databases and caches. -(url-pattern "https://example.com/api") -;; => #<match scheme: "https" host: "example.com" path: "/api"> +```scheme +;; Memory-mapped B+ tree +(define db (mmap-btree-open "data.db" + #:key-type 'string + #:value-type 'bytevector + #:page-size 4096)) + +(mmap-btree-put! db "user:1" (string->utf8 "Alice")) +(mmap-btree-get db "user:1") ;; => #vu8(65 108 105 99 101) + +;; Crash-safe: uses write-ahead log + msync +(mmap-btree-transaction db + (lambda (txn) + (txn-put! txn "counter" (fixnum->bytevector (+ 1 (bytevector->fixnum (txn-get txn "counter"))))) + (txn-put! txn "updated" (fixnum->bytevector (current-time))))) ``` **Implementation**: -- Regex parser → NFA → DFA (subset construction) → code generator, all at compile time -- Code generator emits a `case`-based state machine that Chez compiles to a jump table -- Capture groups track start/end positions during the DFA walk -- Fallback to PCRE2 for features DFA can't handle (backreferences, lookahead) +- B+ tree with 4K pages mapped via mmap +- Write-ahead log for crash recovery +- Copy-on-write pages for MVCC transactions +- `msync` for durability guarantees +- Compaction and defragmentation -**Why faster than PCRE2**: No function call overhead per character. The DFA is inlined code. For simple patterns, 5-10x faster than interpreted regex. - -**LOC**: ~800 +**Files**: `lib/std/db/mmap-btree.sls` (~800 LOC) +**Tests**: 30 tests --- -## Track 5: Data & Collections — The Clojure Playbook - -Clojure proved that immutable persistent data structures can be the default for a practical language. Jerboa should have the best persistent data structures of any Scheme. +## Track 6: Developer Experience — What Makes People Stay -### 5.1 Persistent Vectors (HAMT-Based) +### 6.1 Time-Travel Debugger with Replay