Phase 2 implementation plan: 8 tracks to make Jerboa the superior Scheme
ober
f63262b231fc9506f28b4c85684b92ade56bcf20
--- a/docs/implement.md +++ b/docs/implement.md @@ -1,874 +1,1125 @@ -# Jerboa Implementation Plan: The Superior Scheme +# Jerboa Implementation Plan: Phase 2 — The Superior Scheme -## Vision +## Where We Are -Jerboa is not another Scheme implementation. It is a **systems programming language** with Scheme's elegance, built on Chez Scheme's world-class compiler. Where Racket chose "batteries included but slow," where Gerbil chose "Gambit plus syntax," and where Guile chose "GNU's extension language," Jerboa chooses: **maximum performance, fearless concurrency, and zero-compromise developer experience.** +Jerboa's first 13 phases are complete: 87 modules, 14,876 lines, 346+ tests. We have algebraic effects, gradual typing, STM, actors, distributed computing, structured concurrency, lazy sequences, pattern matching, staging, capability security, and a native binary toolchain — all on stock Chez Scheme with zero shim layers. -The goal: a language you'd choose over Rust for concurrent network services, over Go for systems tooling, over Erlang for distributed systems — because Jerboa gives you all three with less code and no garbage collector pauses that matter. +That's a strong foundation. But it's not yet the standout Scheme. The existing implementation has API surfaces for these features but hasn't pushed any of them to the depth where they become *the reason* someone chooses Jerboa over Rust, Go, or Erlang. Phase 2 is about depth, not breadth. + +This plan identifies **25 additions** organized into 8 tracks, each designed to exploit Chez Scheme's unique strengths in ways no other language implementation can match. --- -## Current State (Step 8 Complete) - -| Component | Status | LOC | -|-----------|--------|-----| -| Core (reader, macros, runtime) | Complete | ~1,700 | -| Standard Library (51 modules) | Complete | ~3,500 | -| FFI (c-lambda, foreign wrappers) | Working | ~500 | -| Actor System (7 layers) | Complete | ~2,500 | -| Gradual Typing (Phase 2-3) | Working | ~320 | -| Build System + Cache | Prototype | ~350 | -| FFI DSL (define-foreign) | Prototype | ~350 | -| Tests | 338 passing | — | -| External Wrappers | 11 chez-* libs | — | -| LSP Server | Separate project, 13/15 e2e | — | +## Track 1: Compiler Infrastructure — The Performance Moat ---- +Chez Scheme has the best optimizing compiler in the Lisp world. Jerboa should be the language that gives users direct access to that power. No other Scheme exposes compile-time optimization hooks this way. + +### 1.1 Profile-Guided Optimization (PGO) -## Implementation Phases +Record type feedback from production runs, feed it back to the compiler. -### Phase 1: Effect System and Algebraic Effects -**Why this is transformative**: No production Scheme has algebraic effects. Racket has parameters and continuations. OCaml 5 just got effects. Jerboa can be the first Scheme with a typed, performant effect system — and Chez's first-class continuations make the implementation natural. +**What Chez gives us**: `cp0` (copy propagation pass 0) already does aggressive inlining and constant folding. With type profiles, we can tell it which branches to favor. -Algebraic effects subsume: exceptions, async/await, generators/iterators, coroutines, backtracking, nondeterminism, and state. One mechanism replaces six. +```scheme +;; Instrument: record which types flow through each call site +(jerboa build myapp.ss --profile) +./myapp --workload production < data.txt +;; Produces myapp.profile — type histograms per call site + +;; Optimize: use the profile to specialize +(jerboa build myapp.ss --pgo myapp.profile -o myapp-fast) +;; Now (+ x y) at line 47 emits fx+ because the profile shows x,y are always fixnums +``` -**Step 9: Core Effect Handlers** -- File: `lib/std/effect.sls` -- Implement `with-handler`, `perform`, `resume` using Chez's `call/1cc` (one-shot continuations for performance; no full `call/cc` overhead) -- One-shot continuations are critical: most effect handlers resume at most once, and `call/1cc` avoids the continuation-copying overhead of full `call/cc` +**Implementation**: +- `lib/std/dev/pgo.sls` — instrumentation macros that wrap call sites with type counters +- A `define-syntax` transformer that reads profile data at compile time and emits specialized code paths +- Integration with `(jerboa build)`: `--profile` flag instruments, `--pgo` flag applies +- Store profiles as FASL files (native Chez serialization, fast to read) + +**Why this is unique**: No Scheme, no Lisp, no ML has PGO. Only C/C++ (GCC/LLVM), Go, and Rust (via LLVM) have it. Jerboa would be the first functional language with PGO. + +**LOC**: ~500 + +### 1.2 Whole-Program Devirtualization + +When the compiler can see all implementations of a method, replace dynamic dispatch with a `cond` on the type. ```scheme -;; Define effects as lightweight structs -(defeffect Async - (await promise) ; suspend until promise resolves - (spawn thunk)) ; launch concurrent task - -(defeffect State - (get) ; read current state - (put val)) ; write new state - -;; Handle effects — the handler chooses how to resume -(with-handler ([Async - (await (p k) (on-resolve p (lambda (v) (resume k v)))) - (spawn (t k) (fork-thread t) (resume k (void)))] - [State - (get (k) (resume k current-state)) - (put (v k) (set! current-state v) (resume k (void)))]) - (let ([data (perform (Async await (fetch-url url)))]) - (perform (State put data)) - (process data))) +;; Before: runtime hashtable lookup +({area} shape) ;; → find-method → eq-hashtable-ref → call + +;; After (when only circle, rect, triangle implement area): +(cond + [(circle? shape) (circle-area shape)] ;; native record predicate, inlineable + [(rect? shape) (rect-area shape)] + [(triangle? shape) (triangle-area shape)] + [else (error 'area "no method" shape)]) ``` -**What this enables**: -- Async/await without colored functions (any function can perform effects) -- Testable code: swap real I/O handler for mock handler -- Composable: stack multiple handlers (async + state + logging) -- Zero-cost when not used: no overhead if no handler is installed - -**Step 10: Effect Typing Integration** -- Extend `std/typed.sls` with effect annotations -- `(def (fetch [url : String]) : (Effect Async String))` — the type tells you this function performs Async effects -- Effect inference: if a function calls `perform`, its effect type is inferred -- Effect polymorphism: `(def (map-effect [f : (-> A (Effect E B))] [xs : (List A)]) : (Effect E (List B)))` - -**Implementation strategy**: -- `defeffect` macro generates: effect struct types, performer functions, pattern-match clauses -- `with-handler` macro generates: `call/1cc` capture point, dispatch table, resume continuation -- Compile-time effect tracking: accumulate effects through function calls, warn on unhandled effects -- Runtime: effect dispatch via eq-hashtable keyed on effect type descriptor (same pattern as method dispatch — O(1)) - -**Performance considerations**: -- One-shot continuations (`call/1cc`) avoid the heap allocation of multi-shot `call/cc` -- Effect dispatch is a single hashtable probe (same as method dispatch) -- When the handler is statically known (common case), the macro can inline the handler body directly -- Chez's cp0 can inline small handlers across the `with-handler` boundary - -**Lines**: ~400 for core, ~200 for typed integration +**Implementation**: +- Collect all `bind-method!` calls during WPO's whole-program analysis +- For each method, if the set of implementing types is closed, emit a `cond` dispatch +- Chez's cp0 can then inline the accessor bodies if they're small +- Result: method call → record predicate check → inlined body. Two instructions. ---- +**Why this matters**: This is the optimization that makes Java's HotSpot fast (speculative devirtualization). Jerboa can do it statically at compile time because WPO sees the whole program. + +**LOC**: ~400 -### Phase 2: Async I/O Runtime -**Why this matters**: Every serious systems language needs non-blocking I/O. Go has goroutines + netpoller, Erlang has the BEAM scheduler, Rust has tokio. Jerboa's actor system already has threads and mailboxes — add an event loop and you have a complete async runtime. +### 1.3 Compile-Time Partial Evaluation -**Step 11: Event Loop on epoll** -- File: `lib/std/async.sls` -- Build on existing `std/os/epoll.sls` wrapper -- Single event loop per scheduler thread (no cross-thread wakeup overhead) -- Integrate with effect system: `(perform (Async await ...))` suspends the current task and registers it with the event loop +Go beyond macros — let the compiler evaluate any pure expression at compile time. ```scheme -;; The event loop is an effect handler -(define (run-async thunk) - (let ([loop (make-event-loop)]) - (with-handler ([Async - (await (promise k) - (event-loop-register! loop promise k)) - (spawn (thunk k) - (event-loop-submit! loop thunk) - (resume k (void))) - (sleep (ms k) - (event-loop-timer! loop ms k))]) - (thunk) - (event-loop-run! loop)))) ; process events until all tasks done - -;; TCP server with async effects — reads like synchronous code -(def (handle-client conn) - (let loop () - (let ([data (perform (Async await (tcp-read conn 4096)))]) - (unless (eof-object? data) - (perform (Async await (tcp-write conn (process data)))) - (loop))))) - -(def (start-server port) - (run-async - (lambda () - (let ([listener (tcp-listen port)]) - (let accept-loop () - (let ([conn (perform (Async await (tcp-accept listener)))]) - (perform (Async spawn (lambda () (handle-client conn)))) - (accept-loop))))))) +(define-ct (fib n) + (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) + +(define answer (fib 30)) +;; At compile time: evaluates to 832040 +;; At runtime: (define answer 832040) — zero-cost constant ``` -**Step 12: Async-Aware Channels and Actors** -- Extend `std/misc/channel.sls`: channel-send/receive become effect-aware (suspend on full/empty instead of blocking OS thread) -- Extend actor mailbox: `receive` suspends via effect when mailbox empty, event loop reschedules when message arrives -- Result: millions of concurrent tasks on a handful of OS threads (like Go goroutines, but with algebraic effects instead of a special runtime) +**Implementation**: +- Extend `(std staging)` with a binding-time analysis: classify expressions as static (known at compile time) or dynamic +- Static expressions are evaluated by the macro expander via `eval` +- Hybrid: partially evaluate a function, leaving dynamic parts as residual code +- Guard: pure functions only (no mutation, no I/O) + +**What this enables**: +- Compile-time regex → DFA conversion (specialized matcher, no regex engine at runtime) +- Compile-time JSON schema → specialized parser +- Compile-time SQL query → optimized access plan +- Compile-time protocol buffer → serializer/deserializer + +**LOC**: ~600 -**Step 13: io_uring Integration (Linux 5.1+)** -- File: `lib/std/os/iouring.sls` -- io_uring provides zero-copy, zero-syscall async I/O -- Submit batches of I/O operations (read, write, accept, connect) in a single syscall -- Completion queue maps directly to effect handler resume points -- This is the performance frontier: even Go and Rust/tokio are still migrating to io_uring +### 1.4 Continuation Mark Optimization + +Chez's `call/1cc` is fast but not free. For the common case where an effect handler resumes exactly once (async/await, state, exceptions), eliminate the continuation capture entirely. + +**Implementation**: +- Extend `with-handler` with a static analysis: if the handler's `resume` is called exactly once and is in tail position, the handler is "linear" +- Linear handlers compile to a direct call (no continuation capture) +- This makes algebraic effects zero-cost for the 90% case ```scheme -(define-ffi-library liburing "liburing" - (io_uring_queue_init (unsigned int pointer) -> int) - (io_uring_get_sqe (pointer) -> pointer) - (io_uring_submit (pointer) -> int) - (io_uring_wait_cqe (pointer pointer) -> int)) - -;; Transparent to user code — same async effect, faster backend -(define (make-iouring-event-loop) - (let ([ring (io-uring-init 256)]) - (make-event-loop - #:backend 'io-uring - #:submit (lambda (ops) (io-uring-submit-batch ring ops)) - #:poll (lambda () (io-uring-poll ring))))) +;; The State effect handler is linear — get/put always resume once +(with-handler ([State + (get (k) (resume k current-state)) + (put (v k) (set! current-state v) (resume k (void)))]) + body) + +;; Compiles to (approximately): +(let ([current-state init]) + (fluid-let ([*state-get* (lambda () current-state)] + [*state-put* (lambda (v) (set! current-state v))]) + body)) +;; No call/1cc at all! ``` -**Lines**: ~600 (event loop) + ~300 (io_uring) + ~200 (async channels) +**LOC**: ~350 --- -### Phase 3: Advanced Type System -**Why this matters**: Typed Racket proved that gradual typing for Scheme is possible but showed it's painfully slow at type boundaries. Jerboa's approach is different: types are *compiler hints*, not runtime contracts. In debug mode, they're assertions. In release mode, they guide optimization. No boundary tax. +## Track 2: Type System — From Gradual to Powerful + +The existing type system is annotations-as-assertions. That's step 1. Step 2 is making the type system powerful enough that typed Jerboa code runs measurably faster than untyped code. -**Step 14: Occurrence Typing** -- Extend `std/typed.sls` -- After a type predicate, narrow the type in the consequent branch -- This is the feature that makes gradual typing actually useful in practice +### 2.1 Algebraic Data Types with GADT Patterns + +Combine sealed struct hierarchies with type-indexed pattern matching. This is the feature that makes Haskell, OCaml, and Rust's type systems so expressive. ```scheme -(def (process [x : (Union String Number)]) - (cond - [(string? x) - ;; Here the compiler knows x is String - ;; Emits (string-length x) directly, no type check - (string-length x)] - [(number? x) - ;; Here x is Number — emit (fx+ x 1) if fixnum range - (+ x 1)])) +;; Typed expression language — the type parameter tracks the result type +(deftype (Expr a) + (IntLit [val : Fixnum] : (Expr Fixnum)) + (BoolLit [val : Boolean] : (Expr Boolean)) + (Add [l : (Expr Fixnum)] + [r : (Expr Fixnum)] : (Expr Fixnum)) + (If [test : (Expr Boolean)] + [then : (Expr a)] + [else : (Expr a)] : (Expr a)) + (Equal [l : (Expr Fixnum)] + [r : (Expr Fixnum)] : (Expr Boolean))) + +;; Type-safe evaluator — the return type matches the GADT index +(define/t (eval-expr [e : (Expr a)]) : a + (match-type e + [(IntLit v) v] ;; v : Fixnum, return : Fixnum ✓ + [(BoolLit v) v] ;; v : Boolean, return : Boolean ✓ + [(Add l r) (fx+ (eval-expr l) (eval-expr r))] + [(If test then else) + (if (eval-expr test) (eval-expr then) (eval-expr else))] + [(Equal l r) (fx= (eval-expr l) (eval-expr r))])) ``` -**Step 15: Row Polymorphism for Records** -- Allow functions to accept "any record with at least these fields" -- This gives structural subtyping without the complexity of full OOP +**Implementation**: +- `deftype` macro generates sealed record types with an extra phantom type parameter tracked at compile time +- `match-type` refines the phantom type in each branch, enabling type-directed code generation +- Runtime representation: standard records (no type parameter at runtime — fully erased) +- This is *the* feature for writing interpreters, compilers, and DSLs in Jerboa + +**LOC**: ~700 + +### 2.2 Type Classes / Protocols + +Haskell's type classes, but simpler. Define a set of operations a type must support, then write generic code against the protocol. ```scheme -;; This function works on any struct with 'name' and 'age' fields -(def (greet [person : (Row name: String age: Number)]) - (format "Hello ~a, you are ~a years old" - (~ person name) (~ person age))) +(defprotocol Printable + (to-string [self] : String)) -(defstruct employee (name age department salary)) -(defstruct student (name age university gpa)) +(defprotocol Hashable + (hash-code [self] : Fixnum)) -;; Both work — row polymorphism checks structurally -(greet (make-employee "Alice" 30 "Engineering" 150000)) -(greet (make-student "Bob" 22 "MIT" 3.9)) +(defprotocol (Functor f) + (fmap [fn : (-> a b)] [fa : (f a)] : (f b))) + +;; Implement for specific types +(implement Printable point + (to-string [self] (format "(~a, ~a)" (point-x self) (point-y self)))) + +(implement (Functor list) + (fmap [fn lst] (map fn lst))) + +;; Generic code — works for any Printable +(define/t (show-all [xs : (List (Printable a))]) : (List String) + (map to-string xs)) ``` -**Implementation**: At compile time, row types resolve to a set of required accessors. The macro emits a record-type-descriptor check + field access. Chez's cp0 can often inline the accessor if the concrete type is known. +**Implementation**: +- `defprotocol` generates a vtable struct per protocol +- `implement` registers a vtable instance in a compile-time registry +- At call sites where the concrete type is known → direct call (no vtable indirection) +- At call sites where the type is abstract → vtable dispatch (one pointer chase) +- Chez cp0 can inline the direct-call case entirely + +**Why not just methods?** Methods dispatch on a single argument. Protocols can dispatch on multiple type parameters (`Functor f` abstracts over the container type). This is what makes generic programming work. -**Step 16: Refinement Types** -- Types with predicates: `(Refine Number positive?)` means "a number that satisfies `positive?`" -- In debug mode: runtime assertion. In release mode: erased (you're asserting correctness). -- Killer feature for FFI: `(Refine Pointer nonnull?)` catches null pointer bugs at the boundary +**LOC**: ~600 + +### 2.3 Linear Types for Resource Safety + +Mark values that must be used exactly once. Prevents resource leaks at compile time. ```scheme -(def (sqrt [x : (Refine Number (lambda (n) (>= n 0)))]) : Number - (fl-sqrt (inexact x))) +(define/t (open-file [path : String]) : (Linear Port) + (open-input-file path)) + +(define/t (read-all [p : (Linear Port)]) : (Values String (Linear Port)) + ;; Must return the port — can't drop it + (let ([data (get-string-all p)]) + (values data p))) + +(define/t (close [p : (Linear Port)]) : Void + ;; Consumes the port — type system ensures it's not used again + (close-port p)) + +;; This is a compile-time error: +(define/t (leak [path : String]) : String + (let ([p (open-file path)]) + (let-values ([(data _p) (read-all p)]) + data))) ;; ERROR: linear value _p not consumed +``` -;; Port numbers, array indices, etc. — refinements catch logic bugs -(def (connect [host : String] - [port : (Refine Fixnum (lambda (p) (<= 1 p 65535)))]) - ...) +**Implementation**: +- Linear types tracked at compile time via the macro expander's environment +- Each linear binding has a "consumed" flag; checked at scope exit +- `values` and `let-values` thread linear bindings through +- In release mode: checks erased (the type system proved correctness) + +**LOC**: ~500 + +### 2.4 Effect Typing — Know What Your Code Does + +Annotate functions with the effects they may perform. The compiler warns on unhandled effects. + +```scheme +(define/t (fetch-user [id : Fixnum]) : (Eff [IO, Async] User) + (let ([response (perform (Async await (http-get (format "/users/~a" id))))]) + (json->user (perform (IO read-body response))))) + +;; Pure function — the type proves it +(define/t (validate [user : User]) : (Eff [] Boolean) + (and (string? (user-name user)) + (> (user-age user) 0))) + +;; Compiler warns: fetch-user performs IO, Async — but no handler installed +(define (main) + (fetch-user 42)) ;; WARNING: unhandled effects [IO, Async] ``` -**Step 17: Type-Directed Compilation** -- When the type system knows a value is a fixnum, emit `fx+` instead of generic `+` -- When it knows a value is a flonum, emit `fl*` instead of generic `*` -- When it knows a list is non-empty, skip the `null?` check -- This is where Jerboa's type system pays for itself in raw performance +**Implementation**: +- Extend the type syntax with `(Eff [effects...] result-type)` +- Effect inference: scan function bodies for `perform` calls, accumulate effect sets +- Handler checking: `with-handler` discharges effects from the body's effect set +- Polymorphic effects: `(define/t (map-eff [f : (-> a (Eff e b))] [xs : (List a)]) : (Eff e (List b)))` -**Lines**: ~500 (occurrence typing) + ~400 (row types) + ~300 (refinements) + ~300 (type-directed compilation) +**LOC**: ~500 --- -### Phase 4: Software Transactional Memory -**Why this matters**: Locks don't compose. If module A takes lock 1 then lock 2, and module B takes lock 2 then lock 1, you get deadlocks. STM makes concurrent data access composable — and Chez's first-class continuations make the implementation elegant. +## Track 3: Concurrency — Beyond Erlang -**Step 18: STM Core** -- File: `lib/std/stm.sls` -- Transactional variables (TVars) with optimistic read/write sets -- `atomically` block: run speculatively, validate read set, commit or retry +Jerboa already has actors, STM, and structured concurrency. Now make them industrial-strength. + +### 3.1 M:N Runtime with Preemptive Scheduling + +Currently actors run on OS threads. For 100,000+ concurrent actors, we need lightweight green threads multiplexed onto OS threads — but unlike Gerbil's approach, do it on top of Chez's native thread support. ```scheme -(define balance-a (make-tvar 1000)) -(define balance-b (make-tvar 2000)) - -;; Transfer is atomic — no locks, no deadlocks, composable -(def (transfer! from to amount) - (atomically - (let ([f (tvar-read from)] - [t (tvar-read to)]) - (when (< f amount) - (retry)) ;; block until balances change, then re-run - (tvar-write! from (- f amount)) - (tvar-write! to (+ t amount))))) - -;; STM composes — this is impossible with locks -(def (transfer-both! a b c amount) - (atomically - (transfer! a b amount) ;; These two transfers are - (transfer! b c amount))) ;; a single atomic operation +;; Spawn 1,000,000 actors on 8 OS threads +(define pool (make-scheduler #:workers 8)) + +(for-each + (lambda (i) + (spawn-actor pool + (lambda (msg) + (match msg + [('ping sender) (send sender 'pong)])))) + (iota 1000000)) + +;; Each actor is ~200 bytes (continuation + mailbox pointer) +;; Total: ~200 MB for 1M actors ``` **Implementation**: -- TVars: boxed values with version counters -- Transaction log: thread-local read-set (tvar + version-seen) and write-set (tvar + new-value) -- Commit: acquire global lock (or per-tvar locks with total ordering), validate read-set versions, apply write-set, bump versions, release -- Retry: register current thread on TVars' wait sets, sleep on condition variable, wake when any TVar changes -- Nested transactions: flatten into parent (no nested commit) +- Extend `(std actor scheduler)` with a timer-interrupt based preemption mechanism +- Use Chez's `timer-interrupt-handler` to yield the current actor after a time slice +- Actor state = saved one-shot continuation (from `call/1cc`) + mailbox ref +- The scheduler dequeues the next ready actor and resumes its continuation +- Work-stealing between worker threads for load balancing + +**Key Chez primitives**: `timer-interrupt-handler`, `set-timer`, `call/1cc`, `engine` (Chez's built-in coroutine mechanism — engines are preemptible computations!) + +**Chez engines**: Chez has a built-in `make-engine` / `engine-return` / `engine-block` mechanism that provides preemptive, timed evaluation. Each engine gets a fuel count (ticks); when fuel runs out, the engine suspends and returns its continuation. This is *exactly* what we need for actor scheduling: -**Integration with effects**: ```scheme -(defeffect STM - (read tvar) - (write tvar val) - (retry)) +(define (run-actor actor fuel) + (let ([eng (make-engine (lambda () (actor-body actor)))]) + (eng fuel + ;; Completed within fuel + (lambda (remaining-fuel value) (actor-complete! actor value)) + ;; Ran out of fuel — preempted + (lambda (remaining-engine) (reschedule! actor remaining-engine))))) ``` -**Lines**: ~500 +**Why this is better than goroutines**: Goroutines can't be inspected or migrated. Jerboa actors have typed mailboxes, supervision trees, and can be transparently distributed across nodes. ---- +**LOC**: ~800 -### Phase 5: Fearless FFI -**Why this matters**: The existing FFI works but it's manual. The goal is to make calling C as easy as calling Scheme — with safety guarantees that prevent use-after-free, buffer overflows, and null pointer dereferences. +### 3.2 Channel Select with Priority and Default -**Step 19: Auto-Generated Bindings from C Headers** -- File: `lib/std/foreign/bind.sls` -- Parse C header files and generate Jerboa FFI bindings automatically -- Handle: functions, structs, enums, typedefs, #defines -- Use the existing `c-lambda` → `foreign-procedure` pipeline +Go's `select` is one of its best features. Jerboa should have it, but better. ```scheme -;; One line replaces hundreds of manual bindings -(define-c-library sqlite3 - (header "sqlite3.h") - (link "-lsqlite3") - (prefix sqlite3_) ;; strip prefix from Scheme names - (include-only ;; only bind what you need - sqlite3_open sqlite3_close sqlite3_prepare_v2 - sqlite3_step sqlite3_column_* sqlite3_finalize)) - -;; Auto-generated: (sqlite3-open path db) → int, etc. -;; Type-safe: pointer arguments checked, strings auto-converted +(select + ;; Receive from channels with priority (first match wins on tie) + [(recv ch1) => (lambda (msg) (handle-request msg))] + [(recv ch2) => (lambda (msg) (handle-event msg))] + ;; Send to a channel (blocks if full) + [(send result-ch answer) => (lambda () (log "sent"))] + ;; Timer + [(after 5000) => (lambda () (log "timeout"))] + ;; Default — non-blocking poll + [default => (lambda () (log "nothing ready"))]) ``` -**Step 20: Ownership-Tracked Pointers** -- Wrap foreign pointers with ownership metadata -- Use Chez guardians for GC-triggered cleanup -- Prevent use-after-free at the type level +**Implementation**: +- `select` macro compiles to a wait on multiple condition variables with a shared "claimed" flag +- When any channel becomes ready, it signals the select's condition variable +- Priority: check channels in order; first ready one wins +- `default`: if no channel is ready, execute immediately (non-blocking) +- `after`: register a timer with the event loop; fires if no channel fires first + +**Integration with actors**: `receive` in an actor body becomes syntactic sugar for `select` on the actor's mailbox. + +**LOC**: ~400 + +### 3.3 Async Streams + +Lazy sequences that produce values asynchronously. The marriage of `(std seq)` and `(std async)`. ```scheme -(defstruct/foreign sqlite3-db - (pointer nonnull) - (destructor sqlite3-close) ;; called by guardian or explicit free - (owned #t)) ;; this Scheme code owns the pointer - -;; Use-after-free is a compile-time error when types are enabled -(def (bad-example) - (let ([db (sqlite3-open ":memory:")]) - (sqlite3-db-free! db) - (sqlite3-prepare db "SELECT 1"))) ;; Type error: db is freed +;; An async stream of lines from a network connection +(define (line-stream conn) + (async-generate + (lambda (yield) + (let loop () + (let ([line (await (tcp-read-line conn))]) + (unless (eof-object? line) + (yield line) + (loop))))))) + +;; Process with familiar sequence operations — but each step may suspend +(async-for-each + (lambda (line) + (await (process-line line))) + (async-filter + (lambda (line) (string-prefix? "DATA:" line)) + (line-stream connection))) ``` -**Step 21: Async Foreign Calls** -- Problem: `foreign-procedure` blocks the OS thread -- Solution: run blocking FFI calls on a dedicated thread pool, suspend the calling task via effect system +**Implementation**: +- `async-generate` creates a producer that yields values via one-shot continuation +- `async-for-each`, `async-map`, `async-filter` — standard operations that `await` between elements +- Back-pressure: the producer suspends when the consumer isn't ready +- Cancellation: dropping the stream reference triggers cleanup via guardian + +**LOC**: ~450 + +### 3.4 Distributed Consensus (Raft) + +CRDTs give eventual consistency. For strong consistency, implement Raft. ```scheme -;; Transparent to caller — looks synchronous, runs async -(define-foreign/async curl-easy-perform - (c-lambda (pointer) int "curl_easy_perform") - #:blocking #t) ;; runs on FFI thread pool - -(def (fetch url) - ;; This suspends the current task, not the OS thread - (let ([handle (curl-easy-init)]) - (curl-easy-setopt handle CURLOPT_URL url) - (curl-easy-perform handle))) ;; non-blocking! +(define cluster (raft-cluster + #:nodes '("node1:9001" "node2:9001" "node3:9001") + #:state-machine (lambda (state command) + (match command + [('set key val) (hash-set state key val)] + [('get key) (values state (hash-ref state key))])))) + +;; Strongly consistent reads and writes +(raft-apply! cluster '(set "user:1" "Alice")) ;; replicated to majority +(raft-query cluster '(get "user:1")) ;; reads from leader → "Alice" ``` -**Lines**: ~600 (header parsing) + ~300 (ownership) + ~300 (async FFI) +**Implementation**: +- Build on `(std actor transport)` for RPC and `(std actor cluster)` for node management +- Leader election, log replication, and safety per the Raft paper +- State machine interface: user provides a pure function `(state, command) → (state, response)` +- Snapshotting for log compaction +- Joint consensus for cluster membership changes + +**LOC**: ~1200 --- -### Phase 6: Pattern Matching 2.0 -**Why this matters**: Jerboa already has `match`. But the best pattern matching in any language is in Rust (exhaustiveness checking) and Scala 3 (extractors). Jerboa can have both, plus features neither has. +## Track 4: Metaprogramming — The Unfair Advantage + +Scheme's macro system is its superpower. Jerboa should push it further than any language has gone. + +### 4.1 Syntax-Level Computation (Typed Macros) -**Step 22: Exhaustiveness Checking** -- When matching on a defstruct hierarchy with `(sealed #t)`, the compiler knows all possible cases -- Warn on non-exhaustive patterns at compile time -- This catches bugs that would be runtime `match-error` in every other Scheme +Macros that carry type information through the expansion. The macro system becomes a type-level programming language. ```scheme -(defstruct shape () sealed: #t) -(defstruct circle shape (radius)) -(defstruct rect shape (width height)) -(defstruct triangle shape (a b c)) - -(def (area [s : shape]) - (match s - ((circle r) (* pi r r)) - ((rect w h) (* w h)))) - ;; WARNING: non-exhaustive match — missing 'triangle' case +;; Type-level natural numbers +(define-type-syntax Zero) +(define-type-syntax (Succ n)) + +;; Type-safe heterogeneous list indexed by length +(define-syntax HList + (syntax-rules () + [(_ ()) '()] + [(_ (t . ts)) (cons t (HList ts))])) + +;; Type-safe vector access — out-of-bounds is a compile-time error +(define-syntax vec-ref/safe + (lambda (stx) + (syntax-case stx () + [(_ vec idx) + (let ([len (syntax-local-value #'vec 'vector-length)] + [i (syntax->datum #'idx)]) + (when (>= i len) + (syntax-error stx "index out of bounds")) + #'(vector-ref vec idx))]))) ``` -**Step 23: Active Patterns (Extractors)** -- User-defined pattern decomposition — patterns that run arbitrary code -- Like Scala extractors or F# active patterns +**Implementation**: +- Extend the macro expander environment with compile-time value bindings +- `syntax-local-value` retrieves compile-time metadata for a binding +- `define-for-syntax` binds values available during macro expansion +- Type-level computation happens entirely at compile time — zero runtime cost + +**LOC**: ~500 + +### 4.2 Declarative Derive System + +Automatically generate implementations from struct definitions. Like Rust's `#[derive]` or Haskell's `deriving`. ```scheme -;; Define an active pattern for parsing -(define-active-pattern (IPv4 s) - (let ([parts (string-split s ".")]) - (and (= (length parts) 4) - (let ([nums (map string->number parts)]) - (and (every (lambda (n) (and n (<= 0 n 255))) nums) - (apply values nums)))))) - -(def (classify-ip ip) - (match ip - ((IPv4 10 _ _ _) 'private-class-a) - ((IPv4 192 168 _ _) 'private-class-c) - ((IPv4 127 _ _ _) 'loopback) - ((IPv4 a b c d) (list 'public a b c d)))) +(defstruct point (x y) + #:derive (equal hash print json serializable)) + +;; Auto-generates: +;; - (point=? a b) — structural equality +;; - (point-hash p) — consistent hash code +;; - Custom print method: #<point x: 3 y: 4> +;; - (point->json p) → {"x": 3, "y": 4} +;; - (json->point j) → (make-point 3 4) +;; - (point->bytes p) / (bytes->point bv) — binary serialization + +;; Users can define custom derivations: +(define-derivation comparable + (lambda (struct-info) + (let ([fields (struct-info-fields struct-info)]) + #`(define (#,(format-id 'compare (symbol->string (struct-info-name struct-info))) + a b) + (let loop ([fs '#,fields]) + (if (null? fs) 0 + (let ([cmp (compare (field-ref a (car fs)) + (field-ref b (car fs)))]) + (if (zero? cmp) (loop (cdr fs)) cmp)))))))) ``` -**Step 24: Pattern Guards and View Patterns** +**Implementation**: +- Extend `defstruct` to accept `#:derive` clause +- Each derivation is a macro that receives struct metadata (name, fields, types, parent) and produces definitions +- Built-in derivations: `equal`, `hash`, `print`, `json`, `serializable`, `comparable`, `copy`, `builder` +- User-extensible via `define-derivation` + +**LOC**: ~700 + +### 4.3 Compile-Time Regular Expressions + +Compile regex patterns to DFA state machines at compile time. No regex engine overhead at runtime. + ```scheme -(match request - ((http-request method: 'GET path: (? string-prefix? "/api/" -> rest)) - (handle-api rest)) - ((http-request method: 'POST body: (json-parse -> data) - (where (hash-has-key? data 'action))) - (handle-action data))) +(define-regex email-pattern + "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$") + +;; At compile time: parses regex, builds NFA, converts to DFA, generates code +;; At runtime: a direct state machine — no interpretation + +(email-pattern "user@example.com") ;; => #t +(email-pattern "invalid") ;; => #f + +;; With capture groups +(define-regex url-pattern + "^(https?)://([^/]+)(/.*)?$" + #:captures (scheme host path)) + +(url-pattern "https://example.com/api") +;; => #<match scheme: "https" host: "example.com" path: "/api"> ``` -**Lines**: ~500 (exhaustiveness) + ~300 (active patterns) + ~200 (guards/views) +**Implementation**: +- Regex parser → NFA → DFA (subset construction) → code generator, all at compile time +- Code generator emits a `case`-based state machine that Chez compiles to a jump table +- Capture groups track start/end positions during the DFA walk +- Fallback to PCRE2 for features DFA can't handle (backreferences, lookahead) + +**Why faster than PCRE2**: No function call overhead per character. The DFA is inlined code. For simple patterns, 5-10x faster than interpreted regex. + +**LOC**: ~800 --- -### Phase 7: Metaprogramming and Staging -**Why this matters**: Chez Scheme has the most powerful macro system of any production language. Jerboa should exploit this to the fullest — not just with macros, but with *multi-stage programming* that lets you write code that writes optimized code. +## Track 5: Data & Collections — The Clojure Playbook -**Step 25: Compile-Time Computation** -- File: `lib/std/staging.sls` -- `(at-compile-time expr)` evaluates at compile time and splices the result -- Build lookup tables, precompute constants, generate specialized code +Clojure proved that immutable persistent data structures can be the default for a practical language. Jerboa should have the best persistent data structures of any Scheme. + +### 5.1 Persistent Vectors (HAMT-Based) + +Immutable vectors with O(log32 n) ≈ O(1) access, update, and append. 32-way branching trie. ```scheme -;; Compile-time regex → specialized matcher (no runtime parsing) -(define-syntax fast-match - (lambda (stx) - (syntax-case stx () - [(_ pattern input) - (let ([dfa (at-compile-time (regex->dfa (syntax->datum #'pattern)))]) - (generate-dfa-matcher dfa #'input))]))) +(define v (persistent-vector 1 2 3 4 5)) -;; The generated code is a direct state machine — no regex engine at runtime -(fast-match "^[a-z]+@[a-z]+\\.[a-z]{2,}$" email) -``` +(persistent-vector-ref v 2) ;; => 3, O(~1) +(define v2 (persistent-vector-set v 2 99)) ;; => [1 2 99 4 5], O(~1) +(persistent-vector-ref v 2) ;; => 3 (original unchanged) -**Step 26: Code Generation DSL** -- Multi-stage programming: write programs that generate programs -- Type-safe quasiquotation with guaranteed well-formedness -- Use case: domain-specific optimizers, JIT-like specialization +(define v3 (persistent-vector-append v 6)) ;; => [1 2 3 4 5 6] -```scheme -;; Generate specialized serializer at compile time based on struct definition -(define-syntax derive-serializer - (lambda (stx) - (syntax-case stx () - [(_ struct-name) - (let ([fields (struct-fields (syntax->datum #'struct-name))]) - #`(def (#,(format-id #'struct-name "serialize-~a" #'struct-name) obj port) - #,@(map (lambda (f) - #`(write-field '#,f (#,(accessor-name #'struct-name f) obj) port)) - fields)))]))) - -(derive-serializer point) -;; Expands to: -;; (def (serialize-point obj port) -;; (write-field 'x (point-x obj) port) -;; (write-field 'y (point-y obj) port)) +;; Transient for batch mutation (like Clojure) +(define v4 (persistent! + (let ([t (transient v)]) + (transient-set! t 0 100) + (transient-set! t 1 200) + (transient-append! t 999) + t))) ``` -**Step 27: Syntax-Rules Extensions** -- `defrule` already works; extend with: - - Ellipsis depth tracking (nested `...`) - - Template guards `(where ...)` - - Recursive templates for tree transformations +**Implementation**: +- 32-way branching trie (5 bits per level, max 7 levels for 2^35 elements) +- Path copying on update (structural sharing) +- Tail optimization: last chunk stored separately for O(1) append +- Transient mode: mutable operations on a thread-owned copy, then freeze -**Lines**: ~400 (staging) + ~500 (codegen DSL) + ~200 (syntax extensions) +**Why this matters**: Persistent vectors are thread-safe by construction. No locks, no copying, no races. Combined with STM, this gives Clojure-style concurrency without the JVM. ---- +**LOC**: ~600 -### Phase 8: Distributed Computing -**Why this matters**: The actor system (Steps 4-8) provides the foundation. Now build the distributed layer that makes Jerboa competitive with Erlang/OTP for building distributed systems. +### 5.2 Persistent Hash Maps (CHAMP) -**Step 28: Node Discovery and Clustering** -- File: `lib/std/actor/cluster.sls` -- Automatic node discovery via UDP multicast or explicit seed nodes -- Cluster membership with failure detection (phi accrual failure detector) -- Node-local actor registry automatically federated +Compressed Hash Array Mapped Trie — the state of the art for immutable hash maps. ```scheme -(define node (start-node! - #:name "worker-1" - #:cookie "my-secret-cookie" ;; Erlang-style shared secret - #:listen "tcp://0.0.0.0:9000" - #:seeds '("tcp://192.168.1.10:9000"))) - -;; Actors on remote nodes are transparent -(let ([db (whereis 'database #:node "db-server")]) - (ask db (query:select "users" #:where '(active = #t)))) +(define m (persistent-map 'a 1 'b 2 'c 3)) + +(persistent-map-ref m 'b) ;; => 2 +(define m2 (persistent-map-set m 'd 4)) ;; => {a:1 b:2 c:3 d:4} +(persistent-map-ref m 'd) ;; => error (original unchanged) + +;; Efficient merge +(persistent-map-merge m m2 (lambda (k v1 v2) v2)) + +;; Works with STM +(define state (make-tvar (persistent-map))) +(atomically + (tvar-write! state + (persistent-map-set (tvar-read state) 'counter + (+ 1 (persistent-map-ref (tvar-read state) 'counter 0))))) ``` -**Step 29: Distributed Supervision** -- Supervisors that manage actors across nodes -- If a node goes down, restart its actors on surviving nodes -- Configurable placement strategies: round-robin, least-loaded, affinity +**Implementation**: +- CHAMP trie with bitmap-indexed nodes (Steindorfer & Vinju 2015) +- ~2x more memory-efficient than HAMT due to compressed node layout +- Equality checking in O(1) for identical tries (pointer equality) +- Efficient diff: `persistent-map-diff` walks only differing subtrees + +**LOC**: ~700 + +### 5.3 Immutable Sorted Maps (Red-Black Trees) -**Step 30: CRDT-Based Distributed State** -- File: `lib/std/actor/crdt.sls` -- Conflict-free replicated data types for eventually-consistent shared state -- G-Counter, PN-Counter, OR-Set, LWW-Register, MV-Register -- Integrate with actor registry: distributed state that survives node failures +For ordered data — range queries, min/max, ordered iteration. ```scheme -;; Distributed counter — no coordination needed -(define visitors (make-distributed-counter 'site-visitors)) +(define s (sorted-map < 3 "c" 1 "a" 5 "e" 2 "b")) -;; Each node increments locally -(crdt-increment! visitors) +(sorted-map-min s) ;; => (1 . "a") +(sorted-map-max s) ;; => (5 . "e") +(sorted-map-range s 2 4) ;; => ((2 . "b") (3 . "c")) +(sorted-map-ref s 3) ;; => "c" -;; Reads merge automatically — eventually consistent -(crdt-value visitors) ;; => 14523 (merged from all nodes) +;; Persistent — all operations return new trees +(define s2 (sorted-map-set s 4 "d")) ``` -**Lines**: ~500 (clustering) + ~400 (distributed supervision) + ~600 (CRDTs) +**Implementation**: +- Okasaki-style persistent red-black trees +- O(log n) insert, delete, lookup +- O(log n) split and join for efficient range operations +- Integration with `(std seq)`: `sorted-map->lazy-seq` for lazy ordered traversal + +**LOC**: ~500 --- -### Phase 9: Developer Experience -**Why this matters**: The best language in the world fails if the tooling is painful. Jerboa should have the best REPL, the best debugger, and the best build system of any Scheme. +## Track 6: Systems Programming — The Rust Alternative + +Make Jerboa the best Scheme for writing the kind of software people currently write in Rust or Go. -**Step 31: Hot Code Reloading** -- File: `lib/std/dev/reload.sls` -- Reload individual modules without restarting the process -- Actors receive a `'code-change` message and can migrate state to the new version -- Erlang's killer feature, now in Scheme +### 6.1 Memory-Mapped I/O + +Direct access to file contents as bytevectors without copying. ```scheme -;; In the REPL during development -(reload! 'my-app/handlers) ;; recompiles and reloads +(define mapping (mmap "large-file.dat" #:mode 'read-only))