docs: add jerboa-on-jerboa self-hosting plan
ober
a8c7b58f3bdc2747b4d05014d9f4ff0a720c1801
new file mode 100644 --- /dev/null +++ b/docs/jerboa-on-jerboa.md @@ -0,0 +1,305 @@ +# Jerboa on Jerboa + +A staged plan to rewrite as much of Jerboa as possible *in Jerboa itself*. + +## Thesis + +Today, Jerboa is a Chez-hosted language: `.ss` source is translated to `.sls` +and executed on Chez Scheme. The language implementation and the standard +library are written in R6RS `.sls`. The character of "Jerboa" — `def`, +`defstruct`, `match`, `[...]` lists, `{...}` hashes, `keyword:` args — appears +only in user code (tests, examples, the `jerboa-*` companion repos). + +This is backwards. A language's stdlib is the most-read corpus of code in +that language. If our stdlib is R6RS, then *Jerboa* is a syntactic veneer +applied to user code, not a language. Self-hosting — in the form of an +idiomatic stdlib and tooling layer written in `.ss` — is what makes Jerboa +its own thing. + +This document is a plan, not a commitment. It is meant to make the work +concrete enough that a contributor can pick a stage and execute it. + +## What "self-hosted" means here + +We use the term in the standard staged-bootstrap sense, the way GCC, Rust, +and Clojure use it: + +- **Stage 0**: the language's reader/translator/core macros, written in the + host language (Chez R6RS). This *must exist* — something has to read the + first `.ss` file. +- **Stage N**: the same components, re-expressed in the language itself. The + stage-0 implementation is retained as bootstrap, not deleted. + +We are not trying to replace Chez Scheme as the runtime. The WASM backend in +`lib/jerboa/wasm/` is a separate question and out of scope for this plan. + +## Current state (2026-05-13) + +Lines of code, excluding `lib/thunderchez/` (now removed): + +| Layer | Files | LOC | +| ------------------------------------------- | ----- | -------- | +| Stdlib (`lib/std/**/*.sls`) | ~510 | ~140,000 | +| Language impl (`lib/jerboa/**/*.sls`) | ~30 | ~14,000 | +| Jerboa source (`tests/`, `examples/`, etc.) | ~400 | ~72,000 | + +The 154k lines of `.sls` are the entire language implementation plus stdlib. +The 72k lines of `.ss` are user-level code that *consumes* Jerboa but doesn't +*define* it. + +## The bootstrap floor + +These files cannot be written in Jerboa — they define what Jerboa is. + +| File | LOC | Why it must stay R6RS | +| --------------------------------- | ---- | ------------------------------------------------------------ | +| `lib/jerboa/reader.sls` | 893 | Parses `.ss` syntax — needs to exist before any `.ss` runs | +| `lib/jerboa/core.sls` | 1248 | Defines `def`, `defstruct`, `match`, `try-catch` macros | +| `lib/jerboa/translator.sls` | 811 | `.ss` → `.sls` transforms | +| `lib/jerboa/runtime.sls` | 426 | Hash/keyword/method-dispatch primitives on Chez records | +| `lib/jerboa/prelude*.sls` | 712 | Re-exports that wire the above together | +| `lib/jerboa/ffi.sls` | 137 | Wraps Chez `foreign-procedure` | + +**Subtotal: ~4,200 LOC of irreducible stage-0.** + +Note: stage-N versions of `reader.sls`, `translator.sls`, and even pieces of +`core.sls` *can* be written in Jerboa once Jerboa exists. The stage-0 files +are then retained only for bootstrap. This is Stage 3 below, and is the most +ambitious step. + +## What's actually rewritable + +### Tier A — pure logic, would be meaningfully better in Jerboa + +Files in `lib/jerboa/` outside the bootstrap floor: + +| Module | LOC | Notes | +| ----------------------------------------------- | ---- | ---------------------------------------------------------------------- | +| `pkg.sls` + `lock.sls` + `registry.sls` | 498 | Package manager. Manifests, semver, lockfiles — natural fit for `defstruct` + `match`. | +| `cache.sls` | 146 | Content-addressed compile cache. | +| `build.sls` + `build/musl.sls` | 1000 | Build orchestration. | +| `hot.sls` | 120 | File-watcher. | +| `embed.sls` | 190 | Sandbox API. | +| `cross.sls` | 159 | Target config. | +| `clojure.sls` + `cloj.sls` | 594 | Clojure compat. Already a re-export layer; minimal logic. | +| `wasm/*` | ~6300 | WASM backend. Pure data manipulation over WASM module structures. | + +**Tier A subtotal: ~9,000 LOC.** None of this depends on Chez-specific +features that aren't already exposed through the Jerboa prelude. + +### Tier B — stdlib pure code + +Most of `lib/std/**` is pure code that doesn't depend on R6RS specifically. +Rough breakdown of the 140k LOC: + +| Subtree | LOC | Self-hostable? | +| ---------------------------------------- | ------ | -------------- | +| `std/net/` (HTTP, URI, DNS, TLS glue) | 14,841 | Mostly yes; some TLS/socket FFI stays stage-0 | +| `std/misc/` | 12,799 | Yes — utility code | +| `std/text/` (JSON, YAML, regex, markup) | 7,219 | Yes | +| `std/srfi/` | 6,575 | Yes | +| `std/security/`, `std/secure/` | 8,570 | Mostly yes | +| `std/typed/` | 4,052 | Yes | +| `std/actor/`, `std/csp/`, `std/concur/` | 7,328 | Yes — built on top of Chez threading primitives | +| `std/os/` | 6,792 | Partial — syscall wrappers stay; higher-level path/process logic can move | +| `std/ffi/` | 3,574 | **No** — these *are* the FFI bindings | +| Top-level (`csv`, `time`, `transducer`, etc.) | 41,441 | Yes | + +**Tier B rough estimate: ~120k LOC self-hostable**, ~10k FFI/syscall glue +stays as stage-0 `.sls`. + +### Total + +- **Must stay R6RS**: ~4k (bootstrap floor) + ~10k (FFI/syscall shims) = ~14k LOC +- **Could move to idiomatic Jerboa `.ss`**: ~140k LOC + +## Staging plan + +The plan is incremental. Each stage produces a shippable artifact and a +clear gate before the next stage begins. + +### Stage 0 — already done + +The current state. Chez-hosted, Jerboa as a syntactic frontend. Tests and +companion repos written in `.ss`. + +### Stage 1 — Tier A migration (`lib/jerboa/` non-bootstrap) + +Move pkg/lock/registry/cache/build/hot/embed/cross from `.sls` to `.ss`. +Rewrite *idiomatically* (see "Migration mechanics" below), not as a sed +pass. The WASM backend (`wasm/*`) is its own substage; defer if too large. + +**Why first**: these are the smallest, most-changing modules. They get the +most benefit from being in Jerboa (`defstruct` for manifests, `match` for +dispatch, `keyword:` args for builder APIs). They're isolated — no other +stdlib module imports `(jerboa pkg)` or `(jerboa build)`. + +**Acceptance**: +- `make binary` succeeds. +- Full test suite passes. +- No file in this tier imports `(chezscheme)` directly except via the + prelude. +- Each migrated file uses at least three Jerboa-specific features + (`def` + `[...]` + `keyword:` is the minimum bar). + +**Estimated effort**: ~9k LOC of rewriting. Probably 3–4 weeks of focused +work, mostly mechanical once the idiom is settled. + +### Stage 2 — stdlib pure-logic modules + +Move the Tier B stdlib subtrees in priority order: + +1. **Top-level utilities first** (`std/csv`, `std/transducer`, `std/transit`, + `std/zipper`, `std/variant`) — small, self-contained, no downstream + dependents on their `.sls` paths. +2. **`std/misc/**`** — heavily used internally; migrate once the pattern is + proven. +3. **`std/text/**`** — JSON, YAML, regex. These are widely depended on; do + them carefully with golden-output tests. +4. **`std/srfi/**`** — by SRFI number, in dependency order. +5. **`std/typed/`, `std/security/`, `std/actor/`, `std/csp/`, `std/concur/`, + `std/net/`, `std/os/` (non-syscall portions)** — large, internally + coherent subtrees; migrate one at a time. + +**Why this order**: minimize coordination cost. Each subtree should be a +single PR if possible. The migration order is "fewest reverse dependencies +first." + +**Acceptance per subtree**: +- `make binary` succeeds. +- `make test` passes. +- For text/data modules, golden-output equivalence with the pre-migration + version (byte-for-byte JSON output, regex match results). +- No new performance regression beyond a documented budget (say, 10%). + +**Estimated effort**: many months. This is the bulk of the work. + +### Stage 3 — re-implement the bootstrap floor in Jerboa + +The ambitious step. Write `reader-jerboa.ss`, `translator-jerboa.ss`, +`core-jerboa.ss` that re-implement the same surface in Jerboa. Use them in +the standard build; keep the `.sls` versions as the stage-0 bootstrap (used +to build the stage-N versions for the first time). + +**Why last**: cannot be done before Stage 2 because the stage-N versions +should *use* the Jerboa stdlib, and the stage-N reader must be expressible +in Jerboa idioms. Doing it before the stdlib migrates would force the +stage-N reader to be written in R6RS-idiomatic-but-`.ss`, which defeats +the purpose. + +**Acceptance**: +- Two-stage build verifies fixed-point: stage-0 builds stage-N1; stage-N1 + builds stage-N2; stage-N1 and stage-N2 produce byte-identical output. +- `make binary` with the stage-N reader/translator passes the full test + suite. +- Stage-0 sources stay in-tree, used only for first-build bootstrap. + +**Estimated effort**: ~3k LOC of careful new code. Multiple weeks at minimum. + +## Migration mechanics + +### What "idiomatic Jerboa" means in a migrated file + +Mechanical translation (`define` → `def`, `(list ...)` → `[...]`) is theater. +A migrated file should use Jerboa features that the R6RS version couldn't +express: + +- `def` with destructuring instead of `define` + `let-values` +- `defstruct` for record types instead of `define-record-type` +- `match` on values instead of `cond` chains +- `[...]` list literals where lists are constructed inline +- `{...}` hash literals where alists were used +- `keyword:` arguments for any function with more than ~3 positional args +- `try-catch` instead of `guard` / `with-exception-handler` +- `defmethod` for any informal type dispatch + +A migration is not "done" until these features are *used*, not just +*available*. + +### How to verify equivalence + +For pure functions (parsers, serializers, validators), the cheapest verifier +is golden output: run the old `.sls` and new `.ss` against a corpus and +diff. For stateful modules, a unit test pass plus the full integration suite +is what we have. + +A new `tools/compare-migration.sh` script would make this repeatable. Not +required for stage 1; will be required for stage 2. + +### What stays as `.sls` + +- FFI shims in `lib/std/ffi/**` — these *are* the C ABI bindings, not code. +- Syscall wrappers in `lib/std/os/**` that call `foreign-procedure` directly. +- The bootstrap floor (see "The bootstrap floor" above). +- `lib/thunderchez/` (already removed as of 2026-05-13). + +## Risks and non-goals + +### Risks + +**Performance regression.** The `.ss` → `.sls` translator may produce +worse code than hand-written R6RS in places. Mitigation: keep the golden +test suite, and benchmark hot paths (JSON parse, regex match, HTTP framing) +before/after each subtree migration. + +**Debug-ability regression.** Stack traces through translated code may be +harder to read than direct Chez. Mitigation: source maps in the translator, +or at least line-preserving translation for runtime errors. + +**Macro-system mismatches.** Some R6RS macros in the stdlib use +`syntax-case` features that don't have direct Jerboa equivalents. Triage +case by case; some macros may need to stay `.sls` as macro-helpers used by +the `.ss` files. + +**Maintenance fork during migration.** Two-version state (`.sls` + `.ss`) +is bad. Migrate a subtree in one PR, do not leave half-migrated trees in the +tree across releases. + +### Non-goals + +- **Replacing Chez Scheme.** This plan is about source-level + self-hosting. The runtime stays Chez. +- **Removing `.sls` entirely.** The bootstrap floor and FFI/syscall shims + stay R6RS. ~14k LOC will always be `.sls`, and that's correct. +- **Self-hosting the WASM backend on WASM.** Out of scope. The + `lib/jerboa/wasm/` code is Tier A as a stdlib migration candidate, but its + *output* is and remains WASM bytecode for browsers. +- **Beating Chez at compile time.** The Chez compiler is staying. + +## Open questions + +These need answers before Stage 2 starts in earnest: + +1. **Translator stability.** Is the `.ss` → `.sls` translator stable enough + that 140k LOC depends on it without flux? Today most `.ss` code is tests; + stdlib code will exercise corners the translator hasn't seen. + +2. **Macro hygiene at scale.** Some R6RS modules cross-import macros from + each other. Does the Jerboa macro system handle 500-module compositions + cleanly? Stage 1 will exercise this on a small scale; Stage 2 is the + real test. + +3. **Conditional expansion.** R6RS `cond-expand` and Chez-specific build + conditionals exist in `lib/std/os/`. The Jerboa equivalent surface may + need extension before that subtree migrates. + +4. **Versioned stdlib.** Once the stdlib is in `.ss`, version-pinning a + stdlib for embedded use becomes easier. Worth a design pass before + Stage 2 ends. + +5. **Companion-repo coordination.** `jerboa-gitsafe`, `jerboa-websearch`, + etc. depend on stdlib import paths. Stable import paths during migration + are a hard requirement. + +## How to start + +If picking this up cold, the smallest useful first step is: + +1. Pick one Tier A file (suggested: `lib/jerboa/lock.sls` — small, isolated, + high-`defstruct` payoff). +2. Rewrite it as `lib/jerboa/lock.ss` using the idiom guide above. +3. Verify `make binary && make test` passes. +4. Open a PR. This is the template for every subsequent migration. + +Stage 1 is the proof-of-concept. If Stage 1 doesn't deliver a tangible +ergonomics win on the migrated files, Stage 2 should not start.