Align docs around Typed Jerboa migration
ober
a06fa3596509b35571bc905169a7a5b8fb508235
--- a/docs/architecture-split.md +++ b/docs/architecture-split.md @@ -10,7 +10,8 @@ Why? Because chez-* libraries were designed for the broader Chez Scheme communit ## The Rule -**If it's just FFI, it's chez-*. If it has logic, it's jerboa.** +**If it's just FFI, it's chez-*. If it has logic, it's Jerboa. If it needs +compile-time guarantees, make it Typed Jerboa.** Specifically: @@ -26,7 +27,7 @@ Examples of what belongs here: - Init/cleanup lifecycle (`ssl-init!`, `ssl-cleanup!`) - Simple error code → condition translation -### jerboa modules (full platform) +### Jerboa modules (full platform) Everything that does real work on top of FFI shims. Protocol state machines, connection management, session lifecycle, error handling, concurrency, resource cleanup — all of this benefits from jerboa's stdlib. @@ -39,12 +40,36 @@ Examples of what belongs here: - Resource lifecycle (uses `(std misc custodian)`, `(std misc guardian-pool)`) - Contract-checked public APIs (uses `(std contract)`) +### Typed Jerboa modules (static Jerboa tier) + +Typed Jerboa is still Jerboa source. Use it when the code benefits from static +interfaces, checked effects, exhaustive variants, resource ownership, and +generated native artifacts. + +Examples of what belongs here: + +- Pure editor models such as split trees and buffer metadata +- Ropes, diff engines, search helpers, parsers, and protocol codecs +- Compiler passes and lowered IR transforms +- Declarative FFI/resource specifications that need ownership and nullability + checked before reaching native code +- Small kernels that should generate safe Rust first and LLVM later + +Generated Rust or LLVM IR from Typed Jerboa is a build artifact, not the source +language. + ## Current Inventory ### Replaced by Rust native backend (`libjerboa_native.so`) These chez-* C FFI shims have Rust replacements in `jerboa-native-rs/`. The Rust modules are the recommended backend; the chez-* modules remain as legacy fallbacks. +These entries should be split over time into two groups: + +- Native boundary modules that remain handwritten Rust because they wrap mature + OS, crypto, TLS, database, or device APIs. +- Safe Jerboa kernels that should move to Typed Jerboa and generate Rust. + | chez-* (legacy) | Rust replacement | Jerboa module | |------------------|-----------------|---------------| | chez-crypto (OpenSSL) | ring | `(std crypto native-rust)` | @@ -156,3 +181,4 @@ SRFIs are a special case. They are standards-defined APIs that should be usable | Does it need concurrency, retry, or pooling? | jerboa | | Does it have a public API that should be contract-checked? | jerboa | | Does it manage resources that need cleanup? | jerboa | +| Does it need checked variants, ownership, effects, or generated native code? | Typed Jerboa | --- a/docs/index.md +++ b/docs/index.md @@ -10,6 +10,8 @@ Updated 2026-03-22. - [pending.md](pending.md) — Roadmap: what's left to make Jerboa better - [architecture-split.md](architecture-split.md) — Design: Rust native / chez-* (FFI backends) vs jerboa (application logic) - [native-rust.md](native-rust.md) — Rust native library: architecture, migration status, C ABI design +- [pure-jerboa-migration.md](pure-jerboa-migration.md) — Audit and migration plan for converting jerboa-* repos to pure Jerboa +- [typed-jerboa.md](typed-jerboa.md) — Typed Jerboa tier for static checks, resources, and generated native artifacts - [rocks.md](rocks.md) — Distribution and packaging ## Security @@ -61,10 +63,15 @@ Updated 2026-03-22. - [compiling-gerbil-projects.md](compiling-gerbil-projects.md) — Compiling Gerbil projects on Jerboa - [packages.md](packages.md) — Package management - [optimization.md](optimization.md) — Performance optimization +- [pure-jerboa-migration.md](pure-jerboa-migration.md) — Pure Jerboa migration scanner and triage workflow ## Rust Native Backend +- [rust-target.md](rust-target.md) — Safe Jerboa-to-Rust subset for explicit Rust delivery artifacts - [native-rust.md](native-rust.md) — Rust native backend architecture and implementation +- [typed-jerboa.md](typed-jerboa.md) — Source-level plan for typed modules that can target Rust and LLVM +- [jerboa-to-rust.md](jerboa-to-rust.md) — Typed Jerboa Rust backend plan +- [jerboa-to-llvmir.md](jerboa-to-llvmir.md) — Later direct LLVM IR backend plan - [vs-rust.md](vs-rust.md) — Jerboa vs Rust comparison - [ffi.md](ffi.md) — FFI interface design --- a/docs/native-rust.md +++ b/docs/native-rust.md @@ -1,5 +1,19 @@ # Replacing C Dependencies with a Unified Rust Native Library +Status note, 2026-05-20: this document describes the current handwritten Rust +native library strategy. The long-term architecture now distinguishes between +two different uses of Rust: + +- Explicit native boundary Rust for OS, crypto, TLS, sandboxing, Qt/Scintilla, + and other interfaces where reimplementing in Jerboa would be less secure or + unrealistic. +- Generated Rust from Typed Jerboa source for hardened Jerboa modules that need + native artifacts or static checking. + +The second category should move to [typed-jerboa.md](typed-jerboa.md) and +[jerboa-to-rust.md](jerboa-to-rust.md). Handwritten Rust should shrink toward +the first category only. + Jerboa's C library dependencies are being replaced with a single Rust shared library (`libjerboa_native.so`). Rust implementations are complete for crypto, compression, regex, databases, and OS integration. TLS (rustls) and LevelDB are still pending. The legacy chez-* C wrappers remain available as fallbacks. Every Rust module is callable from Chez Scheme via the same `foreign-procedure` FFI. --- @@ -677,6 +691,21 @@ The result is a single binary with zero runtime dependencies beyond libc — Che ## Migration Strategy +Typed Jerboa changes the migration target: + +1. Keep native Rust where it is the safest narrow boundary to OS, crypto, TLS, + sandboxing, database engines, Qt/Scintilla, or other mature native APIs. +2. Move application logic, protocol state, validation, resource policy, and + editor data models into dynamic or Typed Jerboa. +3. Replace handwritten Rust kernels with Typed Jerboa modules that generate + Rust through the backend described in [jerboa-to-rust.md](jerboa-to-rust.md). +4. Treat generated Rust as disposable build output with deterministic + generation, source comments, and tests. +5. Add LLVM only after the typed core and Rust backend prove the safety model. + +The rule is: native Rust remains acceptable as an audited boundary; Typed +Jerboa should become the source for new safe native Jerboa code. + ### Current Status We are in **Phase 1** (parallel installation). The Rust implementations are complete for crypto, compression, regex, databases (SQLite, PostgreSQL), and OS integration (epoll, inotify, landlock). However, the "default" modules (e.g., `(std db sqlite)`, `(std crypto cipher)`) still import from chez-* C libraries. The Rust-backed modules are available as separate imports (e.g., `(std db sqlite-native)`, `(std crypto native-rust)`). new file mode 100644 --- /dev/null +++ b/docs/pure-jerboa-migration.md @@ -0,0 +1,142 @@ +# Pure Jerboa Migration + +Pure Jerboa means the source of truth is Jerboa code, not handwritten Rust, +C/C++, JavaScript, shell orchestration, or ad hoc native loaders. + +That includes two Jerboa tiers: + +- Dynamic Jerboa: REPL-driven code, glue, orchestration, scripting, macros, and + user customization. +- Typed Jerboa: statically checked Jerboa modules for data structures, + parsers, compiler internals, protocol codecs, resource boundaries, and other + code that needs Rust-like guarantees. + +Generated backend artifacts are allowed when they are derived from Typed Jerboa +source. Generated Rust is therefore not the same kind of blocker as handwritten +Rust: the project should treat it as a build artifact, with drift checks and +regeneration tests. + +This is a migration target, not a slogan. Some capabilities may remain explicit +non-pure boundaries when the threat model demands hardened native code, kernel +interfaces, device APIs, or TLS implementations that Jerboa should not pretend +to replace yet. + +## Audit Tool + +Use the scanner before changing a sibling project: + +```sh +make pure-audit +make pure-audit PURE_AUDIT_ARGS="--summary --discover /Users/user/mine" +make pure-audit PURE_AUDIT_ARGS="/Users/user/mine/jerboa-websearch" +scheme --libdirs lib --script support/pure-audit.ss --summary --discover /Users/user/mine +``` + +The scanner is read-only. It reports migration blockers; it does not prove that +a project is semantically portable. + +Finding categories: + +| Category | Meaning | Usual action | +|---|---|---| +| `rust-build` | `Cargo.toml` in the project | Remove handwritten crate use, or make it a generated Typed Jerboa backend crate | +| `rust-source` | `.rs` source file | Rewrite as dynamic or Typed Jerboa; keep only generated Rust or an explicit native boundary | +| `rust-wrapper` | Jerboa module imports Rust-backed code | Replace with pure modules or keep as an explicit boundary | +| `c-native` | C/C++ source or headers | Replace with Jerboa or an audited OS boundary | +| `ffi-boundary` | Direct FFI declarations | Remove direct FFI or narrow it behind a capability module | +| `native-loader` | Dynamic shared object loading | Remove runtime native loading for pure Jerboa | +| `node-build` | `package.json` | Replace Node build/runtime dependency or mark build-only | +| `node-source` | JavaScript/TypeScript source | Port runtime code or isolate generated/static assets | +| `external-process` | Shell or subprocess orchestration | Replace with Jerboa APIs where possible | + +## Conversion Workflow + +1. Run `make pure-audit` for the whole `~/mine` tree. +2. Pick one project and run the full Markdown report with + `PURE_AUDIT_ARGS="/path/to/project"`. +3. Classify each finding as dynamic Jerboa rewrite, Typed Jerboa module, + generated backend artifact, or deliberate non-pure native boundary. +4. Write replacement code in Jerboa first. Use the Typed Jerboa Rust backend + for kernels that need native artifacts, static checking, or Rust as a + bootstrap safety checker. +5. Add project tests, then rerun the scanner until the project is clean or the + remaining boundaries are documented. + +## Typed Jerboa Migration Rule + +The default migration target is not "Scheme everywhere at all costs." The +default target is "Jerboa source everywhere feasible." + +Use dynamic Jerboa when: + +- Runtime flexibility matters. +- The code is orchestration, commands, configuration, scripting, or glue. +- Runtime checks and tests are enough. + +Use Typed Jerboa when: + +- Invalid states should be compile-time errors. +- Pattern matches should be exhaustive. +- Resource ownership, close/drop behavior, nullability, blocking, or Qt thread + affinity must be explicit. +- The module is a parser, codec, compiler pass, data structure, or editor model + kernel. +- A Rust or LLVM artifact is needed, but handwritten Rust should not be the + source of truth. + +Keep an explicit native boundary when: + +- The operating system or hardware API is inherently native. +- TLS, cryptography, sandboxing, kernel interfaces, Qt, Scintilla, or other + mature native components are safer to consume than to reimplement. +- The boundary is narrow, typed, audited, and covered by dynamic wrapper tests. + +## Current jerboa-* Inventory + +Snapshot from `/Users/user/mine` on 2026-05-20: + +| Project | Findings | Main blockers | +|---|---:|---| +| `jerboa-asm` | 0 | Clean | +| `jerboa-signal` | 0 | Clean | +| `jerboa-temp-dir` | 0 | Clean | +| `jerboa-db` | 1 | FFI boundary | +| `jerboa-edge` | 1 | Rust wrapper | +| `jerboa-websearch` | 2 | Rust crate | +| `jerboa-top` | 4 | Rust crate | +| `jerboa-ts-mode` | 8 | C/FFI and Node | +| `jerboa-sed` | 9 | FFI and native loader | +| `jerboa-webex` | 13 | C/FFI and native loader | +| `jerboa-gitsafe` | 14 | External processes | +| `jerboa-aws` | 18 | C/FFI and subprocesses | +| `jerboa-yubikey` | 27 | FFI, native loader, Rust crate | +| `jerboa-awk` | 33 | External processes and native loaders | +| `jerboa-dns` | 36 | FFI, native loader, Rust crate | +| `jerboa-lsp` | 37 | C/FFI and subprocesses | +| `jerboa-pgp` | 60 | Subprocesses, FFI, Rust crate | +| `jerboa-secmon` | 90 | FFI, subprocesses, Rust wrapper | +| `jerboa-virus` | 107 | Subprocesses, FFI, native loaders | +| `jerboa-wafter` | 186 | Subprocesses, FFI, Rust crate | +| `jerboa-mcp` | 474 | FFI, Node, Rust wrappers | +| `jerboa-inspect` | 1082 | FFI, native loaders, Rust crate | +| `jerboa-lora` | 3090 | C native code, Node, Rust wrappers | +| `jerboa-shell` | 6172 | FFI, native loaders, C/Rust surfaces | +| `jerboa-emacs` | 6541 | FFI, native loaders, C/Rust surfaces | + +Start with the clean and low-count projects to validate the workflow: +`jerboa-asm`, `jerboa-signal`, `jerboa-temp-dir`, `jerboa-websearch`, +`jerboa-top`, and `jerboa-db`. The large projects need staged plans because the +scanner is reporting broad native surfaces, not just isolated Rust files. + +## Relationship To Typed Backends + +[typed-jerboa.md](typed-jerboa.md), [jerboa-to-rust.md](jerboa-to-rust.md), and +[jerboa-to-llvmir.md](jerboa-to-llvmir.md) define the long-term replacement for +handwritten Rust. The source language is Typed Jerboa. Rust is the first native +backend because it gives the project a mature safety checker while the typed +core matures. LLVM IR comes later, after the typed core and Rust backend have +proven the safety model. + +[rust-target.md](rust-target.md) is now the historical MVP surface for this +path. It should be folded into the Typed Jerboa Rust backend rather than grown +as a separate language. --- a/docs/rust-target.md +++ b/docs/rust-target.md @@ -1,6 +1,14 @@ # Safe Rust Target -Jerboa can generate Rust for a deliberately small safe subset: +This document describes the existing minimal Rust code generator. Going +forward, it should be treated as the MVP seed of the Typed Jerboa Rust backend, +not as a separate language or a permanent user-facing DSL. + +The source of truth should become `(typed-library ...)` modules as described in +[typed-jerboa.md](typed-jerboa.md). Generated Rust should be disposable build +output, checked by tests and rebuilt from Typed Jerboa source. + +Jerboa can currently generate Rust for a deliberately small safe subset: - typed, pure functions - integer and boolean values @@ -10,10 +18,9 @@ Jerboa can generate Rust for a deliberately small safe subset: - `#![forbid(unsafe_code)]` in the generated Rust This is not a full Jerboa-to-Rust compiler. It intentionally rejects mutation, -FFI, `eval`, `read`, `lambda`, `quote`, and dynamic data structures. The goal is -to write small security-sensitive kernels, validators, parsers, and arithmetic -helpers in Jerboa syntax while producing boring safe Rust as the delivery -artifact. +FFI, `eval`, `read`, `lambda`, `quote`, and dynamic data structures. The short +term goal is to preserve this narrow, safe behavior while moving the front end +from the old `(module ... define ...)` form to Typed Jerboa forms. ```scheme (import (jerboa rust codegen)) @@ -54,6 +61,12 @@ The current API is `(jerboa rust codegen)`: - `safe-rust-form?` - `safe-rust-expression?` -Near-term extensions should stay narrow: typed byte slices, explicit `Result` -returns instead of panics on checked arithmetic failure, generated Rust tests, -and FFI wrapper generation from declarative Jerboa specs. +Near-term extensions should stay aligned with Typed Jerboa milestones: + +1. Accept a tiny `(typed-library ...)` subset and lower it to the existing safe + Rust forms. +2. Preserve source spans and produce useful parse/type errors. +3. Add explicit `Result` returns instead of panics on checked arithmetic + failure. +4. Generate Rust tests and Jerboa boundary tests. +5. Add records and variants before expanding FFI wrapper generation. --- a/docs/typed-jerboa.md +++ b/docs/typed-jerboa.md @@ -4,6 +4,12 @@ Typed Jerboa is a first-class typed tier for Jerboa programs. +Repository alignment: Typed Jerboa is the preferred replacement path for +handwritten Rust that exists only to get static checks or native artifacts. +Handwritten Rust should remain only for explicit native boundaries such as OS, +crypto, TLS, sandboxing, Qt/Scintilla, and other APIs where Jerboa should +consume a narrow audited interface instead of reimplementing the subsystem. + The goal is not to replace dynamic Jerboa. The goal is to let a Jerboa project choose stronger guarantees for the parts of the system that need them: