Remove obsolete Jerboa docs
ober
f27400df3e270f6960b010a1e6e6826e48c39ff6
deleted file mode 100644 --- a/docs/jerboa-to-llvmir.md +++ /dev/null @@ -1,736 +0,0 @@ -# Jerboa to LLVM IR Backend Plan - -## Purpose - -The LLVM IR backend is the eventual direct native backend for Typed Jerboa. - -LLVM should be treated as a code generator, not as the source of language -safety. Typed Jerboa must prove or enforce safety before lowering to LLVM IR. -LLVM IR is powerful, but it has raw pointers, undefined behavior, poison values, -alignment requirements, aliasing rules, and optimizer assumptions that can turn -small compiler mistakes into incorrect binaries. - -The LLVM backend should come after the typed core and Rust backend are stable. - -## Goals - -- Compile a safe subset of Typed Jerboa directly to LLVM IR. -- Preserve the same semantics as the Rust backend. -- Support native object files and static linking. -- Make memory layout explicit. -- Keep runtime helpers small and auditable. -- Differential-test LLVM output against the Rust backend. - -## Non-Goals - -- Do not target arbitrary dynamic Jerboa. -- Do not use LLVM to enforce memory safety. -- Do not emit LLVM before type, effect, and ownership checks pass. -- Do not start with full FFI/resource support. -- Do not optimize before correctness is proven. - -## Pipeline - -```text -typed Jerboa source - -> macro expansion - -> typed AST - -> typed core IR - -> safety checks - - type checking - - effect checking - - ownership checking - - bounds policy checking - - nullability checking - -> lowered backend-neutral IR - -> LLVM-safe IR - -> LLVM IR - -> llc / clang / linker - -> object file or native binary -``` - -The critical layer is the LLVM-safe IR. It should be lower-level than typed -core IR but still safe by construction. - -## Why Not LLVM First - -Rust gives the project a mature safety checker and native backend while the -Typed Jerboa language is still evolving. LLVM gives more control, but it also -requires the Typed Jerboa compiler to get every memory and aliasing invariant -right. - -The pragmatic order: - -1. Design typed core IR. -2. Compile typed core IR to Rust. -3. Validate semantics with real modules. -4. Add LLVM backend for a pure subset. -5. Expand LLVM support as safety checks mature. - -## Backend Scope by Phase - -### Phase 1: Pure Scalar Subset - -Support: - -- `Unit` -- `Bool` -- `Int` -- `Nat` -- `Fixnum` -- `Float` -- Arithmetic -- Comparisons -- `if` -- `let` -- Direct function calls - -No heap allocation. - -### Phase 2: Records - -Support: - -- Immutable records -- Record construction -- Field access -- Passing records by value or pointer -- Returning records - -No mutation yet. - -### Phase 3: Variants - -Support: - -- Tagged unions -- Constructors -- Exhaustive pattern matching -- Payload access - -### Phase 4: Managed Heap Values - -Support: - -- Strings -- Bytes -- Vectors -- Heap records -- Heap variants - -Requires runtime allocation helpers. - -### Phase 5: Dynamic Boundary - -Support: - -- Converting dynamic Jerboa values to typed values -- Converting typed values to dynamic Jerboa values -- Structured error return -- Generated wrapper functions - -### Phase 6: Resources and FFI - -Support: - -- Linear resource handles -- Non-null native handles -- Nullable native handles -- Ownership transfer -- Thread-affinity metadata -- Checked FFI wrappers - -This phase should wait until the Rust backend has proven the resource model. - -## LLVM-Safe IR - -The LLVM backend should not lower directly from high-level typed core. It should -use an intermediate representation that makes unsafe operations explicit. - -LLVM-safe IR should contain: - -- Primitive operations with explicit overflow behavior -- Loads and stores with known alignment -- Allocations through runtime helpers -- Bounds-checked indexing -- Explicit null checks before pointer use -- Explicit tag checks for variants -- Explicit ownership operations -- Explicit dynamic boundary conversions -- Explicit cleanup paths - -No LLVM-safe IR node should mean "trust me." If an operation is unsafe, it -should be represented as an unsafe node that requires a prior proof or explicit -unsafe declaration. - -## Type Mapping - -Initial LLVM mapping: - -| Typed Jerboa | LLVM | -| --- | --- | -| `Unit` | `void` or zero-sized convention | -| `Bool` | `i1` at computation, `i8` at ABI boundary | -| `Char` | `i32` Unicode scalar | -| `Int` | `i64` | -| `Nat` | `i64` with non-negative invariant or `u64` convention | -| `Fixnum` | target pointer-sized integer | -| `Float` | `double` | -| `String` | runtime string pointer | -| `Bytes` | runtime bytes pointer | -| Record | LLVM struct | -| Variant | tagged struct | -| `(Option T)` | optimized nullable/tagged representation | -| `(Result T E)` | tagged representation | - -The MVP should avoid clever layout optimizations. Use clear tagged layouts. - -## Numeric Semantics - -Typed Jerboa must define numeric behavior before LLVM lowering. - -Options: - -- Checked arithmetic: overflow returns error or raises condition. -- Wrapping arithmetic: explicit operation names only. -- Saturating arithmetic: explicit operation names only. -- Unbounded integers: runtime big integer support. - -Recommended MVP: - -- `Fixnum`, `Int`, and `Nat` arithmetic is checked by default. -- Explicit operations exist for wrapping arithmetic. -- Overflow paths return a structured runtime error. - -LLVM lowering should use: - -- `llvm.sadd.with.overflow.*` -- `llvm.uadd.with.overflow.*` -- Equivalent subtract/multiply intrinsics - -Do not rely on LLVM `nsw` or `nuw` flags unless the compiler has proven the -operation cannot overflow. - -## Bounds Checking - -All vector, string, byte, and slice indexing should be bounds checked unless -the compiler has a local proof. - -Pattern: - -```text -if index < length: - load element -else: - return bounds error -``` - -Optimization can remove redundant checks later. - -## Nullability - -Typed Jerboa should distinguish: - -- `(Nullable T)` -- `(NonNull T)` -- `(Option T)` - -LLVM pointer values must be checked before use unless their type is already -`NonNull` and the compiler knows how that invariant was established. - -FFI declarations must say whether a pointer can be null. - -## Memory Management Options - -The LLVM backend needs a runtime memory strategy. It can start with one and add -others later. - -### Option 1: Runtime GC - -Pros: - -- Best fit for Scheme-like values -- Simpler source language -- Easy sharing - -Cons: - -- Requires collector integration -- Harder static/runtime story -- More runtime complexity - -### Option 2: Reference Counting - -Pros: - -- Predictable destruction -- Easier interop with resources -- Straightforward in generated IR - -Cons: - -- Cycles need special handling -- Retain/release overhead -- Optimizer must remove redundant refcount traffic - -### Option 3: Regions and Arenas - -Pros: - -- Simple and fast for compiler passes, parsers, temporary trees -- Easy cleanup -- Strong locality - -Cons: - -- Values cannot outlive regions -- Not a complete memory model for general programs - -### Option 4: Hybrid - -Recommended long-term: - -- Managed heap for ordinary persistent values -- Arenas for compiler/parser temporary values -- Linear ownership for resources -- Borrowed views for slices - -MVP recommendation: - -- Use explicit runtime allocation helpers. -- Use owned heap objects with manual retain/release or simple leak-on-exit for - the very first prototype. -- Add real lifetime management before production use. - -## Runtime Object Header - -Heap values need a common header: - -```c -typedef struct JtHeader { - uint32_t type_id; - uint32_t flags; - uint64_t refcount; -} JtHeader; -``` - -Possible flags: - -- Frozen / immutable -- Has destructor -- Contains pointers -- Resource wrapper -- Interned - -The LLVM backend should generate type descriptors: - -```c -typedef struct JtTypeDesc { - uint32_t type_id; - const char *name; - void (*drop)(void *); - void (*trace)(void *, JtTracer *); -} JtTypeDesc; -``` - -Even if the MVP does not use a tracing GC, type descriptors are useful for -debugging, dynamic boundary checks, and future collectors. - -## Records Layout - -For immutable records: - -```llvm -%Position = type { i64, i64 } -``` - -For heap records: - -```llvm -%PositionHeap = type { %JtHeader, i64, i64 } -``` - -The compiler should choose stack vs heap representation based on escape -analysis later. MVP can use heap representation for values crossing boundaries -and stack representation for internal scalar records. - -## Variant Layout - -Simple tagged layout: - -```llvm -%EditOp = type { - i32, ; tag - [N x i8] ; payload storage -} -``` - -Clear but inefficient. - -Later optimization can generate per-variant structs and use more compact -representations. - -## Function ABI - -Internal functions can use LLVM-native signatures. - -Boundary functions should use C ABI-compatible signatures: - -```llvm -define i32 @jt_rope_length(%JtRuntime* %rt, %JtHandle %rope, i64* %out) -``` - -Recommended boundary convention: - -- Return status code. -- Write successful result to out pointer. -- Store structured error in runtime context. - -This avoids complex ABI returns and maps cleanly to Scheme wrappers. - -## Error Handling - -Avoid LLVM exceptions in the MVP. - -Use explicit status returns: - -```text -0 = ok -1 = type error -2 = bounds error -3 = overflow error -4 = resource error -5 = FFI error -``` - -Runtime context stores error details: - -- Error kind -- Message -- Source span -- Optional payload - -This maps well to Jerboa conditions. - -## Effects Lowering - -Effect metadata should guide codegen: - -- `pure`: no runtime side effect calls except allocation if also `alloc` -- `alloc`: can call allocator -- `mut`: can perform stores or call mutating runtime helpers -- `io`: can call runtime IO helpers -- `ffi`: can call external symbols -- `throw`: can return nonzero status -- `block`: marked in metadata -- `qt`: must call Qt-thread trampoline -- `unsafe`: emits unsafe marker metadata and requires audit - -LLVM metadata can annotate functions, but correctness should not depend on -metadata. It should be enforced by the compiler before lowering. - -## Aliasing - -LLVM aliasing is dangerous. Do not emit strong aliasing metadata until the -compiler can prove it. - -Early backend rules: - -- Avoid `noalias` unless generated for a freshly allocated object. -- Avoid `nonnull` unless checked or guaranteed by type. -- Avoid `dereferenceable` unless size and lifetime are known. -- Avoid `nsw` and `nuw` unless overflow is proven impossible. - -Correctness first. Optimizer hints later. - -## Lifetime Markers - -LLVM lifetime markers are optimization hints. Incorrect markers can cause -miscompilation. - -Do not emit lifetime markers in the MVP. - -Add them only after: - -- Escape analysis exists. -- Stack allocation lowering is tested. -- Differential tests cover lifetime-heavy cases. - -## Garbage Collection Integration - -If using a tracing GC later, LLVM supports GC strategies, but integrating them -is nontrivial. - -Simpler initial paths: - -- Use a custom runtime allocation API. -- Make all heap references opaque runtime pointers. -- Let runtime functions manage tracing or refcounts. -- Keep LLVM-generated code conservative. - -Potential allocation API: - -```c -void *jt_alloc(JtRuntime *rt, uint32_t type_id, size_t size); -void jt_retain(void *ptr); -void jt_release(void *ptr); -``` - -## Dynamic Boundary - -The LLVM backend must generate dynamic wrappers, like the Rust backend. - -Boundary conversion should be explicit: - -```text -dynamic value -> checked typed value -> compiled call -> typed result -> dynamic value -``` - -Failure cases: - -- Wrong dynamic type -- Stale handle -- Null handle -- Bounds error -- Overflow error -- Resource already closed - -All should become Scheme conditions with source context. - -## FFI Boundary - -FFI declarations should lower to checked wrappers. - -Typed declaration: - -```scheme -(extern qt-widget-hide! - ([w : (NonNull QtWidget)]) : Unit - #:effects (ffi qt mut)) -``` - -LLVM lowering: - -- Verify `w` is non-null before call, unless statically guaranteed. -- Verify current thread or route through Qt trampoline for `qt` effect. -- Call external symbol. -- Convert error status where applicable. - -Do not call arbitrary C symbols directly from high-level IR. - -## Debug Info - -The LLVM backend should eventually emit DWARF debug info mapped to Typed Jerboa -source. - -MVP can emit comments in `.ll` files and source span metadata in runtime error -messages. - -Later: - -- `DIFile` -- `DISubprogram` -- `DILocation` -- Variable debug info - -Debug info matters for gdb-backed stress tests. - -## Optimization Strategy - -Optimization levels: - -- `O0`: debug, readable IR -- `O1`: simple cleanup -- `O2`: production -- `O3`: optional, benchmark-driven - -MVP should run at `O0` or `O1`. Correctness and diagnostics matter more than -speed at first. - -Later optimization passes: - -- Remove redundant bounds checks. -- Remove redundant refcount operations. -- Inline small functions. -- Stack allocate non-escaping records. -- Specialize generic functions. -- Compact variant layouts. - -## Tooling - -Useful tools: - -- `llvm-as` -- `llvm-dis` -- `opt` -- `llc` -- `clang` -- `lli` for tiny prototypes - -Build targets: - -```make -typed-llvm -typed-llvm-check -typed-llvm-build -typed-llvm-test -typed-llvm-clean -typed-llvm-dump -``` - -Suggested behavior: - -- `typed-llvm-check`: emit and verify LLVM IR -- `typed-llvm-build`: compile object/library -- `typed-llvm-test`: run backend tests -- `typed-llvm-dump`: emit readable IR for inspection - -## Verification - -LLVM IR should be verified before object generation: - -```bash -llvm-as module.ll -o module.bc -opt -verify module.bc -o /dev/null -llc module.bc -filetype=obj -o module.o -``` - -Compiler tests should fail if LLVM verification fails. - -## Differential Testing - -Every LLVM-supported typed module should be runnable through at least two paths: - -- Typed interpreter or evaluator -- Rust backend -- LLVM backend - -Compare outputs for the same generated inputs. - -Good candidates: - -- Pure arithmetic functions -- Record transformations -- Variant pattern matching -- Rope operations -- Split-tree transformations -- Parser helpers - -Fuzzing is valuable once records and variants work. - -## First LLVM Module - -Recommended first LLVM module: pure numeric and record helpers, not rope. - -Example: - -```scheme -(typed-library (jerboa typed demo) - (export clamp position-shift) - - (record Position - ([line : Nat] - [column : Nat])) - - (def (clamp [x : Int] [lo : Int] [hi : Int]) : Int - ...) - - (def (position-shift [p : Position] [columns : Nat]) : Position - ...)) -``` - -Then move to split-tree once variants are supported. - -## Milestones - -### Milestone 1: LLVM Emitter Skeleton - -- Emit one function returning an integer. -- Run LLVM verifier. -- Compile to object. -- Link and call from a tiny C or Jerboa harness. - -### Milestone 2: Scalar Expressions - -- Lower literals, variables, `let`, `if`. -- Lower arithmetic with overflow checks. -- Lower comparisons. -- Add status-code error returns. - -### Milestone 3: Records - -- Lower immutable records. -- Support construction and field access. -- Pass records to functions. -- Return records. - -### Milestone 4: Variants - -- Lower tagged unions. -- Lower pattern matching. -- Add exhaustive-match assumptions from typed checker. - -### Milestone 5: Runtime Allocation - -- Add runtime allocator. -- Add string and bytes representation. -- Add vector representation. -- Add retain/release or temporary allocation strategy. - -### Milestone 6: Dynamic Boundary - -- Generate C ABI wrappers. -- Convert dynamic Jerboa values. -- Return structured errors. -- Add Jerboa wrapper `.sls`. - -### Milestone 7: Backend Parity - -- Run same typed module through Rust and LLVM. -- Differential-test outputs. -- Add CI target for LLVM-supported subset. - -### Milestone 8: Resource and FFI Prototype - -- Add non-null handle type. -- Add checked FFI wrapper. -- Add one safe native call. -- Prove no null pointer can reach generated native call. - -## Risks - -- LLVM UB can silently miscompile code. -- Runtime memory model can become too complex. -- Debugging generated IR can be slow. -- ABI design mistakes are painful to change. -- Premature optimization can obscure correctness bugs. - -Mitigations: - -- Start with a tiny safe subset. -- Verify IR every build. -- Avoid aggressive LLVM attributes early. -- Differential-test against Rust backend. -- Keep runtime helpers small and audited. -- Require source spans on generated operations. - -## Success Criteria - -The LLVM backend is successful when: - -- It compiles a safe subset of Typed Jerboa to verified LLVM IR. -- The resulting object can be linked and called from Jerboa. -- Runtime errors are reported as Scheme conditions. -- It matches Rust backend behavior on supported modules. -- It does not require changing Typed Jerboa source code. -- The compiler enforces safety before LLVM lowering. - -The LLVM backend is production-ready when: - -- Memory management is sound. -- Bounds checks and overflow checks are correct. -- Resource ownership is enforced. -- FFI nullability and thread affinity are enforced. -- Debug info points back to Typed Jerboa source. -- CI runs backend parity tests. -- Static binary builds can include LLVM-generated modules. deleted file mode 100644 --- a/docs/jerboa-to-rust.md +++ /dev/null @@ -1,612 +0,0 @@ -# Jerboa to Rust Backend Plan - -## Purpose - -The Rust backend is the first native backend for Typed Jerboa. - -Rust is not the desired surface language. Typed Jerboa is. Rust is the first -backend because it gives the project a mature compiler, strong memory safety, -good diagnostics, good cross-compilation, and a practical way to validate typed -Jerboa semantics before taking on a direct LLVM backend. - -The Rust backend should compile typed Jerboa modules into Rust crates, compile -those crates into native artifacts, and generate Jerboa wrappers so dynamic -Jerboa code can call typed compiled code safely. - -## Goals - -- Compile typed Jerboa modules to safe Rust where possible. -- Use Rust's type checker as a second line of defense. -- Generate readable Rust for debugging. -- Generate stable FFI exports. -- Generate dynamic Jerboa wrappers. -- Preserve Jerboa source locations in diagnostics where possible. -- Keep Rust-specific concepts out of the Typed Jerboa surface language. -- Make generated artifacts deterministic. - -## Non-Goals - -- Do not translate arbitrary dynamic Jerboa to Rust. -- Do not expose Rust lifetimes directly in Typed Jerboa syntax. -- Do not require users to edit generated Rust. -- Do not use `unsafe` except in small, generated runtime and FFI shims. -- Do not make Rust crates the canonical source of typed modules. - -## Pipeline - -```text -typed Jerboa source - -> macro expansion - -> typed AST - -> typed core IR - -> effect and ownership checks - -> lowered backend-neutral IR - -> Rust AST / Rust text - -> cargo or rustc - -> static library, dynamic library, or object file - -> generated Jerboa wrappers -``` - -The Rust backend should consume a lowered IR, not surface syntax. That keeps -the language independent from Rust and makes the later LLVM backend practical. - -## Output Layout - -Candidate layout: - -```text -build/typed/rust/Cargo.toml -build/typed/rust/src/lib.rs -build/typed/rust/src/modules/<module>.rs -build/typed/rust/src/runtime.rs -build/typed/rust/include/<module>.h -build/typed/rust/target/... -lib/jerboa-emacs/typed/<module>.sls -``` - -The generated Rust crate should be disposable. The source of truth remains the -typed Jerboa source. - -## Artifact Modes - -The backend should eventually support three artifact modes: - -### Dynamic Library - -Useful during development: - -- Faster incremental builds -- Easy `LD_PRELOAD` or `load-shared-object` -- Clear separation between Jerboa and compiled code - -### Static Library - -Useful for static binary builds: - -- Link into `jemacs-qt` -- No runtime `.so` dependency -- Better deployment story - -### Object File - -Useful for advanced build integration: - -- Direct linker control -- Works well with an eventual whole-program static build - -MVP should start with a dynamic library, then add static library mode. - -## Rust Crate Shape - -Generated crate: - -```rust -pub mod runtime; -pub mod modules; - -#[no_mangle] -pub extern "C" fn jerboa_typed_init() -> i32 { - 0 -} -``` - -Each typed module gets a Rust module: - -```rust -pub mod jerboa_emacs_typed_split_tree; -pub mod jerboa_emacs_typed_rope; -``` - -Exports are exposed through C ABI functions: - -```rust -#[no_mangle] -pub extern "C" fn jt_rope_length(rope: JtHandle) -> JtResultUsize { - ... -} -``` - -The C ABI layer should be thin. Internal generated code should use normal safe -Rust types. - -## Type Mapping - -Initial type mapping: - -| Typed Jerboa | Rust | -| --- | --- | -| `Unit` | `()` | -| `Bool` | `bool` | -| `Char` | `char` | -| `Int` | `i64` | -| `Nat` | `u64` or checked `usize` depending on context | -| `Fixnum` | `isize` | -| `Float` | `f64` | -| `String` | `String` | -| `Bytes` | `Vec<u8>` | -| `Symbol` | runtime interned symbol | -| `(Option T)` | `Option<T>` | -| `(Result T E)` | `Result<T, E>` | -| `(Vector T)` | `Vec<T>` | -| `(List T)` | `Vec<T>` in MVP, persistent list later | -| Record | `struct` | -| Variant | `enum` | - -Boundary-facing values should use ABI-safe wrappers instead of raw Rust layout. - -## Records - -Typed Jerboa: - -```scheme -(record Position - ([line : Nat] - [column : Nat])) -``` - -Rust: - -```rust -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Position { - pub line: u64, - pub column: u64, -} -``` - -Mutable fields should generate methods instead of public mutable fields when -possible: - -```rust -impl Cursor { - pub fn set_point(&mut self, point: u64) { - self.point = point; - } -} -``` - -## Variants - -Typed Jerboa: - -```scheme -(variant EditOp - (Insert [at : Nat] [text : String]) - (Delete [start : Nat] [end : Nat])) -``` - -Rust: - -```rust -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum EditOp { - Insert { at: u64, text: String }, - Delete { start: u64, end: u64 }, -} -``` - -Pattern matching should lower directly to Rust `match` after the Typed Jerboa -checker has already verified exhaustiveness. - -## Functions - -Typed Jerboa: - -```scheme -(def (line-count [text : String]) : Nat - ...) -``` - -Rust: - -```rust -pub fn line_count(text: String) -> u64 { - ... -} -``` - -Internal Rust functions should use safe Rust signatures. Exported functions -should get separate ABI wrappers. - -## Generics - -The first Rust backend should monomorphize generics. - -Typed Jerboa: - -```scheme -(def (option-default [x : (Option A)] [fallback : A]) : A - ...) -``` - -For each concrete use, generate a concrete Rust function: - -```rust -fn option_default_string(x: Option<String>, fallback: String) -> String { ... } -fn option_default_u64(x: Option<u64>, fallback: u64) -> u64 { ... } -``` - -This avoids exposing generic ABI problems at the boundary. - -## Effects - -Typed Jerboa effects should influence generated Rust: - -- `pure`: ordinary safe function -- `alloc`: may allocate -- `mut`: takes `&mut` or owned updated values -- `io`: returns `Result` -- `ffi`: calls generated FFI shim