Add typed Jerboa planning docs
ober
c5c4d58c80b44b91949657cfb0ebe7c0f4847ff6
new file mode 100644 --- /dev/null +++ b/docs/jerboa-to-llvmir.md @@ -0,0 +1,736 @@ +# Jerboa to LLVM IR Backend Plan + +## Purpose + +The LLVM IR backend is the eventual direct native backend for Typed Jerboa. + +LLVM should be treated as a code generator, not as the source of language +safety. Typed Jerboa must prove or enforce safety before lowering to LLVM IR. +LLVM IR is powerful, but it has raw pointers, undefined behavior, poison values, +alignment requirements, aliasing rules, and optimizer assumptions that can turn +small compiler mistakes into incorrect binaries. + +The LLVM backend should come after the typed core and Rust backend are stable. + +## Goals + +- Compile a safe subset of Typed Jerboa directly to LLVM IR. +- Preserve the same semantics as the Rust backend. +- Support native object files and static linking. +- Make memory layout explicit. +- Keep runtime helpers small and auditable. +- Differential-test LLVM output against the Rust backend. + +## Non-Goals + +- Do not target arbitrary dynamic Jerboa. +- Do not use LLVM to enforce memory safety. +- Do not emit LLVM before type, effect, and ownership checks pass. +- Do not start with full FFI/resource support. +- Do not optimize before correctness is proven. + +## Pipeline + +```text +typed Jerboa source + -> macro expansion + -> typed AST + -> typed core IR + -> safety checks + - type checking + - effect checking + - ownership checking + - bounds policy checking + - nullability checking + -> lowered backend-neutral IR + -> LLVM-safe IR + -> LLVM IR + -> llc / clang / linker + -> object file or native binary +``` + +The critical layer is the LLVM-safe IR. It should be lower-level than typed +core IR but still safe by construction. + +## Why Not LLVM First + +Rust gives the project a mature safety checker and native backend while the +Typed Jerboa language is still evolving. LLVM gives more control, but it also +requires the Typed Jerboa compiler to get every memory and aliasing invariant +right. + +The pragmatic order: + +1. Design typed core IR. +2. Compile typed core IR to Rust. +3. Validate semantics with real modules. +4. Add LLVM backend for a pure subset. +5. Expand LLVM support as safety checks mature. + +## Backend Scope by Phase + +### Phase 1: Pure Scalar Subset + +Support: + +- `Unit` +- `Bool` +- `Int` +- `Nat` +- `Fixnum` +- `Float` +- Arithmetic +- Comparisons +- `if` +- `let` +- Direct function calls + +No heap allocation. + +### Phase 2: Records + +Support: + +- Immutable records +- Record construction +- Field access +- Passing records by value or pointer +- Returning records + +No mutation yet. + +### Phase 3: Variants + +Support: + +- Tagged unions +- Constructors +- Exhaustive pattern matching +- Payload access + +### Phase 4: Managed Heap Values + +Support: + +- Strings +- Bytes +- Vectors +- Heap records +- Heap variants + +Requires runtime allocation helpers. + +### Phase 5: Dynamic Boundary + +Support: + +- Converting dynamic Jerboa values to typed values +- Converting typed values to dynamic Jerboa values +- Structured error return +- Generated wrapper functions + +### Phase 6: Resources and FFI + +Support: + +- Linear resource handles +- Non-null native handles +- Nullable native handles +- Ownership transfer +- Thread-affinity metadata +- Checked FFI wrappers + +This phase should wait until the Rust backend has proven the resource model. + +## LLVM-Safe IR + +The LLVM backend should not lower directly from high-level typed core. It should +use an intermediate representation that makes unsafe operations explicit. + +LLVM-safe IR should contain: + +- Primitive operations with explicit overflow behavior +- Loads and stores with known alignment +- Allocations through runtime helpers +- Bounds-checked indexing +- Explicit null checks before pointer use +- Explicit tag checks for variants +- Explicit ownership operations +- Explicit dynamic boundary conversions +- Explicit cleanup paths + +No LLVM-safe IR node should mean "trust me." If an operation is unsafe, it +should be represented as an unsafe node that requires a prior proof or explicit +unsafe declaration. + +## Type Mapping + +Initial LLVM mapping: + +| Typed Jerboa | LLVM | +| --- | --- | +| `Unit` | `void` or zero-sized convention | +| `Bool` | `i1` at computation, `i8` at ABI boundary | +| `Char` | `i32` Unicode scalar | +| `Int` | `i64` | +| `Nat` | `i64` with non-negative invariant or `u64` convention | +| `Fixnum` | target pointer-sized integer | +| `Float` | `double` | +| `String` | runtime string pointer | +| `Bytes` | runtime bytes pointer | +| Record | LLVM struct | +| Variant | tagged struct | +| `(Option T)` | optimized nullable/tagged representation | +| `(Result T E)` | tagged representation | + +The MVP should avoid clever layout optimizations. Use clear tagged layouts. + +## Numeric Semantics + +Typed Jerboa must define numeric behavior before LLVM lowering. + +Options: + +- Checked arithmetic: overflow returns error or raises condition. +- Wrapping arithmetic: explicit operation names only. +- Saturating arithmetic: explicit operation names only. +- Unbounded integers: runtime big integer support. + +Recommended MVP: + +- `Fixnum`, `Int`, and `Nat` arithmetic is checked by default. +- Explicit operations exist for wrapping arithmetic. +- Overflow paths return a structured runtime error. + +LLVM lowering should use: + +- `llvm.sadd.with.overflow.*` +- `llvm.uadd.with.overflow.*` +- Equivalent subtract/multiply intrinsics + +Do not rely on LLVM `nsw` or `nuw` flags unless the compiler has proven the +operation cannot overflow. + +## Bounds Checking + +All vector, string, byte, and slice indexing should be bounds checked unless +the compiler has a local proof. + +Pattern: + +```text +if index < length: + load element +else: + return bounds error +``` + +Optimization can remove redundant checks later. + +## Nullability + +Typed Jerboa should distinguish: + +- `(Nullable T)` +- `(NonNull T)` +- `(Option T)` + +LLVM pointer values must be checked before use unless their type is already +`NonNull` and the compiler knows how that invariant was established. + +FFI declarations must say whether a pointer can be null. + +## Memory Management Options + +The LLVM backend needs a runtime memory strategy. It can start with one and add +others later. + +### Option 1: Runtime GC + +Pros: + +- Best fit for Scheme-like values +- Simpler source language +- Easy sharing + +Cons: + +- Requires collector integration +- Harder static/runtime story +- More runtime complexity + +### Option 2: Reference Counting + +Pros: + +- Predictable destruction +- Easier interop with resources +- Straightforward in generated IR + +Cons: + +- Cycles need special handling +- Retain/release overhead +- Optimizer must remove redundant refcount traffic + +### Option 3: Regions and Arenas + +Pros: + +- Simple and fast for compiler passes, parsers, temporary trees +- Easy cleanup +- Strong locality + +Cons: + +- Values cannot outlive regions +- Not a complete memory model for general programs + +### Option 4: Hybrid + +Recommended long-term: + +- Managed heap for ordinary persistent values +- Arenas for compiler/parser temporary values +- Linear ownership for resources +- Borrowed views for slices + +MVP recommendation: + +- Use explicit runtime allocation helpers. +- Use owned heap objects with manual retain/release or simple leak-on-exit for + the very first prototype. +- Add real lifetime management before production use. + +## Runtime Object Header + +Heap values need a common header: + +```c +typedef struct JtHeader { + uint32_t type_id; + uint32_t flags; + uint64_t refcount; +} JtHeader; +``` + +Possible flags: + +- Frozen / immutable +- Has destructor +- Contains pointers +- Resource wrapper +- Interned + +The LLVM backend should generate type descriptors: + +```c +typedef struct JtTypeDesc { + uint32_t type_id; + const char *name; + void (*drop)(void *); + void (*trace)(void *, JtTracer *); +} JtTypeDesc; +``` + +Even if the MVP does not use a tracing GC, type descriptors are useful for +debugging, dynamic boundary checks, and future collectors. + +## Records Layout + +For immutable records: + +```llvm +%Position = type { i64, i64 } +``` + +For heap records: + +```llvm +%PositionHeap = type { %JtHeader, i64, i64 } +``` + +The compiler should choose stack vs heap representation based on escape +analysis later. MVP can use heap representation for values crossing boundaries +and stack representation for internal scalar records. + +## Variant Layout + +Simple tagged layout: + +```llvm +%EditOp = type { + i32, ; tag + [N x i8] ; payload storage +} +``` + +Clear but inefficient. + +Later optimization can generate per-variant structs and use more compact +representations. + +## Function ABI + +Internal functions can use LLVM-native signatures. + +Boundary functions should use C ABI-compatible signatures: + +```llvm +define i32 @jt_rope_length(%JtRuntime* %rt, %JtHandle %rope, i64* %out) +``` + +Recommended boundary convention: + +- Return status code. +- Write successful result to out pointer. +- Store structured error in runtime context. + +This avoids complex ABI returns and maps cleanly to Scheme wrappers. + +## Error Handling + +Avoid LLVM exceptions in the MVP. + +Use explicit status returns: + +```text +0 = ok +1 = type error +2 = bounds error +3 = overflow error +4 = resource error +5 = FFI error +``` + +Runtime context stores error details: + +- Error kind +- Message +- Source span +- Optional payload + +This maps well to Jerboa conditions. + +## Effects Lowering + +Effect metadata should guide codegen: + +- `pure`: no runtime side effect calls except allocation if also `alloc` +- `alloc`: can call allocator +- `mut`: can perform stores or call mutating runtime helpers +- `io`: can call runtime IO helpers +- `ffi`: can call external symbols +- `throw`: can return nonzero status +- `block`: marked in metadata +- `qt`: must call Qt-thread trampoline +- `unsafe`: emits unsafe marker metadata and requires audit + +LLVM metadata can annotate functions, but correctness should not depend on +metadata. It should be enforced by the compiler before lowering. + +## Aliasing + +LLVM aliasing is dangerous. Do not emit strong aliasing metadata until the +compiler can prove it. + +Early backend rules: + +- Avoid `noalias` unless generated for a freshly allocated object. +- Avoid `nonnull` unless checked or guaranteed by type. +- Avoid `dereferenceable` unless size and lifetime are known. +- Avoid `nsw` and `nuw` unless overflow is proven impossible. + +Correctness first. Optimizer hints later. + +## Lifetime Markers + +LLVM lifetime markers are optimization hints. Incorrect markers can cause +miscompilation. + +Do not emit lifetime markers in the MVP. + +Add them only after: + +- Escape analysis exists. +- Stack allocation lowering is tested. +- Differential tests cover lifetime-heavy cases. + +## Garbage Collection Integration + +If using a tracing GC later, LLVM supports GC strategies, but integrating them +is nontrivial. + +Simpler initial paths: + +- Use a custom runtime allocation API. +- Make all heap references opaque runtime pointers. +- Let runtime functions manage tracing or refcounts. +- Keep LLVM-generated code conservative. + +Potential allocation API: + +```c +void *jt_alloc(JtRuntime *rt, uint32_t type_id, size_t size); +void jt_retain(void *ptr); +void jt_release(void *ptr); +``` + +## Dynamic Boundary + +The LLVM backend must generate dynamic wrappers, like the Rust backend. + +Boundary conversion should be explicit: + +```text +dynamic value -> checked typed value -> compiled call -> typed result -> dynamic value +``` + +Failure cases: + +- Wrong dynamic type +- Stale handle +- Null handle +- Bounds error +- Overflow error +- Resource already closed + +All should become Scheme conditions with source context. + +## FFI Boundary + +FFI declarations should lower to checked wrappers. + +Typed declaration: + +```scheme +(extern qt-widget-hide! + ([w : (NonNull QtWidget)]) : Unit + #:effects (ffi qt mut)) +``` + +LLVM lowering: + +- Verify `w` is non-null before call, unless statically guaranteed. +- Verify current thread or route through Qt trampoline for `qt` effect. +- Call external symbol. +- Convert error status where applicable. + +Do not call arbitrary C symbols directly from high-level IR. + +## Debug Info + +The LLVM backend should eventually emit DWARF debug info mapped to Typed Jerboa +source. + +MVP can emit comments in `.ll` files and source span metadata in runtime error +messages. + +Later: + +- `DIFile` +- `DISubprogram` +- `DILocation` +- Variable debug info + +Debug info matters for gdb-backed stress tests. + +## Optimization Strategy + +Optimization levels: + +- `O0`: debug, readable IR +- `O1`: simple cleanup +- `O2`: production +- `O3`: optional, benchmark-driven + +MVP should run at `O0` or `O1`. Correctness and diagnostics matter more than +speed at first. + +Later optimization passes: + +- Remove redundant bounds checks. +- Remove redundant refcount operations. +- Inline small functions. +- Stack allocate non-escaping records. +- Specialize generic functions. +- Compact variant layouts. + +## Tooling + +Useful tools: + +- `llvm-as` +- `llvm-dis` +- `opt` +- `llc` +- `clang` +- `lli` for tiny prototypes + +Build targets: + +```make +typed-llvm +typed-llvm-check +typed-llvm-build +typed-llvm-test +typed-llvm-clean +typed-llvm-dump +``` + +Suggested behavior: + +- `typed-llvm-check`: emit and verify LLVM IR +- `typed-llvm-build`: compile object/library +- `typed-llvm-test`: run backend tests +- `typed-llvm-dump`: emit readable IR for inspection + +## Verification + +LLVM IR should be verified before object generation: + +```bash +llvm-as module.ll -o module.bc +opt -verify module.bc -o /dev/null +llc module.bc -filetype=obj -o module.o +``` + +Compiler tests should fail if LLVM verification fails. + +## Differential Testing + +Every LLVM-supported typed module should be runnable through at least two paths: + +- Typed interpreter or evaluator +- Rust backend +- LLVM backend + +Compare outputs for the same generated inputs. + +Good candidates: + +- Pure arithmetic functions +- Record transformations +- Variant pattern matching +- Rope operations +- Split-tree transformations +- Parser helpers + +Fuzzing is valuable once records and variants work. + +## First LLVM Module + +Recommended first LLVM module: pure numeric and record helpers, not rope. + +Example: + +```scheme +(typed-library (jerboa typed demo) + (export clamp position-shift) + + (record Position + ([line : Nat] + [column : Nat])) + + (def (clamp [x : Int] [lo : Int] [hi : Int]) : Int + ...) + + (def (position-shift [p : Position] [columns : Nat]) : Position + ...)) +``` + +Then move to split-tree once variants are supported. + +## Milestones + +### Milestone 1: LLVM Emitter Skeleton + +- Emit one function returning an integer. +- Run LLVM verifier. +- Compile to object. +- Link and call from a tiny C or Jerboa harness. + +### Milestone 2: Scalar Expressions + +- Lower literals, variables, `let`, `if`. +- Lower arithmetic with overflow checks. +- Lower comparisons. +- Add status-code error returns. + +### Milestone 3: Records + +- Lower immutable records. +- Support construction and field access. +- Pass records to functions. +- Return records. + +### Milestone 4: Variants + +- Lower tagged unions. +- Lower pattern matching. +- Add exhaustive-match assumptions from typed checker. + +### Milestone 5: Runtime Allocation + +- Add runtime allocator. +- Add string and bytes representation. +- Add vector representation. +- Add retain/release or temporary allocation strategy. + +### Milestone 6: Dynamic Boundary + +- Generate C ABI wrappers. +- Convert dynamic Jerboa values. +- Return structured errors. +- Add Jerboa wrapper `.sls`. + +### Milestone 7: Backend Parity + +- Run same typed module through Rust and LLVM. +- Differential-test outputs. +- Add CI target for LLVM-supported subset. + +### Milestone 8: Resource and FFI Prototype + +- Add non-null handle type. +- Add checked FFI wrapper. +- Add one safe native call. +- Prove no null pointer can reach generated native call. + +## Risks + +- LLVM UB can silently miscompile code. +- Runtime memory model can become too complex. +- Debugging generated IR can be slow. +- ABI design mistakes are painful to change. +- Premature optimization can obscure correctness bugs. + +Mitigations: + +- Start with a tiny safe subset. +- Verify IR every build. +- Avoid aggressive LLVM attributes early. +- Differential-test against Rust backend. +- Keep runtime helpers small and audited. +- Require source spans on generated operations. + +## Success Criteria + +The LLVM backend is successful when: + +- It compiles a safe subset of Typed Jerboa to verified LLVM IR. +- The resulting object can be linked and called from Jerboa. +- Runtime errors are reported as Scheme conditions. +- It matches Rust backend behavior on supported modules. +- It does not require changing Typed Jerboa source code. +- The compiler enforces safety before LLVM lowering. + +The LLVM backend is production-ready when: + +- Memory management is sound. +- Bounds checks and overflow checks are correct. +- Resource ownership is enforced. +- FFI nullability and thread affinity are enforced. +- Debug info points back to Typed Jerboa source. +- CI runs backend parity tests. +- Static binary builds can include LLVM-generated modules. new file mode 100644 --- /dev/null +++ b/docs/jerboa-to-rust.md @@ -0,0 +1,612 @@ +# Jerboa to Rust Backend Plan + +## Purpose + +The Rust backend is the first native backend for Typed Jerboa. + +Rust is not the desired surface language. Typed Jerboa is. Rust is the first +backend because it gives the project a mature compiler, strong memory safety, +good diagnostics, good cross-compilation, and a practical way to validate typed +Jerboa semantics before taking on a direct LLVM backend. + +The Rust backend should compile typed Jerboa modules into Rust crates, compile +those crates into native artifacts, and generate Jerboa wrappers so dynamic +Jerboa code can call typed compiled code safely. + +## Goals + +- Compile typed Jerboa modules to safe Rust where possible. +- Use Rust's type checker as a second line of defense. +- Generate readable Rust for debugging. +- Generate stable FFI exports. +- Generate dynamic Jerboa wrappers. +- Preserve Jerboa source locations in diagnostics where possible. +- Keep Rust-specific concepts out of the Typed Jerboa surface language. +- Make generated artifacts deterministic. + +## Non-Goals + +- Do not translate arbitrary dynamic Jerboa to Rust. +- Do not expose Rust lifetimes directly in Typed Jerboa syntax. +- Do not require users to edit generated Rust. +- Do not use `unsafe` except in small, generated runtime and FFI shims. +- Do not make Rust crates the canonical source of typed modules. + +## Pipeline + +```text +typed Jerboa source + -> macro expansion + -> typed AST + -> typed core IR + -> effect and ownership checks + -> lowered backend-neutral IR + -> Rust AST / Rust text + -> cargo or rustc + -> static library, dynamic library, or object file + -> generated Jerboa wrappers +``` + +The Rust backend should consume a lowered IR, not surface syntax. That keeps +the language independent from Rust and makes the later LLVM backend practical. + +## Output Layout + +Candidate layout: + +```text +build/typed/rust/Cargo.toml +build/typed/rust/src/lib.rs +build/typed/rust/src/modules/<module>.rs +build/typed/rust/src/runtime.rs +build/typed/rust/include/<module>.h +build/typed/rust/target/... +lib/jerboa-emacs/typed/<module>.sls +``` + +The generated Rust crate should be disposable. The source of truth remains the +typed Jerboa source. + +## Artifact Modes + +The backend should eventually support three artifact modes: + +### Dynamic Library + +Useful during development: + +- Faster incremental builds +- Easy `LD_PRELOAD` or `load-shared-object` +- Clear separation between Jerboa and compiled code + +### Static Library + +Useful for static binary builds: + +- Link into `jemacs-qt` +- No runtime `.so` dependency +- Better deployment story + +### Object File + +Useful for advanced build integration: + +- Direct linker control +- Works well with an eventual whole-program static build + +MVP should start with a dynamic library, then add static library mode. + +## Rust Crate Shape + +Generated crate: + +```rust +pub mod runtime; +pub mod modules; + +#[no_mangle] +pub extern "C" fn jerboa_typed_init() -> i32 { + 0 +} +``` + +Each typed module gets a Rust module: + +```rust +pub mod jerboa_emacs_typed_split_tree; +pub mod jerboa_emacs_typed_rope; +``` + +Exports are exposed through C ABI functions: + +```rust +#[no_mangle] +pub extern "C" fn jt_rope_length(rope: JtHandle) -> JtResultUsize { + ... +} +``` + +The C ABI layer should be thin. Internal generated code should use normal safe +Rust types. + +## Type Mapping + +Initial type mapping: + +| Typed Jerboa | Rust | +| --- | --- | +| `Unit` | `()` | +| `Bool` | `bool` | +| `Char` | `char` | +| `Int` | `i64` | +| `Nat` | `u64` or checked `usize` depending on context | +| `Fixnum` | `isize` | +| `Float` | `f64` | +| `String` | `String` | +| `Bytes` | `Vec<u8>` | +| `Symbol` | runtime interned symbol | +| `(Option T)` | `Option<T>` | +| `(Result T E)` | `Result<T, E>` | +| `(Vector T)` | `Vec<T>` | +| `(List T)` | `Vec<T>` in MVP, persistent list later | +| Record | `struct` | +| Variant | `enum` | + +Boundary-facing values should use ABI-safe wrappers instead of raw Rust layout. + +## Records + +Typed Jerboa: + +```scheme +(record Position + ([line : Nat] + [column : Nat])) +``` + +Rust: + +```rust +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Position { + pub line: u64, + pub column: u64, +} +``` + +Mutable fields should generate methods instead of public mutable fields when +possible: + +```rust +impl Cursor { + pub fn set_point(&mut self, point: u64) { + self.point = point; + } +} +``` + +## Variants + +Typed Jerboa: + +```scheme +(variant EditOp + (Insert [at : Nat] [text : String]) + (Delete [start : Nat] [end : Nat])) +``` + +Rust: + +```rust +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum EditOp { + Insert { at: u64, text: String }, + Delete { start: u64, end: u64 }, +} +``` + +Pattern matching should lower directly to Rust `match` after the Typed Jerboa +checker has already verified exhaustiveness. + +## Functions + +Typed Jerboa: + +```scheme +(def (line-count [text : String]) : Nat + ...) +``` + +Rust: + +```rust +pub fn line_count(text: String) -> u64 { + ... +} +``` + +Internal Rust functions should use safe Rust signatures. Exported functions +should get separate ABI wrappers. + +## Generics + +The first Rust backend should monomorphize generics. + +Typed Jerboa: + +```scheme +(def (option-default [x : (Option A)] [fallback : A]) : A + ...) +``` + +For each concrete use, generate a concrete Rust function: + +```rust +fn option_default_string(x: Option<String>, fallback: String) -> String { ... } +fn option_default_u64(x: Option<u64>, fallback: u64) -> u64 { ... } +``` + +This avoids exposing generic ABI problems at the boundary. + +## Effects + +Typed Jerboa effects should influence generated Rust: + +- `pure`: ordinary safe function +- `alloc`: may allocate +- `mut`: takes `&mut` or owned updated values +- `io`: returns `Result` +- `ffi`: calls generated FFI shim