Add LLVM IR backend handoff
ober
a9229b943537987d04fdd1c78a8cc71114233ba9
--- a/data/cookbooks.sexp +++ b/data/cookbooks.sexp @@ -4897,4 +4897,16 @@ "security") ("title" . - "Persist records portably by decomposing to plain data (avoid fasl pitfalls)"))) + "Persist records portably by decomposing to plain data (avoid fasl pitfalls)")) + (("code" + . + "jerbuild exec --libdirs ./lib:vendor/jerboa-sqlite/lib:vendor/jerboa-websearch/src:$(jerbuild --jerboa-home)/lib /dev/stdin <<'EOF'\n(import (jerboa prelude) (jcode ui tui-theme))\n(displayln (get-registered-themes))\n(displayln (set-theme-by-name! \"opencode-dark\"))\n(displayln (current-theme-name))\nEOF") ("id" . "jerbuild-exec-local-module-smoke-test") + ("imports") + ("notes" + . + "Use this after `jerboa_make build` or `make build` has transpiled/compiled the repo into `lib/`. Plain `jerboa /dev/stdin` may not know project libdirs, and the colon reader form can fail in stdin scripts; use canonical module imports like `(jcode ui tui-theme)`.") + ("tags" "jerbuild" "exec" "libdirs" "local-module" + "smoke-test" "stdin") + ("title" + . + "Smoke test a local Jerboa module with jerbuild exec and libdirs"))) --- a/data/features.sexp +++ b/data/features.sexp @@ -1159,4 +1159,21 @@ ("use_case" . "Iteratively compile-checking and running many files in one project across a long session.") + ("votes" . 0)) + (("description" + . + "jerboa_verify and jerboa_compile_check can fail inside the MCP tool with an exception like `Exception in string-ref: <n> is not a valid index for \"<entire file>\"` while checking larger project files. The failure prevents normal syntax/compile diagnostics and forces fallback to `jerboa_make`. The tool should report a structured internal-tool error with filename and offending offset at minimum, and ideally complete the verification normally.") + ("estimated_token_reduction" + . + "~1000-3000 tokens per affected verification because the fallback dumps large file contents and requires extra build/debug steps.") + ("example_scenario" + . + "After editing src/jcode/mcp/client.ss, src/jcode/core/agent.ss, and src/jcode/ui/serve.ss, both jerboa_verify and jerboa_compile_check crashed with string-ref invalid-index exceptions instead of returning syntax or compile diagnostics.") + ("id" . "fix-verify-compile-check-string-index-crash") + ("impact" . "medium") + ("tags" "verify" "compile-check" "string-ref" "tooling") + ("title" . "Fix verifier string-ref crash on larger files") + ("use_case" + . + "Use when validating changed Jerboa source files before build, especially larger modules such as src/jcode/core/agent.ss or src/jcode/ui/serve.ss.") ("votes" . 0))) new file mode 100644 --- /dev/null +++ b/docs/llvmir-handoff.md @@ -0,0 +1,702 @@ +# Typed Jerboa to LLVM IR Backend Handoff + +Audience: Opus 4.8 or another implementation agent. + +Branch: `llvmir`. + +Repository: `/Users/user/mine/jerboa`. + +## Mission + +Add an experimental Typed Jerboa backend that lowers the existing typed core IR +to textual LLVM IR. + +The first backend does not replace the Rust backend. Rust remains the reference +backend and correctness oracle. The LLVM backend should start as a narrow, +inspectable, verifier-driven path that proves Typed Jerboa can lower directly +to LLVM without going through generated Rust. + +The target pipeline is: + +```text +typed Jerboa source + -> typed parser/checker + -> typed core IR + -> textual LLVM IR (.ll) + -> llvm-as / opt -verify + -> opt -O2 + -> llc or clang + -> executable or object file +``` + +## Hard Constraints + +- Only edit files in this repo unless the user explicitly names another path. +- All user-facing Jerboa source files are `.ss`; do not create `.sls` files. +- Follow the existing Typed Jerboa architecture. Do not lower from raw source + datums when typed core IR already exists. +- Before writing Jerboa code, use the Jerboa MCP workflow: + `jerboa_howto`, API/export/signature checks, then `jerboa_verify`. +- After changing `.ss` files, run the appropriate Jerboa build/check target and + fix failures before stopping. +- Before committing on macOS, run `make binary`, not `make docker-build`. +- Keep generated LLVM IR deterministic and textual at first. Do not start with + LLVM C API bindings. + +## Current State To Reuse + +Existing typed files: + +```text +lib/jerboa/typed/parser.ss +lib/jerboa/typed/checker.ss +lib/jerboa/typed/core.ss +lib/jerboa/typed/rust.ss +lib/jerboa/typed/wrapper.ss +support/typecheck.ss +support/typed-rust.ss +support/typed-wrappers.ss +``` + +Existing tests and fixtures: + +```text +tests/test-typed-core.ss +tests/test-typed-checker.ss +tests/test-typed-rust.ss +tests/test-typed-wrapper-e2e.ss +tests/fixtures/typed/rust-basic.ss +tests/fixtures/typed/rust-bitwise.ss +tests/fixtures/typed/rust-float.ss +tests/fixtures/typed/rust-for-fold.ss +tests/fixtures/typed/rust-bytes-build.ss +``` + +Important docs: + +```text +docs/typed-jerboa.md +docs/jerboa-to-rust.md +docs/rust-target.md +docs/optimization.md +``` + +The typed core IR already has records such as: + +```text +typed-ir-lit +typed-ir-var +typed-ir-begin +typed-ir-let +typed-ir-if +typed-ir-match +typed-ir-call +``` + +The checker exposes `check-and-elaborate-typed-module` and ordered +`elaborated-def` records. The Rust backend has already adopted this IR path. +The LLVM backend should consume the same elaborated module/def data. + +## Non-Goals For The First Pass + +Do not implement these in the first working slice: + +- Arbitrary dynamic Jerboa. +- Macros beyond what the typed pipeline already expands/parses. +- Strings, bytes, records, variants, Option, Result, resources, handles, or + wrappers across the Chez FFI boundary. +- Garbage collection. +- Closures. +- Generic optimization passes in Jerboa. +- LLVM C API bindings. +- Full runtime interop. + +These can come later after scalar functions compile and verify. + +## MVP Success Criteria + +The first useful backend should compile a typed module containing scalar +functions to `.ll`, verify it with LLVM, and build a runnable executable for at +least one smoke fixture. + +Required MVP language subset: + +```text +Nat / Int-like 64-bit integers +Bool +Float if easy after integers +numeric literals +boolean literals +let +begin +if +direct same-module function calls +primitive arithmetic +primitive comparisons +one generated main or exported entry wrapper for smoke tests +``` + +MVP verification: + +```text +make typed-llvmir +make typed-llvmir-check +make typed-llvmir-smoke +``` + +Those targets do not exist yet; add them once the backend skeleton exists. + +## Suggested Files To Add + +Add: + +```text +lib/jerboa/typed/llvmir.ss +support/typed-llvmir.ss +tests/test-typed-llvmir.ss +tests/fixtures/typed/llvmir-basic.ss +tests/fixtures/typed/llvmir-if.ss +tests/fixtures/typed/llvmir-call.ss +``` + +Possibly add later: + +```text +tests/fixtures/typed/llvmir-float.ss +tests/fixtures/typed/llvmir-for-fold.ss +tests/fixtures/typed/llvmir-bitwise.ss +``` + +Modify: + +```text +Makefile +docs/typed-jerboa.md +docs/jerboa-to-rust.md +docs/index.md +``` + +Only update existing docs once a working slice lands. This handoff is the design +starting point, not a claim that LLVM support already exists. + +## Output Layout + +Use a disposable generated tree, parallel to the Rust backend: + +```text +build/typed/llvmir/ + <module>.ll + <module>.bc + <module>.o + <module> +``` + +Keep the typed source as the source of truth. Generated `.ll`, `.bc`, `.o`, and +executables are build artifacts. + +## LLVM Representation + +LLVM integer types do not encode signedness. Signedness is chosen by operations. + +Initial representation: + +```text +Typed Bool -> i1 internally +Typed Nat -> i64, use unsigned comparisons/division +Typed Int -> i64, use signed comparisons/division if Int exists +Typed Float -> double +Unit -> void for procedures, i8 0 if expression position needs a value +``` + +Defer: + +```text +Char -> i32 code point +String -> runtime-owned ptr+len or opaque handle +Bytes -> runtime-owned ptr+len or opaque handle +Record -> LLVM struct for by-value records, pointer/handle for recursive data +Variant -> tagged struct or pointer/handle +``` + +For the MVP, prefer whole-program generated executables over Chez FFI wrappers. +Crossing dynamic Jerboa boundaries can wait. + +## Naming And Mangling + +Generated LLVM identifiers must be deterministic and valid. + +Suggested function symbol shape: + +```text +@jt_llvm_<module_mangle>__<def_mangle> +``` + +Suggested local SSA names: + +```text +%v0 +%v1 +%tmp3 +``` + +Suggested block labels: + +```text +entry +then0 +else0 +join0 +``` + +Do not trust user identifiers as raw LLVM identifiers. Implement one small +mangler shared by function names, local names, and module names. + +## Emitter Shape + +Keep the emitter structured. Do not build nested expressions by ad hoc string +concatenation. + +Suggested internal records: + +```scheme +llvm-type ;; textual LLVM type plus typed-kind metadata if useful +llvm-value ;; type + operand text, e.g. i64 / "%v3" +llvm-instr ;; rendered instruction line +llvm-block ;; label + instructions + terminator +llvm-function ;; name + params + return type + blocks +llvm-module ;; target-ish metadata + function declarations/definitions +llvm-env ;; local variable map, function signature map, counters +``` + +The core lowering function should return both emitted instructions and the +result value: + +```text +lower-expr(env, expr) -> env + llvm-value +``` + +In practice this may be threaded as mutable counters plus accumulated blocks. +Choose the style that matches `lib/jerboa/typed/rust.ss`, but preserve the +separation between values, instructions, blocks, and final rendering. + +## Expression Lowering Details + +### Literals + +Lower numeric and boolean literals directly: + +```text +42 -> i64 42 +#t -> i1 true +#f -> i1 false +4.25 -> double 4.250000e+00 or a stable decimal form LLVM accepts +``` + +Use a stable float formatter. If Jerboa's formatter produces syntax LLVM does +not accept, add a small LLVM float literal formatter and test it. + +### Variables + +Local variables should map to SSA operands in the environment. For the MVP, +lower `let` by evaluating the initializer and binding the name to the produced +SSA value. Avoid `alloca` unless mutation requires it later. + +### `begin` + +Lower each expression in order and return the last value. If the typed IR has +effect-free scalar expressions only, this is straightforward. + +### `let` + +For immutable `let`, no stack slot is needed: + +```text +let x = expr +body +``` + +becomes: + +```text +%v0 = ... +; bind x -> %v0 +... +``` + +If a later expression form introduces mutation, add `alloca`/`load`/`store` +only for mutable locals. + +### `if` + +Use real basic blocks and a `phi` for expression-valued conditionals. + +Shape: + +```llvm + br i1 %cond, label %then0, label %else0 + +then0: + %v1 = ... + br label %join0 + +else0: + %v2 = ... + br label %join0 + +join0: + %v3 = phi i64 [ %v1, %then0 ], [ %v2, %else0 ] +``` + +If the branch type is `void`, do not emit a `phi`. + +### Direct Calls + +Resolve function signatures before lowering bodies. The Rust backend already +has call-kind and info on `typed-ir-call`; reuse that instead of repeating +surface name lookup. + +Call shape: + +```llvm +%v4 = call i64 @jt_llvm_mod__add(i64 %a, i64 %b) +``` + +For `void` return: + +```llvm +call void @jt_llvm_mod__side_effect(...) +``` + +### Arithmetic + +Map by operand/result type: + +```text +i64 Nat/Int: + + -> add + - -> sub + * -> mul + Nat / -> udiv + Int / -> sdiv + +double: + + -> fadd + - -> fsub + * -> fmul + / -> fdiv +``` + +Consider adding `nsw`/`nuw` only after Typed Jerboa overflow semantics are +explicit. Incorrect overflow flags are miscompilation bugs. + +### Comparisons + +Map comparisons carefully: + +```text +Nat: + < -> icmp ult + <= -> icmp ule + > -> icmp ugt + >= -> icmp uge + = -> icmp eq + +Int: + < -> icmp slt + <= -> icmp sle + > -> icmp sgt + >= -> icmp sge + = -> icmp eq + +Float: + < -> fcmp olt + <= -> fcmp ole + > -> fcmp ogt + >= -> fcmp oge + = -> fcmp oeq +``` + +Use ordered float comparisons unless Typed Jerboa explicitly wants NaN-aware +unordered behavior. + +### Boolean Operators + +If the typed core represents `and`/`or` as calls, preserve short-circuit +semantics if the source form promises it. For the MVP, support only primitive +boolean operations whose semantics are already explicit in the IR. + +Simple boolean values: + +```text +not -> xor i1 %x, true +and -> and i1 %a, %b ; only when both sides are already evaluated +or -> or i1 %a, %b +``` + +### Bitwise + +Add after scalar arithmetic: + +```text +bitwise-and -> and i64 +bitwise-ior -> or i64 +bitwise-xor -> xor i64 +bitwise-not -> xor i64 %x, -1 or not-equivalent via xor +shift-left -> shl +Nat shift-right -> lshr +Int shift-right -> ashr +``` + +Mask or validate shift counts only if Typed Jerboa semantics require it. + +## Generated Main For Smoke Tests + +For early smoke tests, add a small convention rather than full wrappers: + +- If a typed fixture exports or defines `main`, emit an LLVM `@main`. +- `@main` should call the mangled typed function and return an `i32`. +- If the typed function returns `Bool`, return `0` for true and `1` for false + or choose one convention and document it in tests. +- If it returns `Nat`/`Int`, truncate to `i32` for process status only for tiny + fixture values. + +Example: + +```llvm +define i32 @main() { +entry: + %v0 = call i64 @jt_llvm_demo__main_value() + %v1 = trunc i64 %v0 to i32 + ret i32 %v1 +} +``` + +This is only for smoke testing. Real runtime/FFI integration comes later. + +## CLI Design + +Add `support/typed-llvmir.ss` parallel to `support/typed-rust.ss`. + +Suggested behavior: + +```bash +scheme --libdirs lib --script support/typed-llvmir.ss tests/fixtures/typed/llvmir-basic.ss +``` + +Outputs: + +```text +build/typed/llvmir/<module>.ll +``` + +Useful options, if the existing CLI pattern supports them: + +```text +--out-dir build/typed/llvmir +--emit-main +--verify +--build +``` + +Do not overbuild the CLI first. A minimal command that emits `.ll` for a fixture +is enough to start tests. + +## Makefile Targets + +Add targets in small steps: + +```make +typed-llvmir: + # generate .ll for LLVM fixtures + +typed-llvmir-check: typed-llvmir + # run llvm-as and opt -verify when tools exist + +typed-llvmir-smoke: typed-llvmir-check + # build and run a tiny executable +``` + +Tool discovery should tolerate Homebrew LLVM names on macOS if practical: + +```bash +llvm-as +opt +llc +clang +``` + +If LLVM tools are missing, unit tests should still validate deterministic `.ll` +text generation, but smoke targets may fail with a clear message. + +## Testing Strategy + +Start with tests that do not require LLVM installed: + +- Parse/check a tiny typed fixture. +- Emit `.ll`. +- Assert key lines are present. +- Assert output is deterministic across two emissions. + +Then add verifier tests gated by tool availability: + +```bash +llvm-as file.ll -o file.bc +opt -verify file.bc -o /dev/null +opt -O2 file.bc -o file.opt.bc +``` + +Then add executable smoke: + +```bash +llc -filetype=obj file.opt.bc -o file.o +clang file.o -o file +./file +``` + +Behavior parity: + +- Prefer comparing a fixture's result against the Rust backend where possible. +- For early process-exit smoke tests, keep result values small and exact. +- Later, add stdout printing or a C ABI test harness so results are not limited + to process exit status. + +## First Implementation Plan + +### Phase 1: Skeleton + +1. Use MCP cookbook/API checks for existing typed core and Rust emitter patterns. +2. Add `lib/jerboa/typed/llvmir.ss`. +3. Export one high-level function, likely: + + ```scheme + typed-module->llvmir-string + typed-library-form->llvmir-string + typed-library-forms->llvmir-module + ``` + + Match existing Rust backend naming where possible. + +4. Add a minimal CLI in `support/typed-llvmir.ss`. +5. Add one fixture and one unit test that proves the CLI writes deterministic + textual `.ll`. + +### Phase 2: Scalar Functions + +1. Lower typed function signatures. +2. Lower literals and direct returns. +3. Lower primitive arithmetic. +4. Lower `let`. +5. Lower `if` with blocks and `phi`. +6. Lower direct same-module calls. +7. Add `llvmir-basic`, `llvmir-if`, and `llvmir-call` fixtures. + +### Phase 3: LLVM Verification + +1. Add Makefile target to generate `.ll`. +2. Add optional verifier target using `llvm-as` and `opt -verify`. +3. Fix all malformed IR before adding features. +4. Keep verifier failures readable: print the generated `.ll` path. + +### Phase 4: Smoke Executable + +1. Emit `@main` for a fixture convention. +2. Build with `llc` and `clang`, or directly with `clang file.ll -o exe` if that + is reliable enough on the local LLVM. +3. Run the executable and check exit status. + +### Phase 5: Parity With Rust Backend + +1. Reuse simple Rust fixtures where they only contain scalar functionality. +2. Generate Rust and LLVM outputs from equivalent typed sources. +3. Compare observed result, not generated source. + +### Phase 6: Expand Scalar Coverage + +Add in this order: + +```text +Float +bitwise +for/fold over in-range +bytes-build only if Bytes representation is settled +``` + +Do not add records/variants before the scalar control-flow backend is boring. + +## Common LLVM IR Pitfalls + +- Every basic block must end with exactly one terminator. +- A `phi` node's predecessor labels must match the blocks that branch to the + current block. +- Do not emit instructions after a terminator in the same block. +- Integer signedness lives in operations, not types. +- `i1` is not ABI-friendly everywhere; it is fine internally. +- LLVM textual syntax is picky about string constants and float constants. +- Avoid overflow flags until language semantics prove they are valid. +- Avoid `alloca` for immutable locals; SSA values are simpler and optimize + better. +- Do not let a block-building helper accidentally reuse a label. + +## Later Runtime And ABI Direction + +Once scalar executable tests pass, decide how LLVM artifacts integrate with +dynamic Jerboa. + +Possible paths: + +1. Generate a C ABI similar to the Rust backend's exported wrappers. +2. Link LLVM-generated objects into the final Jerboa binary. +3. Load a shared library and call via Chez `foreign-procedure`. +4. Share the Rust backend's Scheme wrapper generator where ABI shapes match. + +Do not design this fully during the MVP. The first LLVM backend should prove +correct typed core lowering and LLVM verification. + +## Documentation Updates After Working Slice + +When a real slice lands, update: + +```text +docs/typed-jerboa.md +docs/jerboa-to-rust.md +docs/index.md +``` + +Add: + +```text +docs/llvmir-backend.md +``` + +That future doc should describe actual commands and supported language forms. +This handoff is for implementation planning. + +## Definition Of Done For The First PR + +A good first PR on `llvmir` should include: + +- `lib/jerboa/typed/llvmir.ss` +- `support/typed-llvmir.ss` +- at least three scalar fixtures +- deterministic unit tests +- optional LLVM verifier target if tools are installed +- one smoke executable if practical +- docs updated to state exact supported subset +- no changes to sibling repos +- no stale generated artifacts committed +- clean Jerboa verification for changed `.ss` files +- `make binary` run before commit on macOS + +## Recommended First Prompt To Opus + +Use this exact framing: + +```text +You are on branch llvmir in /Users/user/mine/jerboa. Implement the first +experimental Typed Jerboa -> textual LLVM IR backend. Read docs/llvmir-handoff.md, +docs/typed-jerboa.md, docs/jerboa-to-rust.md, then inspect only the existing +typed core/Rust backend files needed for the first slice. Start with scalar +Nat/Bool functions, let, if, and direct calls. Add deterministic tests and a +minimal support/typed-llvmir.ss CLI. Use Jerboa MCP tools before writing Scheme +and verify changed .ss files before stopping. +```