docs/data: document expanded LLVM backend (String/Bytes/records/whole-program)
ober
fa627137abae88dd5dde3c1bf50b0dd4175f341f
--- a/data/changelog.sexp +++ b/data/changelog.sexp @@ -2,7 +2,20 @@ . "Machine-readable changelog of Jerboa API drift. Consumers (LLM tooling, lints, jerboa_verify) use this to invalidate stale recommendations and to suggest migrations when a symbol is renamed or relocated.") ("entries" - (("added" "typed-module->llvmir-string" "typed-library-form->llvmir-string" + (("added" "typed-modules->llvmir-string" "typed-library-forms->llvmir-string") + ("date" . "2026-06-03") + ("modules_added") + ("moved") + ("notes" + . + "LLVM IR backend expanded past the scalar MVP to cover the pure compute subset Typed Jerboa kernels actually use: String/Bytes as a by-value { ptr, i64 } fat pointer (string->utf8/utf8->string are value identities; string literals intern deduped private constants; bytevector-length/u8-ref via extractvalue/gep/load; bytes-build via malloc + store loop), by-value records ({ f0ty, ... } structs, insertvalue ctor / extractvalue accessor, usable as for/fold accumulators via a struct phi), and whole-program cross-module emission (topo-sorted, registry-checked, global name->symbol map so imported calls target @jt_llvm_<defining-module>__<def>; duplicate names across modules rejected). CLI gains --whole-program OUT.ll. Proven end-to-end on jerboa-secmon: its six pure detection kernels (analytics/strbytes/triage/lolbin/obfuscate/dga) compile whole-program to one native object (68 functions, verify + -O2) and a C harness asserts every headline kernel against the Rust backend's values. The crypto trio (sha256/hmac/hkdf/aes-gcm/x25519) is intentionally left on the RustCrypto FFI path. See docs/llvmir-backend.md.") + ("removed") + ("renamed") + ("tier_changes") + ("tools_added" "make typed-llvmir-parity" + "support/typed-llvmir.ss --whole-program") + ("version" . #f)) + (("added" "typed-module->llvmir-string" "typed-library-form->llvmir-string" "typed-library-forms->llvmir-module" "llvm-symbol-name" "llvm-module-mangle" "llvm-function-symbol" "llvm-float-literal") ("date" . "2026-06-03") --- a/data/cookbooks.sexp +++ b/data/cookbooks.sexp @@ -4932,4 +4932,26 @@ "chez") ("title" . - "Plain let inits evaluate right-to-left in Chez — use let* in stateful code"))) + "Plain let inits evaluate right-to-left in Chez — use let* in stateful code")) + (("code" + . + "(import (jerboa prelude))\n\n(def (eval-to-client expr ip op)\n ;; IP and OP are the TCP client input/output ports. Bind all standard\n ;; current ports while evaluating so user code that calls display,\n ;; write, read, display-condition, or current-error-port talks to the\n ;; REPL client instead of the host process terminal.\n (parameterize ((current-input-port ip)\n (current-output-port op)\n (current-error-port op))\n (call-with-values\n (lambda () (eval expr (interaction-environment)))\n (lambda results\n (for-each\n (lambda (v)\n (unless (eq? v (void))\n (write v op)\n (newline op)))\n results)\n (flush-output-port op)))))") ("id" . "repl-eval-bind-current-ports") + ("imports" "(jerboa prelude)") + ("notes" + . + "Without the parameterize wrapper, evaluated expressions that call display or display-condition with no explicit port use the process defaults. In a TUI process that owns the terminal through termbox, those writes can bleed into the UI and cannot be fixed by a redraw because they bypass the TUI renderer.") + ("tags" "repl" "eval" "current-output-port" + "current-error-port" "tcp" "debug-repl") + ("title" . "Bind current ports around eval in a TCP REPL")) + (("code" + . + ";; Lower several typed modules (with imports) into ONE textual LLVM module so\n;; cross-module calls resolve. String/Bytes are { ptr, i64 } fat pointers;\n;; records are by-value structs; bytes-build mallocs a fresh buffer.\n(import (jerboa typed parser) (jerboa typed llvmir))\n\n(def lib\n '(typed-library (demo strbytes)\n (export first-byte)\n (def (first-byte (s : String)) : Nat\n (bytevector-u8-ref (string->utf8 s) 0))))\n\n(def app\n '(typed-library (demo app)\n (export starts-h?)\n (import (demo strbytes))\n (def (starts-h? (s : String)) : Bool\n (= (first-byte s) 104)))) ; 104 = 'h'\n\n;; topo-sorts by imports, emits @jt_llvm_demo_strbytes__first_byte once and\n;; the app's call targets that exact symbol\n(display (typed-library-forms->llvmir-string (list app lib)))\n\n;; Shell: scheme --libdirs lib --script support/typed-llvmir.ss \\\n;; --whole-program out.ll a.ss b.ss\n;; then: llvm-as out.ll | opt -passes=verify | opt -O2 | llc | cc harness.c obj") ("id" . "typed-llvmir-whole-program-strings-records") + ("imports" "(jerboa typed parser)" "(jerboa typed llvmir)") + ("notes" + . + "Buffer ABI: String/Bytes = by-value { ptr, i64 } (data,len), matching a C `struct { const char*; uint64_t; }` on SysV/AAPCS, so a C harness can call kernels directly. string->utf8/utf8->string are value identities (same rep). string literals intern one deduped private constant. records lower to structural { f0ty, ... } structs (insertvalue ctor, extractvalue accessor) and can be for/fold accumulators (phi of struct). bytes-build = malloc + store loop, never freed (short-lived kernels; GC is future work). Duplicate function names across modules are rejected (flat LLVM symbols). NOT supported: variants/match/Option/Result, and crypto prims (sha256/aes-gcm/x25519 stay on RustCrypto FFI — never reimplement crypto in LLVM). Real example: jerboa-secmon `make llvmir-bin` builds 6 pure kernels to a native binary asserted against the Rust backend. See docs/llvmir-backend.md.") + ("tags" "typed" "llvmir" "llvm" "whole-program" "string" + "bytes" "record") + ("title" + . + "Whole-program Typed Jerboa -> LLVM IR with String/Bytes, records, cross-module calls"))) --- a/docs/llvmir-backend.md +++ b/docs/llvmir-backend.md @@ -59,32 +59,43 @@ fail with a clear message. Exactly this; everything else is rejected at lowering with a `typed-llvmir` error: -- Types: `Nat`, `Int`, `Bool`, `Float`; `Unit` as a return type only. -- Numeric and boolean literals. +- Types: `Nat`, `Int`, `Bool`, `Float`, `String`, `Bytes`, and by-value + `record`s; `Unit` as a return type only. +- Numeric, boolean, and string/bytes literals. - `let` (sequential bindings, no stack slots), `begin`, `if`. -- Direct same-module function calls. +- Same-module **and cross-module** (imported) function calls, via + whole-program emission. - Arithmetic `+ - * /`, comparisons `= < <= > >=`, scalar `equal?`. - Boolean `not`/`and`/`or` (non-short-circuit `and`/`or` over already evaluated `i1` values). - Bitwise `bitwise-and/ior/xor/not`, shifts `bitwise-arithmetic-shift-left/right`. - `exact->inexact`, `log2`. -- `(for/fold ((acc init)) ((i (in-range [start] end))) body)`. +- `string->utf8`, `utf8->string`, `string-length`, `bytevector-length`, + `bytevector-u8-ref`, and `bytes-build`. +- `record` declarations: `make-R` constructors, `R-field` accessors, + `R?` predicates; records usable as `for/fold` accumulators. +- `(for/fold ((acc init)) ((i (in-range [start] end))) body)` with a scalar, + `Float`, or record accumulator. - A generated `@main` for smoke tests (below). -Not supported (deliberately, this pass): strings, bytes, records, variants, -Option/Result, resources, match, closures, GC, cross-module imports, the Chez -FFI boundary, and LLVM C API bindings. Modules containing record/variant/ -resource declarations are rejected whole. +Not supported (this pass): variants, `match`, `Option`/`Result`, resources, +closures, GC, the Chez FFI boundary, crypto primitives (`sha256`, +`hmac-sha256`, `hkdf-sha256`, `aes-256-gcm-*`, `x25519-*` — these stay on the +RustCrypto FFI path; reimplementing them in LLVM is out of scope and a security +anti-pattern), and LLVM C API bindings. Modules using those forms are rejected. ## Representation ```text -Bool -> i1 -Nat -> i64 unsigned ops: udiv, icmp ult/ule/ugt/uge, lshr -Int -> i64 signed ops: sdiv, icmp slt/sle/sgt/sge, ashr -Float -> double fadd/fsub/fmul/fdiv, ordered fcmp (oeq/olt/...) -Unit -> void (procedure returns only) +Bool -> i1 +Nat -> i64 unsigned ops: udiv, icmp ult/ule/ugt/uge, lshr +Int -> i64 signed ops: sdiv, icmp slt/sle/sgt/sge, ashr +Float -> double fadd/fsub/fmul/fdiv, ordered fcmp (oeq/olt/...) +String -> { ptr, i64 } by-value fat pointer (data, length); UTF-8 bytes +Bytes -> { ptr, i64 } identical rep, so string->utf8/utf8->string are no-ops +record -> { f0ty, f1ty, ... } by-value struct; ctor=insertvalue, field=extractvalue +Unit -> void (procedure returns only) ``` Signedness lives in operations, not types. Mixed int/float operands in one @@ -95,6 +106,22 @@ miscompilation bugs. Float literals are emitted as IEEE-754 bit patterns (`0x4011000000000000` for `4.25`), the always-valid LLVM spelling. `log2` declares and calls the `llvm.log2.f64` intrinsic. +String/Bytes literals intern one private `unnamed_addr constant` per distinct +content (deduped), built into the fat pointer with `insertvalue`. `bytes-build` +emits `malloc` + a store loop and returns a fresh buffer; **buffers are never +freed** — analysis kernels are short-lived, and ownership/GC is future work. + +## Whole-Program Emission + +`typed-modules->llvmir-string` (CLI: `--whole-program OUT.ll file.ss ...`) +lowers several modules into ONE textual LLVM module so an imported function in +another module is callable. Modules are topologically sorted by their imports +and batch-checked with the registry; a global name→signature map carries each +callee's defining-module symbol, so a cross-module call emits +`@jt_llvm_<defining-module>__<def>`. Duplicate function names across modules are +rejected (flat LLVM symbols cannot disambiguate them). Literal globals and +intrinsic declares are emitted once for the whole program. + ## Naming ```text @@ -158,12 +185,38 @@ the LLVM executable's exit status must equal the exit status of a rustc-compiled harness calling the generated Rust (`lib.rs` built as an rlib directly with `rustc`, no cargo), and both must equal the expected constant. +## Worked Example: secmon Detection Kernels + +`~/mine/jerboa-secmon` compiles its six **pure** typed detection kernels — +`analytics`, `strbytes`, `triage`, `lolbin`, `obfuscate`, `dga` — straight to +one native object with no Rust crate in the path: + +```bash +cd ~/mine/jerboa-secmon +make llvmir-bin # whole-program .ll -> verify -> -O2 -> llc -> cc harness -> run +``` + +68 functions verify and survive `-O2`; a C harness (`llvmir/harness.c`) calls +the headline kernels through the `{ ptr, i64 }` buffer ABI and asserts each +result against the Rust backend's value (host-risk scoring, DGA entropy/ +consonant-run/score, LOLBin command scoring, triage predicates, obfuscation +round-trip). This exercises every expanded feature: String/Bytes, records +(`dga`'s `CrState` fold), `Float`/`log2` entropy, `bytes-build`, and +cross-module calls (`lolbin`/`triage`/`dga` import `strbytes`). + +The crypto trio (`crypto`, `ecies`, `psk`) is **excluded by design**: its +`sha256`/`hmac`/`hkdf`/`aes-256-gcm`/`x25519` kernels are thin wrappers over +vetted RustCrypto crates, which the typed sources themselves say must never be +reimplemented. They stay on the Rust backend; the LLVM path is for the pure +compute kernels. + ## Files ```text lib/jerboa/typed/llvmir.ss emitter library (jerboa typed llvmir) -support/typed-llvmir.ss CLI: typed .ss -> OUT-DIR/<module>.ll +support/typed-llvmir.ss CLI: per-module, or --whole-program OUT.ll tests/test-typed-llvmir.ss deterministic emitter unit tests -tests/fixtures/typed/llvmir-{basic,if,call,float,bitwise,for-fold,smoke}.ss +tests/fixtures/typed/llvmir-{basic,if,call,float,bitwise,for-fold,bytes, + record,smoke}.ss build/typed/llvmir/ disposable generated artifacts (.ll/.bc/.o) ```