Add project documentation from gherkin, updated for jerboa
ober
80d67feefc635bfa9b081a054e8c38ecc935ea46
new file mode 100644 --- /dev/null +++ b/docs/compiling-gerbil-projects.md @@ -0,0 +1,398 @@ +# Compiling Gerbil Projects with Jerboa + +This guide walks through the complete process of taking an existing Gerbil +Scheme project and compiling it to run on Chez Scheme via Jerboa. + +## Prerequisites + +1. **Chez Scheme 10.x** with threads (`./configure --threads && make && make install`) +2. **Jerboa** built and ready: + ```bash + git clone https://github.com/ober/jerboa ~/mine/jerboa + cd ~/mine/jerboa && make + ``` +3. **Gerbil source tree** (for the modules your project imports): + ```bash + # Only needed if your project imports :std/* modules + git clone https://github.com/mighty-gerbils/gerbil ~/mine/gerbil + ``` + +## Project Layout + +A jerboa project wraps an existing Gerbil project with build scripts and a +compatibility layer: + +``` +my-jerboa-project/ + my-gerbil-project/ # Gerbil source (submodule or copy) + gerbil.pkg + *.ss # Gerbil source files (never modified) + src/ + compat/ # Chez compatibility shims (handwritten) + sugar.sls # :std/sugar equivalents + format.sls # :std/format equivalents + ... + mylib/ # Generated .sls files (gitignored) + build-jerboa.ss # Jerboa translation driver + build-all.ss # Chez compilation trigger + build-binary.ss # Optional: standalone binary builder + main.ss # Entry point program + Makefile +``` + +## Step 1: Create the Makefile + +```makefile +SCHEME = scheme +JERBOA = $(or $(JERBOA_DIR),$(HOME)/mine/jerboa/src) +LIBDIRS = src:$(JERBOA) +COMPILE = $(SCHEME) -q --libdirs $(LIBDIRS) --compile-imported-libraries + +.PHONY: all jerboa compile binary run clean + +all: jerboa compile + +# Translate .ss -> .sls via jerboa +jerboa: + $(COMPILE) < build-jerboa.ss + +# Compile .sls -> .so via Chez +compile: jerboa + $(COMPILE) < build-all.ss + +# Build standalone binary +build: binary +binary: clean jerboa + $(SCHEME) -q --libdirs $(LIBDIRS) --program build-binary.ss + +# Run interpreted +run: all + $(SCHEME) -q --libdirs $(LIBDIRS) --program main.ss + +clean: + find src -name '*.so' -o -name '*.wpo' | xargs rm -f 2>/dev/null || true + rm -f src/mylib/*.sls +``` + +The `JERBOA` variable points to jerboa's `src/` directory. Override it: +```bash +JERBOA_DIR=/path/to/jerboa/src make +``` + +If your project depends on other jerboa-compiled libraries, add them to +`LIBDIRS`: +```makefile +JERBOA_AWS = $(or $(JERBOA_AWS_DIR),jerboa-aws/src) +LIBDIRS = src:$(JERBOA):$(JERBOA_AWS) +``` + +## Step 2: Write the Import Map + +The import map is the most important part. It tells jerboa how to translate +Gerbil's `:module/path` imports into R6RS `(library name)` imports. + +```scheme +(define my-import-map + '(;; Standard library -> compat shims + (:std/sugar . (compat sugar)) + (:std/format . (compat format)) + (:std/sort . (compat sort)) + (:std/pregexp . (compat pregexp)) + (:std/error . (runtime error)) + (:std/misc/string . (compat misc)) + (:std/misc/list . (compat misc)) + (:std/misc/hash . (compat misc)) + + ;; Strip imports that have no Chez equivalent + (:std/iter . #f) ;; jerboa compiles for-loops natively + (:std/foreign . #f) ;; no FFI at this level + (:gerbil/core . #f) ;; runtime provides these + (:gerbil/runtime . #f) + + ;; Project-internal imports (relative) + ("./util" . (mylib util)) + + ;; Cross-project dependencies + (:ober/aws/s3 . (jerboa-aws s3-api)) + )) +``` + +Three mapping types: +- **`(library name)`**: Replace the import with this R6RS library +- **`#f`**: Strip the import entirely (provided by base imports or unnecessary) +- Unmapped imports cause a compile error — this is intentional, so you catch + missing mappings early + +## Step 3: Define Base Imports + +Base imports are injected into every compiled module. They provide the runtime +environment that Gerbil code expects: + +```scheme +(define my-base-imports + '(;; Chez Scheme with exclusions for Gambit-compatible replacements + (except (chezscheme) void box box? unbox set-box! + andmap ormap iota last-pair find + 1+ 1- fx/ fx1+ fx1- + error error? raise with-exception-handler identifier? + hash-table? make-hash-table) + ;; Jerboa runtime + (compat types) + (runtime util) + (runtime table) + (runtime mop) + (runtime error) + (runtime hash) + ;; Gambit compatibility (threading, u8vectors, etc.) + (except (compat gambit) number->string make-mutex + with-output-to-string))) +``` + +The `(except (chezscheme) ...)` clause is critical. Jerboa's runtime +redefines several Chez builtins with Gambit-compatible versions. Without +the exclusions, R6RS reports import conflicts. + +## Step 4: Write build-jerboa.ss + +The complete translation driver: + +```scheme +#!chezscheme +(import + (except (chezscheme) void box box? unbox set-box! + andmap ormap iota last-pair find + 1+ 1- fx/ fx1+ fx1- + error error? raise with-exception-handler identifier? + hash-table? make-hash-table) + (compiler compile)) + +(define output-dir "src/mylib") + +;; Import map and base imports (as defined above) +(define my-import-map '(...)) +(define my-base-imports '(...)) + +;; Compile one module +(define (compile-module name) + (let* ((input (string-append "my-gerbil-project/" name ".ss")) + (output (string-append output-dir "/" name ".sls")) + (lib `(mylib ,(string->symbol name)))) + (printf " Compiling ~a.ss~n" name) + (guard (exn + (#t (printf " ERROR: ~a: ~a~n" name (condition-message exn)) + #f)) + (let* ((lib-form (gerbil-compile-to-library + input lib my-import-map my-base-imports)) + (lib-form (fix-import-conflicts lib-form))) + (call-with-output-file output + (lambda (port) + (display "#!chezscheme\n" port) + (parameterize ([print-gensym #f]) + (pretty-print lib-form port))) + 'replace) + (printf " OK: ~a~n" output) + #t)))) + +;; Compile in dependency order (leaves first) +(display "=== Translating .ss -> .sls ===\n") +(compile-module "util") +(compile-module "core") +(compile-module "main") + +(display "=== Translation complete ===\n") +``` + +### Key functions from `(compiler compile)`: + +- **`gerbil-compile-to-library`** `(input-path lib-name import-map base-imports)` + Reads a `.ss` file, compiles it through jerboa, wraps in an R6RS library form. + +- **`fix-import-conflicts`** `(lib-form)` + Post-processes the library to add `(except ...)` for local definitions that + shadow imports. Also fixes `set!`'d exports with `identifier-syntax`. + +- **`jerboa-compile-file`** `(input-path)` + Lower-level: compiles a file and returns the compiled forms (no library wrapper). + +- **`jerboa-compile-string`** `(code-string)` + Compiles a string of Gerbil code. Useful for testing. + +## Step 5: Write build-all.ss + +This simply imports every module, which triggers Chez's `--compile-imported-libraries`: + +```scheme +#!chezscheme +(import (mylib util) (mylib core) (mylib main)) +(printf "All modules compiled.~n") +``` + +## Step 6: Write Compatibility Shims + +Your project likely imports Gerbil standard library modules (`:std/sugar`, +`:std/format`, etc.). These need Chez equivalents in `src/compat/`. + +### Example: src/compat/sugar.sls + +```scheme +#!chezscheme +(library (compat sugar) + (export try catch finally with-catch + hash hash-ref hash-set! hash-update! hash-keys hash-values + defstruct defrules) + (import (chezscheme)) + + (define-syntax try + (syntax-rules (catch finally) + [(_ body (catch (var) handler ...)) + (guard (var [#t handler ...]) body)] + [(_ body (finally cleanup ...)) + (dynamic-wind void (lambda () body) (lambda () cleanup ...))])) + + ;; ... more shims + ) +``` + +### Common Compat Modules + +| Gerbil Module | Chez Equivalent | What It Provides | +|---------------|-----------------|------------------| +| `:std/sugar` | `(compat sugar)` | `try/catch/finally`, `hash` literals, `with-catch` | +| `:std/format` | `(compat format)` | `fprintf`, `printf`, `format` (Gambit-style) | +| `:std/sort` | `(compat sort)` | `sort`, `stable-sort` (Gambit arg order) | +| `:std/pregexp` | `(compat pregexp)` | Portable regex (pure Scheme) | +| `:std/misc/*` | `(compat misc)` | `string-split`, `string-join`, `path-expand`, etc. | +| `:std/error` | `(runtime error)` | `Error`, `error?`, `with-exception-catcher` | +| `:std/os/signal` | `(compat signal)` | Signal handling | +| `:std/os/fdio` | `(compat fdio)` | File descriptor I/O | + +You don't always need to write these from scratch. Check if jerboa-shell's +`src/compat/` already has what you need — many shims are reusable. + +## Step 7: Build and Test + +```bash +make # translate + compile +make run # run interpreted +make build # build standalone binary (if build-binary.ss exists) +``` + +## Common Issues and Fixes + +### Import Conflicts + +**Problem**: R6RS forbids a local `define` that shadows an imported name. + +**Fix**: `fix-import-conflicts` handles this automatically. If you see +"multiple definitions" errors, ensure you're calling it on the library form +before writing it out. + +### set! on Exported Variables + +**Problem**: R6RS forbids `set!` on exported variables. + +**Fix**: `fix-import-conflicts` rewrites these using `identifier-syntax` with +a vector cell: +```scheme +;; Before (forbidden in R6RS): +(export foo) +(define foo 0) +(set! foo 42) + +;; After (automatic rewrite): +(export foo) +(define foo-cell (vector 0)) +(define-syntax foo + (identifier-syntax + (id (vector-ref foo-cell 0)) + ((set! id v) (vector-set! foo-cell 0 v)))) +``` + +### Keyword Arguments + +**Problem**: Gerbil uses `key: value` keyword arguments. Jerboa translates +these, but `defclass` constructors may pass keyword values as positional args. + +**Fix**: Post-build patch in `build-jerboa.ss`: +```scheme +(patch-file! "src/mylib/types.sls" + "(lp (cddr rest) (cons (cadr rest) acc))]" + "(lp (cddr rest) acc)]") +``` + +### make-mutex String Arguments + +**Problem**: Gambit's `make-mutex` accepts strings; Chez requires symbols. + +**Fix**: Post-build patch: +```scheme +(patch-file! "src/mylib/foo.sls" + "(make-mutex \"my-lock\")" + "(make-mutex 'my-lock)") +``` + +### void Recursion + +**Problem**: Gerbil defines `(define (void) ...)` which creates a recursive +definition in the translated output. + +**Fix**: Post-build patch: +```scheme +(patch-file! "src/mylib/foo.sls" + "(define (void) (void))" + "(define (void . _) (if #f #f))") +``` + +### Chez Lazy Library Invocation + +**Problem**: Chez only invokes a library when one of its exports is first +referenced at runtime. If a module registers side effects at load time (like +registering builtins), they won't execute. + +**Fix**: Add a dummy reference in a module that's guaranteed to be invoked: +```scheme +;; In main.sls, force invocation of side-effecting modules: +(define _force-builtins special-builtin?) ;; references a builtins export +``` + +### Threading + +**Problem**: Programs in Chez boot files cannot create threads (GC futex +deadlock). + +**Fix**: Put only libraries in the boot file. Load the program at runtime via +`memfd` + `Sscheme_script`. See [docs/single-binary.md](single-binary.md). + +## FFI (C Bindings) + +For projects that need C FFI: + +1. Write a C shim (`ffi-shim.c`) with the functions you need +2. Compile to a shared library: `gcc -shared -fPIC -o libffi.so ffi-shim.c` +3. Create `src/compat/ffi.sls` using Chez's `foreign-procedure` and `load-shared-object` +4. Map Gerbil's FFI imports to your compat module + +For standalone binaries, the C shim is compiled directly into the binary. See +[docs/single-binary.md](single-binary.md). + +## Optimization + +For production builds: + +```bash +make compile-opt3 # optimize-level 3 +make compile-wpo # whole-program optimization +``` + +See [docs/optimization.md](optimization.md) for tuning `cp0-effort-limit`, +per-file directives, and benchmark results. + +## Reference: Existing Projects + +Study these for real-world patterns: + +| Project | Modules | Key Patterns | +|---------|---------|--------------| +| [jerboa-shell](https://github.com/ober/jerboa-shell) | 30+ | FFI shim, binary building, post-build patches | +| jerboa-kunabi | 10+ | Cross-project deps (jerboa-aws), sed-based patches | +| jerboa-lsp | 53 | Large module count, JSON/HTTP compat | new file mode 100644 --- /dev/null +++ b/docs/goals.md @@ -0,0 +1,177 @@ +# Making Jerboa (Gerbil-on-Chez) Superior to Other Schemes/Lisps + +## Strategic Advantages to Exploit + +### 1. True SMP Concurrency with Ergonomic Syntax + +**The gap**: Chez has real OS threads and SMP. Gerbil has actors/channels syntax. But currently threading.sls is just a thin SRFI-18 shim. Nobody in Scheme-land has truly ergonomic parallel programming. + +**The opportunity**: Build a work-stealing scheduler on Chez's native threads with Gerbil's actor syntax as the front-end. Think Go's goroutines but with Gerbil's `spawn`, channels, and `select`. Chez's GC is already thread-safe. You'd have: + +```scheme +(spawn (lambda () (channel-put ch (heavy-computation)))) +(for/collect ([x (in-channel ch)]) (process x)) +``` + +**Why this wins**: Racket CS has green threads but no true SMP parallelism in user code. Gambit has SMP but it's fragile. Guile has Fibers but limited. Chez alone has robust SMP but no ergonomic API. You'd be the only Scheme with both. + +--- + +### 2. Zero-Copy FFI with Chez's Native Calling Convention + +**The gap**: FFI is currently stubbed. But Chez's `foreign-procedure` is one of the fastest FFIs in any Scheme -- direct C calling convention, no marshaling for simple types. + +**The opportunity**: Build a Gerbil-syntax FFI that compiles to Chez's native foreign calls. Something like: + +```scheme +(extern libsqlite3 + (sqlite3_open (string (* void)) -> int) + (sqlite3_exec ((* void) string (* void) (* void) (* (* char))) -> int)) +``` + +Compile-time type checking, automatic resource cleanup via `unwind-protect`, and GC-safe pinning. Chez's `foreign-callable` lets C call back into Scheme with full GC -- exploit this for event-driven libraries. + +**Why this wins**: Racket's FFI has overhead. Gambit's is capable but requires C stub files. Chez's is the fastest but has no high-level DSL. You could have the speed of Chez's FFI with the ergonomics of Gerbil's macro system. + +--- + +### 3. Ahead-of-Time Native Binaries with Tree Shaking + +**The gap**: `jerboa-make-binary` exists in skeleton form. Chez can produce standalone executables via `compile-whole-program`. But nobody in Scheme-land does proper dead code elimination + native binary in one step. + +**The opportunity**: Since the compiler already tracks module dependencies via the loader, you have the dependency graph. Add: +- Whole-program compilation via `compile-whole-program` +- Dead export elimination (you know what's imported where) +- Single static binary output (no .boot file needed) +- Startup time measured in microseconds, not milliseconds + +``` +$ jerboa build --static myapp.ss -o myapp +$ ldd myapp + not a dynamic executable +$ time ./myapp + real 0.003s +``` + +**Why this wins**: Go and Rust win converts partly on "single binary deployment." No Scheme does this well. Racket CS binaries are 30+ MB with slow startup. Gambit can do it but the tooling is painful. A `jerboa build` that produces a 2MB static binary starting in 3ms would be a category killer for CLI tools and microservices. + +--- + +### 4. First-Class Structured Concurrency + +**The gap**: No Scheme has structured concurrency (nurseries/task groups a la Trio/Java 21). Gerbil's actors are fire-and-forget. + +**The opportunity**: Build structured concurrency as a core primitive: + +```scheme +(with-task-group + (lambda (tg) + (task-group-spawn tg (lambda () (fetch-url url1))) + (task-group-spawn tg (lambda () (fetch-url url2))) + ;; both complete or both cancel when scope exits + )) +``` + +Built on Chez's threads + Chez's delimited continuations for cancellation. The scope guarantees no leaked goroutines, no orphan threads. This is what Go, Erlang, and most actor systems get wrong. + +**Why this wins**: This is cutting-edge in every language. Java just got it in Java 21. Python has Trio. No Lisp/Scheme has it. You'd be first. + +--- + +### 5. Module System with Hermetic Builds + +**The gap**: The module loader already caches to `/tmp/jerboa-modules/` with timestamp invalidation. But it's ad-hoc. + +**The opportunity**: Content-addressed module cache. Hash the source + dependencies -> deterministic output. This gives you: +- Reproducible builds (same source -> same binary, always) +- Distributed build cache (share compiled artifacts across machines) +- Incremental recompilation (only rebuild what changed) +- Parallel module compilation (Chez's thread safety enables this) + +``` +/cache/ + abc123.so <- hash of (std/sort) source + deps + def456.so <- hash of (std/text/json) source + deps +``` + +**Why this wins**: Racket has a compilation manager but it's filesystem-timestamp-based and single-threaded. Nobody has content-addressed Scheme builds. This is what Bazel/Nix do for C++ -- apply it to Scheme. + +--- + +### 6. Gradual Typing That Doesn't Suck + +**The gap**: Typed Racket exists but imposes 10-100x overhead at typed/untyped boundaries. No other Scheme has gradual typing. + +**The opportunity**: Since the compiler controls code generation, you can: +- Add optional type annotations that compile to Chez assertions in debug mode +- Eliminate type checks entirely in release mode +- Use Chez's profile-guided optimization data to specialize hot paths + +```scheme +(def (fibonacci [n : fixnum]) : fixnum + (if (fx< n 2) n + (fx+ (fibonacci (fx- n 1)) (fibonacci (fx- n 2))))) +``` + +Annotations are optional. When present, the compiler emits specialized code. No contracts at module boundaries -- just direct calls. Zero overhead in release mode. + +**Why this wins**: Typed Racket's boundary costs are its fatal flaw. Common Lisp has `declare` but it's ugly and compiler-dependent. The position between Gerbil's syntax and Chez's optimizer is ideal for this. + +--- + +### 7. Embeddable Runtime + +**The gap**: No Scheme is easy to embed as a library in C/C++/Rust applications the way Lua is. + +**The opportunity**: Chez is already a C library (`scheme.h`). Build a clean embedding API: + +```c +jerboa_t *j = jerboa_new(); +jerboa_eval(j, "(def greeting \"hello\")"); +const char *s = jerboa_get_string(j, "greeting"); +jerboa_call(j, "my-function", 2, jerboa_int(42), jerboa_string("foo")); +jerboa_destroy(j); +``` + +Multiple independent instances, each with their own heap. Thread-safe. This opens up the "scripting language for applications" niche that Lua dominates and Guile targets but fails at (too heavy). + +--- + +### 8. LSP + IDE Integration from Day One + +**The gap**: Scheme IDE support is universally terrible. Even Racket's is mediocre outside DrRacket. + +**The opportunity**: The compiler already has source locations from the reader, module dependency graphs from the loader, and type information from the MOP. Wire this into an LSP server: +- Go-to-definition (you track where things are defined) +- Completion (you know module exports) +- Inline errors (the compiler gives file/line/column) +- Hover types (from the MOP's class info) + +An LSP server written *in Jerboa itself* that's fast because it runs on Chez. + +--- + +## Prioritized Roadmap + +If picking the **top 3** that would create the most distance from the competition: + +| Priority | Feature | Why | +|----------|------------------------------------------|--------------------------------------------------| +| **1** | SMP actors + structured concurrency | Unique selling point; no Scheme has this | +| **2** | Static native binaries with tree shaking | Practical; wins converts from Go/Rust | +| **3** | Zero-overhead FFI DSL | Unlocks real-world libraries (SQLite, TLS, etc.) | + +Everything else (gradual types, LSP, embedding) is valuable but can come later. The concurrency story + deployment story + C interop story are what make people choose a language for real projects vs. hobby use. + +--- + +## What You Already Have That Others Don't + +Don't underestimate what's already unique: +- **Gerbil's syntax on Chez's runtime** -- nobody else has this combination +- **A self-hosting compiler in ~1100 lines** -- Racket CS's equivalent is 50K+ lines +- **51 stdlib modules** -- practical coverage of crypto, db, networking, OS +- **8 chez-* FFI libraries** -- sqlite, postgresql, crypto, epoll, inotify, ssl, zlib, pcre2 +- **Subprocess-batched testing** -- the OOM solution is actually a good architecture for parallel test execution + +The foundation is strong. The question is whether to go deep on making the existing modules fully functional (practical completeness) or go wide on the differentiators above. Recommendation: get FFI working (item 3) because it unblocks items 1 and 2, then build the concurrency story on top. new file mode 100644 --- /dev/null +++ b/docs/lsp-conversion.md @@ -0,0 +1,269 @@ +# Things I Learned: Converting gerbil-lsp to jerboa-lsp + +Notes from porting a 53-module Gerbil Scheme LSP server to run on Chez Scheme +via the Jerboa translation framework. Covers bugs found, compatibility issues, +and design decisions. + +## Jerboa Framework Bugs Found and Fixed + +### 1. Dotted Pair Reversal in the Reader + +**File:** `src/reader/reader.sls` (line 249) +**Commit:** `87acce6` + +The reader reversed elements in improper lists. `(foo a b . rest)` was read as +`(b a foo . rest)`. This caused function parameter lists to be scrambled whenever +a rest argument was present — the last required parameter became the function +name. + +**Root cause:** Double-reversal. As the reader consumed tokens, it accumulated +them in `acc` via prepend, giving `acc = (b a foo)`. The code then called +`(reverse acc)` producing `(foo a b)`, but the subsequent `build` loop folded +left-to-right, reversing again to `(b a foo . rest)`. + +**Fix:** Removed `(reverse acc)`. The accumulator is already in the right order +for the fold: + +```scheme +;; acc = (b a foo) — fold produces (foo a b . rest) +(let build ((items acc) (result tail)) + (if (null? items) result + (build (cdr items) (cons (car items) result)))) +``` + +### 2. Dot-Notation Applied to Uppercase Identifiers + +**File:** `src/compiler/compile.sls` (function `dot-notation?`) +**Commit:** `61337f7` + +`CompletionItemKind.Snippet` compiled to `(slot-ref CompletionItemKind 'Snippet)` +instead of staying as a plain identifier. This affected all uppercase dotted +constants throughout the LSP types module (`DiagnosticSeverity.Error`, +`SymbolKind.File`, etc.). + +**Root cause:** `dot-notation?` treated any symbol containing a dot as a slot +access expression. It had no way to distinguish `object.field` (instance access) +from `EnumType.Value` (a constant identifier). + +**Fix:** Added `char-lower-case?` check on the first character. By Gerbil +convention, instance variables start lowercase; class/enum constants start +uppercase: + +```scheme +(define (dot-notation? sym) + (and (symbol? sym) + (let ((s (symbol->string sym))) + (and (> (string-length s) 2) + (char-lower-case? (string-ref s 0)) ;; only lowercase = slot access + ...)))) +``` + +### 3. Relative `./foo` Imports Without Source Directory Context + +**File:** `src/compiler/compile.sls` +**Commit:** `953fa8d` + +`./parser` imported from `lsp/analysis/symbols.ss` resolved to `(lsp parser)` +instead of `(lsp analysis-parser)`. Broke all cross-references within +subdirectories. + +**Root cause:** The compiler stripped `./` to get `"parser"`, combined with +`*default-package* = lsp` to produce `(lsp parser)`. It had no knowledge of the +source file's directory (`lsp/analysis/`). + +**Fix:** Threaded a `source-dir` parameter through `gerbil-compile-to-library` -> +`compile-library-imports` -> `resolve-import`. Added `resolve-with-context` that +strips the package prefix from the source directory and prepends the subdirectory: + +``` +source-dir = "lsp/analysis", default-pkg = lsp +-> subdir = "analysis" +-> "./parser" -> "analysis-parser" +-> library name: (lsp analysis-parser) +``` + +A secondary bug: `rel-path` (a symbol) was passed to `normalize-relative-path` +which expects a string. Fixed with `(if (string? rel-path) rel-path +(symbol->string rel-path))`. + +### 4. Keyword Objects in `case` Clause Datums + +**File:** `src/compiler/compile.sls` (function `compile-case-clause`) +**Commit:** `be65745` + +Gerbil keywords like `package:` used as `case` datums compiled to +`#[keyword-object "package"]` — an internal notation Chez cannot read. + +**Root cause:** `gerbil-compile-expression` handled keywords in expression +positions but `case` datums are literal data passed through unchanged. + +**Fix:** Map over datums and convert keyword objects to colon-suffixed symbols: + +```scheme +(define (compile-case-clause clause) + (if (eq? (car clause) 'else) + `(else ,@(map gerbil-compile-expression (cdr clause))) + (let ((datums (map (lambda (d) + (cond + ((|##keyword?| d) + (string->symbol + (string-append (|##keyword->string| d) ":"))) + (else d))) + (car clause)))) + `(,datums ,@(map gerbil-compile-expression (cdr clause)))))) +``` + +## Gerbil <-> Chez Compatibility Issues + +### `#:keyword` Syntax + +Gerbil's `#:parse-error` keyword literal syntax is not recognized by the Jerboa +reader. Changed to plain symbol `'parse-error`. Note: `'parse-error:` (with +colon suffix) also doesn't work — Chez sees it as a keyword-object literal. + +### R6RS Definition Ordering + +Gerbil allows `define` anywhere in a body. R6RS requires all definitions before +expressions. Had to reorder code in `completion-data.ss` where a `define` +appeared after `lsp-debug` and `when` calls. + +### Binary vs Textual Ports + +Chez's `current-input-port` is textual. The LSP transport reads raw bytes via +`read-u8` / `read-subu8vector`. Gambit's `read-u8` works on any port; Chez's +`get-u8` requires a binary port. + +Fix: port-type dispatch in `read-u8`: + +```scheme +(define (read-u8 port) + (if (binary-port? port) + (get-u8 port) + (let ((c (read-char port))) + (if (eof-object? c) + (eof-object) + (char->integer c))))) +``` + +### Chez `hashtable` vs Jerboa Runtime Hash Tables + +The Jerboa runtime wraps hash tables in its own `gerbil-struct` record type. +Chez has a native `hashtable` type. If a compat module creates hash tables with +Chez primitives (`make-hashtable`), they crash when passed to Jerboa's +`hash-ref` / `hash-put!`. + +**All compat modules that create hash tables must use `(runtime hash)`:** +- `(make-hashtable symbol-hash eq?)` -> `(make-hash-table)` +- `(hashtable-set! ht k v)` -> `(hash-put! ht k v)` +- `(hashtable? val)` -> `(hash-table? val)` +- `(hashtable-entries ht)` -> `(hash->list ht)` (returns alist) + +### R6RS Import Conflicts + +Multiple "multiple definitions" errors when both `(chezscheme)` and a compat +module export the same name. Fixed with `except` clauses: + +```scheme +(import + (except (chezscheme) sort sort! fprintf printf iota path-extension) + (compat misc) + (compat sort) + (compat format) + ...) +``` + +Important: don't accidentally exclude `format` from `(chezscheme)` — the compat +format module only provides `fprintf`/`printf`, not `format`. + +### `begin-ffi` Is Gambit-Only + +The original `main.ss` had a ~100-line `begin-ffi` block for SIGVTALRM spin +detection (a Gambit threading workaround). Removed entirely — Chez doesn't have +this issue. + +## Compat Modules: What and Why + +| Module | Replaces | Key Functions | +|--------|----------|---------------| +| `json.sls` | `:std/text/json` | `read-json`, `write-json`, `string->json-object`, `json-object->string` | +| `getopt.sls` | `:std/cli/getopt` | `call-with-getopt`, `flag`, `option` | +| `process.sls` | `:std/os/process` | `run-process` with `coprocess:` keyword | +| `format.sls` | `:std/format` | `fprintf`, `printf` (not `format` — Chez has that) | +| `sort.sls` | `:std/sort` | `sort`, `sort!` with Gerbil's comparator convention | +| `misc.sls` | Various `:std/*` | `iota`, `read-line`, `path-extension`, `path-strip-extension`, etc. | +| `sugar.sls` | `:std/sugar` | `try`/`catch`/`finally`, `hash`, `defrule` | +| `gambit.sls` | Gambit builtins | `make-mutex`, `mutex-lock!`, `fork-thread`, `with-output-to-string` | +| `lsp/compat.sls` | `lsp/compat/compat.ss` | Bridges all the above + Gambit internal stubs | + +### Gambit Internal Stubs in `lsp/compat.sls` + +These Gambit-specific functions had no Chez equivalent and needed stubs: + +- `##readenv-current-filepos`, `##filepos-line`, `##filepos-col` — parser error + line/column extraction (always returns 0; code path never reached since + `datum-parsing-exception?` returns `#f`) +- `thread-terminate!` — no-op (Chez threads must finish cooperatively) +- `input-port-line`, `input-port-column` — return 1 (Chez doesn't track these) +- `keyword?` — checks if symbol ends with `:` (Jerboa's keyword convention) +- `type-of` — runtime type dispatch returning a symbol + +### FFI Stubs in `gambit.sls` + +The gambit.sls from jerboa-shell had `foreign-procedure` calls for process and +terminal management (`ffi_do_waitpid`, `ffi_set_raw_mode`, etc.). These required +jerboa-shell's C shared library. Replaced with no-op stubs since the LSP server +doesn't need them. + +## Build Infrastructure Decisions + +### Module Naming Convention + +Nested paths are flattened with hyphens: +- `lsp/analysis/parser.ss` -> library `(lsp analysis-parser)`, file `src/lsp/analysis-parser.sls` +- `lsp/util/log.ss` -> library `(lsp util-log)`, file `src/lsp/util-log.sls` + +The top-level directory becomes the R6RS library package. The `*default-package*` +import map key tells the compiler to use `lsp` for all unmapped relative imports. + +### 4-Stage Build Pipeline + +1. **Jerboa translate** (`.ss` -> `.sls`): `build-jerboa.ss` drives the Jerboa + compiler, fixes import conflicts, and handles mutable exports +2. **Chez compile** (`.sls` -> `.so`): `build-all.ss` with + `compile-imported-libraries` +3. **Boot file creation**: `make-boot-file` bundles all `.so` files in dependency + order +4. **Binary linking**: C main (`lsp-main.c`) embeds boot files as C byte arrays + via `memfd_create` + `Sregister_boot_file_bytes` + +### Compilation Order Matters + +Modules must be compiled in dependency order. The build script uses 5 tiers: + +1. Compat layer +2. Utilities (no inter-dependencies) +3. Core protocol (types, jsonrpc, transport, state, server) +4. Analysis modules +5. Handlers + main + +### Link Flags + +The standalone binary needs `-lncurses` because Chez Scheme's `libkernel.a` +contains the expeditor (line editor) which depends on ncurses terminal functions. +Without it, linking fails with undefined references to `cur_term`, `tputs`, +`setupterm`. + +### Unused Modules in Boot File + +`compile-program` only compiles modules reachable via the import chain. Several +compat modules (`pregexp`, `signal`, `signal-handler`, `fdio`) were listed in the +boot file but never imported. Their `.so` files didn't exist, causing +`make-boot-file` to fail. Removed them from the boot file list. + +## Result + +- 5.6 MB standalone ELF binary +- Only system library dependencies: libc, libm, libtinfo +- All 53 original `.ss` files preserved (3 with minor modifications) +- 13/15 e2e protocol tests pass +- 4 Jerboa framework bugs fixed upstream new file mode 100644 --- /dev/null +++ b/docs/optimization.md @@ -0,0 +1,234 @@ +# Chez Scheme Optimization Techniques for Jerboa + +Techniques from the Chez Scheme compiler (v10.4.0) that can improve performance of Gerbil-to-Chez converted code. + +## 1. Compiler Parameter Tuning + +The most impactful knobs for compiled Jerboa output: + +| Parameter | Default | Recommendation | +|-----------|---------|----------------| +| `optimize-level` | 2 | Use 3 for well-tested code (removes all type checks) | +| `cp0-effort-limit` | 200 | Increase to 500+ for generated code with deep nesting | +| `cp0-score-limit` | 20 | Increase to 50 to allow more aggressive inlining | +| `enable-type-recovery` | #t | Keep enabled — helps recover types after Gerbil wrapping layers | +| `enable-cross-library-optimization` | #t | Critical since Jerboa emits many small R6RS libraries | + +Set these per-file in the compilation pipeline via `(eval-when (compile) ...)` for hot modules. + +## 2. Record Type Flags + +When compiling `defstruct`/`defclass` to Chez records, emit: + +- **`(sealed #t)`** — prevents subclassing, enables direct field access without vtable indirection +- **`(immutable)` on fields** where Gerbil code never uses `set!` on slots — cp0 can then fold field accesses +- **`(nongenerative <uid>)`** — enables cross-library type identity and optimization + +Currently the MOP uses `gerbil-struct` wrappers. For performance-critical structs, consider emitting native Chez `define-record-type` directly with these flags. + +## 3. Numeric Specialization + +Chez unboxes flonums at level 3 but only with explicit flonum ops: + +```scheme +;; Generic (boxes intermediate results, GC pressure) +(+ (* x y) z) + +;; Specialized (raw FPU instructions, no allocation) +(fl+ (fl* x y) z) +``` + +The compiler could emit `fx+`/`fl+` when type annotations are available from Gerbil's `def` forms (e.g., `(def (f (x : fixnum)) ...)`). + +## 4. Local Binding Optimization + +Chez's cp0 aggressively inlines **locally-bound, unassigned procedures**. Top-level `define`s across library boundaries are harder to optimize. This matters because Jerboa emits many small libraries. + +**Actionable**: When a Gerbil helper function is only used in one place, emit it as a local `let` binding rather than a top-level `define`. + +## 5. Type Guard Patterns + +cp0 + cptypes recognize predicate-guarded branches: + +```scheme +;; cptypes knows x is a pair in the consequent +(if (pair? x) (car x) ...) + +;; cptypes knows x is a fixnum +(if (fixnum? x) (fx+ x 1) ...) +``` + +The `match` compiler already emits type tests — make sure the pattern is `(if (predicate? x) <then> <else>)` rather than intermediate wrappers that obscure the type information from cptypes. + +## 6. Avoid the WPO Trap + +WPO breaks `identifier-syntax` mutable export cells. But for pure-Scheme modules without mutable exports, WPO could still be applied selectively. Consider a build flag that enables WPO per-module. + +## 7. Letrec Decomposition + +Chez's `cpletrec` pass decomposes `letrec*` into simpler forms. Since Gerbil's top-level `def` forms compile to `letrec*` bodies in R6RS libraries, this pass is already working for you. But ordering matters — put pure definitions before impure ones to help cpletrec separate them. + +## 8. Primitive Inlining Flags + +Each Chez primitive has optimization flags (`pure`, `unrestricted`, `cp02`, `arith-op`, etc.) in `primdata.ss`. When the compat layer wraps Chez primitives (e.g., `hash-ref` wrapping `hashtable-ref`), the wrapper obscures these flags from the optimizer. + +**Actionable**: For hot-path compat wrappers, consider: +- Using `define-syntax` + `syntax-rules` instead of `define` to make them transparent to cp0 +- Or mark them with integration info if applicable + +## 9. Closure Avoidance in Loops + +Chez doesn't optimize closures allocated inside loops. The `for`/`for/collect` compilation should ensure lambda bodies are hoisted when they don't capture loop variables. + +## 10. Quick Wins Checklist + +- [ ] Add `(optimize-level 3)` option to gxc for release builds +- [ ] Emit `sealed` records for `defstruct` without known subclasses +- [ ] Emit `immutable` fields for non-mutated struct slots +- [ ] Emit `fx+`/`fx-` for integer arithmetic in type-annotated code +- [ ] Ensure `match` type tests are direct `(if (pred? x) ...)` — not wrapped +- [ ] Increase `cp0-effort-limit` for generated code (deep nesting from macro expansion) +- [ ] Consider selective WPO for pure modules without mutable exports +- [ ] Profile the compat wrappers (hash, MOP dispatch) — these are likely the hottest paths + +--- + +## 11. Chez Scheme Optimization Functions & APIs + +### `expand/optimize` — Inspect Optimizer Output + +Use this to see what Chez's optimizer actually does with generated code. Invaluable for checking whether compat wrappers are being inlined or blocking optimization: + +```scheme +(expand/optimize '(let ([y '(3 . 4)]) (+ (car y) (cdr y)))) +;=> 7 (fully folded at compile time) + +(print-gensym #f) +(expand/optimize + '(let ([y '(3 . 4)]) + (lambda (x) (* (+ (car y) (cdr y)) x)))) +;=> (lambda (x) (#2%* 7 x)) +``` + +### `compile-whole-program` — Whole-Program Optimization + +The most aggressive optimization Chez offers. Discards unused code and optimizes across all library boundaries: + +```scheme +(parameterize ([generate-wpo-files #t]) + (compile-program "main.ss" "main.so")) +(compile-whole-program "main.wpo" "main-opt.so") +``` + +Returns a list of libraries that couldn't be incorporated (missing `.wpo` files). Breaks `identifier-syntax` mutable export cells — use only for modules without mutable exports. + +### `compile-whole-library` — Same for Libraries + +Like `compile-whole-program` but for library bundles. Produces both `.so` and `.wpo` output. + +### `$primitive` / `#%` Syntax — Force Original Primitives + +Bypass any user redefinitions and control per-call optimization level: + +```scheme +(#3%car x) ; car at optimize-level 3 (no type check) +(#2%+ a b) ; + at optimize-level 2 (with checks) +(#%vector-ref v i) ; original primitive, current optimize level +``` + +## 12. Complete Parameter Reference + +| Parameter | Default | Effect | +|-----------|---------|--------| +| `optimize-level` | 0 | 0-2 = safe; **3 = unsafe** (no type/bounds checks) | +| `run-cp0` | 2-pass | Controls the source optimizer; can disable or run N passes | +| `cp0-effort-limit` | 200 | Max work per inlining attempt | +| `cp0-score-limit` | 20 | Max code growth per inlining | +| `cp0-outer-unroll-limit` | 0 | Loop unrolling (set to 1+ to enable) |