Initialize Jerboa DuckDB Phase 0 contracts
Jaime Fournier
23e016e4695e33684605c53f640c115b7d7889d7
new file mode 100644 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +.DS_Store +*.so +*.wpo +dist/ +tmp/ new file mode 100644 --- /dev/null +++ b/.gitsafe.json @@ -0,0 +1,5 @@ +{ + "exclude": [ + "contracts/generated/upstream-inventory.sexp" + ] +} new file mode 100644 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,504 @@ +## The Jerboa Language — Quick Reference + +Jerboa is a Scheme dialect built on Chez Scheme. It is Gerbil-inspired but its own language. **All user-facing code is `.ss` files. Never write `.sls` files for the user** — those are internal implementation files. + +### File Structure + +Every Jerboa file looks like this: + +```scheme +(import (jerboa prelude)) ;; ONE import gives you the ENTIRE language +;; Optional extra imports for modules NOT in the prelude: +;; (import (std net request)) + +(def (my-function x y) + (+ x y)) + +(displayln (my-function 1 2)) +``` + +Run with: `jerboa run file.ss` + +**NEVER** write `(library ...)` forms — that's `.sls` internal syntax. + +### Reader Syntax Extensions + +``` +[...] → plain parentheses — same as Gerbil and Chez Scheme +{method obj args} → (~ obj 'method args) — method dispatch +name: → keyword #:name +:std/sort → (std sort) — Gerbil-style module path +#<<END ... END → heredoc string +``` + +Square brackets `[...]` are interchangeable with `(...)`, exactly like Gerbil and stock Chez Scheme. You can freely use them in bindings, match clauses, and anywhere you'd use parentheses: +```scheme +;; All of these are correct: +(let ([x 1] [y 2]) (+ x y)) +(for/collect ([x (in-range 5)]) (* x x)) +(match val ([list a b] (+ a b))) +(cond [(> x 0) "positive"] [else "non-positive"]) +``` + +### CRITICAL: Things That DO NOT EXIST in Jerboa/Chez + +Claude frequently hallucinates these from Gerbil, Gambit, Racket, or R7RS training data. +**NONE of them are real in Jerboa/Chez. STOP and use the correct form.** + +#### AI Compatibility Aliases (these now work in the prelude) +The following names from other Scheme dialects are aliased in `(jerboa prelude)`: +- `hash-has-key?` → `hash-key?` (Racket) +- `hash-table-set!` → `hash-put!` (Racket) +- `directory-exists?` → `file-directory?` (Gambit) +- `eql?` → `eqv?` (Common Lisp) +- `random-integer` → `random` (Gambit) +- `read-line` → `get-line` wrapper (Gambit) — works with or without port arg +- `force-output` → `flush-output-port` wrapper (Gambit) — works with or without port arg +- `string-map` → char-level map (Racket/R7RS) — `(string-map f str)` + +#### Hallucinated Functions (still do NOT exist) +- `symbol<?` — use `(lambda (a b) (string<? (symbol->string a) (symbol->string b)))` +- `string-contains?` — use `(string-contains str sub)` (returns index or #f, NOT boolean) +- `define-struct` — use `(defstruct name (fields ...))` +- `raise` with a string — use `(error 'who "message" irritants ...)` +- `environment-bound?` — Gerbil-only. No direct Chez equivalent + +#### Gerbil/Gambit-isms (from training data — wrong in Jerboa) +- `time->seconds` — use `(time-second (current-time))` for epoch seconds +- `thread-sleep!` — Gambit. Use `(sleep (make-time 'time-duration 0 seconds))` +- `thread-yield` — no Chez equivalent. Use `(sleep (make-time 'time-duration 0 0))` as workaround +- `path-expand` with 2 args — Gerbil takes `(path-expand rel base)`. Jerboa takes 1 arg. Use `(path-join base rel)` for 2-arg version +- `process-status` — Gerbil. Use `(std misc process)` API in Jerboa +- `user-info-home` — Gerbil. Use `(getenv "HOME")` +- `the-environment` — Gerbil. Use `(interaction-environment)` in Chez +- `condition/report-string` — Gerbil. Use `(with-output-to-string (lambda () (display-condition c)))` +- `make-class-type` — Gerbil. Use `(defstruct ...)` or `(defclass ...)` in Jerboa +- `string-subst` — Gerbil. Not in prelude. Use `(string-replace str old new)` or implement manually +- `open-fd-pair` — Gambit. Does not exist in Chez; requires different API + +#### R6RS/Racket-isms (wrong variant) +- `make-equal-hashtable` — R6RS. Use `(make-hash-table)` from Jerboa prelude +- `arithmetic-shift` — Racket. Use `(bitwise-arithmetic-shift n k)` or `(ash n k)` in Chez +- `pregexp-match` — Racket. Use `(std text regex)` or `(std pregexp)` API in Jerboa + +### CRITICAL: Common Arity Mistakes + +- `(list-of? pred)` → returns a PREDICATE. It takes 1 arg. Use: `((list-of? number?) lst)` +- `(maybe pred)` → returns a PREDICATE. It takes 1 arg. Use: `((maybe string?) val)` +- `(in-range end)` or `(in-range start end)` or `(in-range start end step)` — NOT `(in-range start step end)` +- `(hash-ref ht key)` or `(hash-ref ht key default)` — NOT `(hash-ref key ht)` +- `(string-split str delimiter)` where delimiter is a CHAR: `(string-split "a,b" #\,)` +- `(make-rwlock)` — takes **0 args**, NOT `(make-rwlock 'name)` (Gerbil takes a name; Jerboa does not) +- `(path-expand path)` — takes **1 arg**, NOT `(path-expand rel base)` (Gerbil takes 2; use `path-join` for 2-arg) +- `(sort list predicate)` — Jerboa `(std sort)`/prelude order. Raw Chez `sort` is predicate-first, but Jerboa-facing code should use list first. + +### Core Forms (all from `(import (jerboa prelude))`) + +#### Definitions +```scheme +(def x 42) ;; variable +(def (f x y) (+ x y)) ;; function +(def (f x (y 10)) body) ;; optional param with default +(def (f x . rest) body) ;; rest args +(def* f ((x) ...) ((x y) ...)) ;; multi-arity +(defrule (name pat) template) ;; macro +``` + +#### Data Structures +```scheme +(defstruct point (x y)) ;; → make-point, point?, point-x, point-y, point-x-set! +(defstruct (circle shape) (radius)) ;; inheritance (single only) +(defmethod (area (self circle)) body) ;; method on type +(~ obj 'method arg ...) ;; dispatch (or {method obj arg ...}) +(defrecord person (name age)) ;; struct + pretty-print + ->alist +(define-enum color (red green blue)) ;; → color-red, color?, color->name +``` + +#### Pattern Matching +```scheme +(match value + (42 "exact") ;; literal + ((list a b c) (+ a b c)) ;; list destructure + ((cons h t) h) ;; pair + ((? number?) "num") ;; predicate + ((? string? s) (string-upcase s)) ;; predicate + bind + ((and (? number?) (? positive?)) "positive number") + ((or "yes" "y") #t) + ((=> string->number n) n) ;; view pattern + (n (where (> n 0)) "positive") ;; guard + (_ "default")) ;; wildcard +``` + +#### Error Handling +```scheme +(try expr (catch (e) handler) (finally cleanup)) +(try expr (catch (error? e) handler)) +(unwind-protect body cleanup) +(with-resource (var init cleanup) body) +``` + +#### Result Type (Rust-inspired ok/err) +```scheme +(ok 42) (err "bad") (ok? r) (err? r) +(unwrap (ok 42)) ;; → 42 (raises on err) +(unwrap-or (err "x") 0) ;; → 0 +(map-ok f result) (map-err f result) +(and-then result f) ;; monadic bind +(try-result expr) ;; exceptions → (err condition) +(try-result* expr) ;; exceptions → (err "message string") +(sequence-results list-of-results) ;; → (ok list) or first (err) +(->? (ok 10) (+ 5) (* 2)) ;; → (ok 30), short-circuits on err +``` + +#### Iterators +```scheme +(for ((x (in-range 5))) (displayln x)) +(for/collect ((x (in-range 5))) (* x x)) ;; → (0 1 4 9 16) +(for/fold ((sum 0)) ((x (in-range 10))) (+ sum x)) ;; → 45 +(for/or ((x lst)) (and (pred? x) x)) ;; first truthy +(for/and ((x lst)) (pred? x)) ;; all truthy + +;; Iterators: in-list, in-vector, in-string, in-range, in-hash-keys, +;; in-hash-values, in-hash-pairs, in-naturals, in-indexed, +;; in-port, in-lines, in-chars, in-bytes, in-producer +``` + +#### Threading Macros +```scheme +(-> x (f a) (g b)) ;; thread first: (g (f x a) b) +(->> x (f a) (g b)) ;; thread last: (g b (f a x)) +(as-> x v (f v) (g v)) ;; named +(some-> x (f) (g)) ;; short-circuit on #f +(cond-> x test (f) t2 (g)) ;; conditional steps +(->? (ok x) (f) (g)) ;; result-aware thread first +``` + +#### Ergo Typing +```scheme +(: expr pred?) ;; checked cast +(using (p (make-point 1 2) : point?) + (+ p.x p.y)) ;; dot-access → (point-x p) etc. +((list-of? number?) '(1 2 3)) ;; predicate factory → #t +((maybe string?) #f) ;; accepts #f or string → #t +``` + +#### Hash Tables +```scheme +(def ht (make-hash-table)) +(hash-put! ht "key" "val") +(hash-ref ht "key") ;; error if missing +(hash-ref ht "key" "default") ;; with default +(hash-get ht "key") ;; → val or #f +(hash-key? ht "key") ;; → #t/#f +(hash-remove! ht "key") +(hash->list ht) (hash-keys ht) (hash-values ht) +(hash-for-each (lambda (k v) ...) ht) +(list->hash-table '(("a" . 1) ("b" . 2))) +``` + +#### Strings +```scheme +(string-split "a,b,c" #\,) ;; → ("a" "b" "c") NOTE: char delimiter +(string-join '("a" "b") ",") ;; → "a,b" +(string-trim " hi ") ;; → "hi" +(string-prefix? "he" "hello") ;; → #t +(string-suffix? "lo" "hello") ;; → #t +(string-contains "hello" "ell") ;; → 1 (index, not boolean!) +(string-empty? "") ;; → #t +(str "age: " 42 "!") ;; → "age: 42!" (auto-coerce) +``` + +#### Lists +```scheme +(flatten '(1 (2 (3)))) ;; → (1 2 3) +(unique '(1 2 2 3)) ;; → (1 2 3) +(take lst n) (drop lst n) (take-last lst n) (drop-last lst n) +(every pred lst) (any pred lst) (filter-map f lst) +(group-by f lst) (zip lst1 lst2) (frequencies lst) +(partition pred lst) (interleave l1 l2) (mapcat f lst) +(distinct lst) (keep f lst) (split-at lst n) +(append-map f lst) (snoc lst elem) +``` + +#### Functional Combinators +```scheme +(compose f g) (comp f g) ;; (f (g x)) +(partial f arg ...) ;; partial application +(complement pred) (negate pred) ;; logical not +(identity x) (constantly v) ;; basic combinators +(curry f arg) (flip f) ;; currying, arg swap +(conjoin p1 p2) (disjoin p1 p2) ;; predicate AND/OR +(juxt f g) ;; → (lambda (x) (list (f x) (g x))) +(cut f <> y) ;; SRFI-26 partial: (lambda (x) (f x y)) +``` + +#### JSON +```scheme +(string->json-object "{\"key\":\"val\"}") ;; → hash table +(json-object->string ht) ;; → JSON string +(read-json port) (write-json obj port) +``` + +#### CSV +```scheme +(csv->alists "name,age\nAlice,30") ;; → (((name . "Alice") (age . "30"))) +(read-csv-file "data.csv") ;; → list of row lists +(write-csv-file "out.csv" rows) +``` + +#### DateTime +```scheme +(datetime-now) (datetime-utc-now) +(make-datetime 2026 3 27 12 0 0) +(parse-datetime "2026-03-27T12:00:00Z") +(datetime->iso8601 dt) (datetime->epoch dt) +(datetime-add dt duration) (datetime-diff dt1 dt2) +(datetime<? dt1 dt2) (day-of-week dt) (leap-year? 2024) +``` + +#### Paths +```scheme +(path-join "/home" "user" "f.txt") ;; → "/home/user/f.txt" +(path-directory "/a/b/f.txt") ;; → "/a/b" +(path-extension "file.txt") ;; → "txt" +(path-absolute? "/home") ;; → #t +``` + +#### File I/O +```scheme +(read-file-string "f.txt") ;; → entire file +(read-file-lines "f.txt") ;; → list of lines +(write-file-string "f.txt" "data") +``` + +#### Pretty Printing +```scheme +(pp expr) (pp-to-string expr) (pprint expr) +``` + +#### Formatting +```scheme +(format "~a is ~a" "Alice" 30) ;; → "Alice is 30" +(printf "x = ~a\n" 42) +(displayln "hello" " " "world") +``` + +#### Anaphoric / Conditional Binding +```scheme +(awhen (find x) (use it)) ;; binds result to `it` +(aif (find x) (use it) (default)) +(when-let (x (find y)) (use x)) +(if-let (x (find y)) (use x) (default)) +``` + +#### Loops +```scheme +(while test body ...) +(until test body ...) +(dotimes (i 10) body ...) ;; 0..9 +``` + +#### Misc Sugar +```scheme +(assert! (> x 0)) (assert! (> x 0) "message") +(alist (name "Alice") (age 30)) ;; → ((name . "Alice") (age . 30)) +(let-alist data (name age) body) +``` + +### What's NOT in the Prelude (requires separate import) + +```scheme +(import (std net request)) ;; HTTP client +(import (std net httpd)) ;; HTTP server +(import (std db sqlite)) ;; SQLite +(import (std actor)) ;; Actor system +(import (std async)) ;; Async/await +(import (std crypto digest)) ;; SHA, MD5 +(import (std text regex)) ;; Regex +(import (std text xml)) ;; XML +(import (std text yaml)) ;; YAML +(import (std os env)) ;; Environment variables +(import (std os signal)) ;; Signal handling +(import (std security sandbox)) ;; Sandboxing +``` + +### Chez Scheme Conflicts (handled by prelude) + +The prelude shadows these Chez builtins with Jerboa versions: +`make-hash-table`, `hash-table?`, `sort`, `sort!`, `printf`, `fprintf`, +`path-extension`, `path-absolute?`, `with-input-from-string`, +`with-output-to-string`, `iota`, `1+`, `1-`, `partition`, +`make-date`, `make-time` + +All standard Chez Scheme is still available — the prelude just re-exports +improved versions of the above. + +--- + +## Repository Boundaries + +When working in a Jerboa project, **ONLY modify files in the current repo** unless the user explicitly names another path. + +Common sibling repos that exist but must NOT be touched without explicit instruction: +- `~/mine/jerboa-emacs` — **NEVER touch**. Another model owns it. +- `~/mine/jerboa-mcp` — Legacy node MCP, superseded. The active MCP server now lives in THIS repo at `mcp/` + `data/`. Don't modify the legacy repo unless told. +- `~/mine/jerboa-shell` — Only modify when user explicitly says to work there. +- `~/mine/gerbil-mcp` — **NEVER touch**. Deprecated. +- `~/mine/gerbil-orig` — Read-only reference for upstream Gerbil. Never modify. + +If a user instruction mentions a file path, use EXACTLY that path. Do not substitute a similar-looking path from another repo. + +--- + +## Build & Verification + +After modifying any `.ss` or `.sls` (Jerboa Scheme) source files, always run the build command and fix any errors before moving on. Common issues include: missing imports, wrong function names, and duplicate definitions. + +### Stale Artifacts + +If edits seem to have no effect after `make build`, delete stale compiled files: +```bash +find lib -name "*.so" -delete && find lib -name "*.wpo" -delete && make build +``` +Run `jerboa_stale_static` to detect stale `.so` files before debugging "why doesn't my edit work?". + +## Pre-commit Requirements + +**ALWAYS** run a clean build **before** committing any code to this repository. Pick the right target for the *current* platform: + +- **Linux**: run `make docker-build` — the Docker image must build cleanly against the full musl-static release pipeline. +- **macOS / FreeBSD / other**: run `make binary` — the native local build must succeed. Do **not** run `make docker-build` here; Docker on non-Linux hosts is slow and not the canonical pipeline for those platforms. + +Do not commit if the build fails. + +## Act First, Read Less + +When making changes, read only what you need to make the edit, then make it. +Do not read more than 3 files before acting. Do not re-read files you already +read. Do not verify things you already know. If you have enough context to make +a change, make it. The user will interrupt you if you are wrong. + +## Jerboa MCP Tools — MANDATORY Usage + +Jerboa is a niche Scheme dialect with limited training data. **Never guess — always verify** with MCP tools. Tool descriptions are available at runtime via the MCP server; this section covers **when** and **why** to use each tool. + +### MANDATORY Workflow Order (for writing new code) + +1. **`jerboa_howto`** — BEFORE writing code, search cookbook for verified patterns +2. **`jerboa_module_exports`** / **`jerboa_function_signature`** — confirm APIs exist and check arities +3. Write the code +4. **`jerboa_verify`** — combined syntax + compile + lint + arity + duplicate check (use instead of individual tools) +5. **`jerboa_security_scan`** — for code involving FFI, shell commands, or file I/O + +### Essential Tools (use proactively) + +| When | Tool | +|---|---| +| Before writing ANY Jerboa code | `jerboa_howto` — cookbook has verified patterns with correct imports | +| Check what a module exports | `jerboa_module_exports` — never guess function names | +| Check function arity/args | `jerboa_function_signature` — prevents wrong-arg-count errors | +| Validate your code | `jerboa_verify` — one-stop syntax+compile+lint+arity check | +| Test expressions interactively | `jerboa_eval` — use `imports` param for modules, `env` for FFI paths | +| Persistent interactive testing | `jerboa_repl_session` — maintains state across evaluations | +| Debug an error message | `jerboa_explain_error` + `jerboa_error_fix_lookup` | +| Understand unfamiliar code | `jerboa_file_summary` + `jerboa_document_symbols` | +| Find where something is defined | `jerboa_find_definition` — source file, module, kind, arity | +| Search for symbol by substring | `jerboa_apropos` or `jerboa_smart_complete` | +| Build the project | `jerboa_build_and_report` or `jerboa_make` — prefer over bash `make` | +| Run tests | `jerboa_run_tests` — prefer over ad hoc shell test invocations | +| Check for stale .so artifacts | `jerboa_stale_static` — common cause of "edit has no effect" | +| Macro expansion | `jerboa_expand_macro` / `jerboa_trace_macro` | +| Inspect struct/class types | `jerboa_class_info` — fields, inheritance, constructor signature | +| FFI work | `jerboa_ffi_scaffold` / `jerboa_ffi_type_check` / `jerboa_ffi_null_safety` | +| Port Gerbil code | `jerboa_migration_check` + `jerboa_translate_scheme` | +| Detect paren imbalance | `jerboa_check_balance` — use BEFORE `make build` after deep edits | +| Full project audit | `jerboa_project_health_check` — balance, exports, cycles, duplicates | +| Security audit | `jerboa_security_audit` + `jerboa_import_policy_check` | +| Static build audit | `jerboa_static_symbol_audit` + `jerboa_boot_library_audit` | +| Explore a module | `jerboa_module_catalog` (replaces multiple `jerboa_doc` calls) | +| Look up any symbol | `jerboa_doc` — type, arity, qualified name, related symbols | +| Read stdlib source | `jerboa_stdlib_source` — see internal implementations | + +### Cookbook & Knowledge Management + +- **`jerboa_howto`** / **`jerboa_howto_get`**: Search and retrieve verified recipes +- **`jerboa_howto_add`**: Save new patterns to cookbook (MANDATORY when you discover something non-trivial) +- **`jerboa_howto_run`** / **`jerboa_howto_verify`**: Validate recipes still work +- **`jerboa_error_fix_add`**: Save error→fix mappings for common mistakes +- **`jerboa_anti_pattern_lookup`**: Search reusable local-model mistakes and failed strategies + +**The knowledge base is `data/*.sexp` in THIS repo**, embedded into `jmcp` at build time. The write tools above edit it live — the server reads `data/` from disk first, with the embedded copy as fallback (`JERBOA_MCP_REPO` points every client at this repo). When you add a stdlib/language feature, also update `data/` (a cookbook recipe + `api-signatures.sexp` + `changelog.sexp`) and **commit it**. Run `make jmcp` (or `make jmcp-portable`) only to refresh the embedded copy shipped in portable binaries. + +### Code Generation & Refactoring + +`jerboa_rename_symbol`, `jerboa_balanced_replace`, `jerboa_wrap_form`, `jerboa_splice_form`, `jerboa_scaffold_test`, `jerboa_generate_module`, `jerboa_translate_scheme`, `jerboa_project_template`, `jerboa_httpd_handler_scaffold`, `jerboa_db_pattern_scaffold`, `jerboa_actor_ensemble_scaffold` + +### Feature Suggestions + +- **`jerboa_list_features`** / **`jerboa_suggest_feature`** / **`jerboa_vote_feature`**: Track and submit tooling improvement ideas + +--- + +## MANDATORY: Save What You Learn + +Jerboa is niche — every non-trivial pattern you discover prevents future sessions from re-discovering it. + +### Save to cookbook (`jerboa_howto_add`) whenever you: +- Discover a working pattern through `jerboa_eval` or trial-and-error +- Figure out correct imports, arities, or calling conventions that weren't obvious +- Find a workaround for a Jerboa quirk or undocumented behavior + +**Before saving**: check `jerboa_howto` to avoid duplicates. **Do NOT save**: trivial one-liners, project-specific logic, or existing recipes. + +**Recipe format**: `id` (kebab-case), `tags` (4-6 search keywords incl. module name), `imports` (all required), `code` (complete working example), `notes` (gotchas/alternatives). + +### Save anti-patterns (`data/anti-patterns.sexp`) whenever you: +- See a plausible local-model strategy that failed verification +- Find a weak verifier pattern that allowed false success +- See a repeated repair loop, such as broad-reading after a concrete error +- Find a generic runtime mistake, such as missing lower-bound checks before vector access + +**Before saving**: check `jerboa_anti_pattern_lookup` to avoid duplicates. If none exists, call `jerboa_anti_pattern_add`; only edit `data/anti-patterns.sexp` directly if the writer tool is unavailable. Save the normalized reusable mistake, not the whole trace or benchmark name. + +**Anti-pattern format**: `id`, `title`, `kinds`, `severity`, `tags`, `pattern`, `avoid`, `advice`, `tools`. + +### Save error fixes (`jerboa_error_fix_add`) whenever you: +- See exact compiler/runtime/verifier text with a repeatable repair +- Hit an error that `jerboa_failure_advisor` should classify better next time +- Debug a local-model generated-code failure where a short diagnosis prevents another failed iteration + +**Before saving**: check `jerboa_error_fix_lookup` with the exact error text. **Do NOT save**: one-off project business-logic mistakes. + +**Error-fix format**: `id`, `pattern`, `fix`; optional `type`, `explanation`, `code_example`. + +### Suggest tooling improvements (`jerboa_suggest_feature`) whenever you: +- Make multiple sequential tool calls that could be one tool +- Fall back to bash because an MCP tool is missing or insufficient + +**Before suggesting**: check `jerboa_list_features`; vote with `jerboa_vote_feature` if it already exists. + +### Save Discoveries Mechanisms +- **`/save-discoveries` skill**: invoke anytime to review session and save recipes, anti-patterns, error fixes, feature suggestions, and security patterns +- **PreCompact hook**: add `PreCompact` hook with `type: "prompt"` in `.claude/settings.json` to auto-save before context compaction + +--- + +## Common Workflows + +- **Write new code**: `jerboa_howto` -> `jerboa_module_exports` -> write code -> `jerboa_verify` -> `jerboa_security_scan` +- **Debug an error**: `jerboa_explain_error` -> follow suggested tools -> `jerboa_howto` for fix patterns +- **Understand unfamiliar code**: `jerboa_file_summary` -> `jerboa_document_symbols` -> `jerboa_module_deps` +- **Refactor a module**: `jerboa_check_exports` -> `jerboa_find_callers` -> `jerboa_rename_symbol` -> `jerboa_check_import_conflicts` +- **Build project**: `jerboa_build_conflict_check` -> `jerboa_make` -> `jerboa_build_and_report` +- **Port from Gerbil**: `jerboa_migration_check` -> `jerboa_translate_scheme` -> `jerboa_verify` -> `jerboa_check_syntax` +- **Audit project quality**: `jerboa_verify` -> `jerboa_lint` -> `jerboa_dead_code` -> `jerboa_dependency_cycles` +- **Debug a crash**: `jerboa_stale_static` -> `jerboa_bisect_crash` -> `jerboa_ffi_type_check` +- **Learn a module**: `jerboa_stdlib_source` -> `jerboa_module_catalog` -> `jerboa_module_quickstart` +- **Security audit**: `jerboa_security_audit` -> `jerboa_import_policy_check` -> `jerboa_unsafe_import_lint` +- **Static build audit**: `jerboa_static_symbol_audit` -> `jerboa_boot_library_audit` -> `jerboa_rust_musl_build` +- **Safe-by-default check**: `jerboa_safe_prelude_check` -> `jerboa_resource_leak_check` -> `jerboa_safe_prelude_generate` +- **Debug editor command**: `jerboa_command_trace` with `project_path` and `buffer_type` + +## Workflow Conventions + +When implementing new features, always complete the documentation update in the same session. Document non-trivial solutions as howto recipes in the cookbook system. new file mode 100644 --- /dev/null +++ b/Makefile @@ -0,0 +1,69 @@ +JERBUILD ?= $(if $(wildcard ../jerboa/dist/jerbuild),$(abspath ../jerboa/dist/jerbuild),jerbuild) +JERBOA ?= jerboa +JERBOA_HOME := $(shell "$(JERBUILD)" --jerboa-home 2>/dev/null) +ifeq ($(JERBOA_HOME),) +$(error jerbuild not found on PATH or --jerboa-home failed) +endif + +ROOT := $(abspath .) +LIBDIRS := $(ROOT):$(JERBOA_HOME)/lib +DUCKDB_SOURCE ?= /Users/user/duckdb +UPSTREAM_COMMIT := 117e1a46be1c903c5a36ee3c881c125597f93c60# gitsafe:ignore +GENERATOR := support/generate/upstream-contracts.ss +INVENTORY := contracts/generated/upstream-inventory.sexp +UNIT_TESTS := $(wildcard test/unit/*.ss) + +.PHONY: all check-upstream contracts contracts-check unit test check-docs verify clean help +.DEFAULT_GOAL := help + +all: verify + +check-upstream: + @test -d "$(DUCKDB_SOURCE)/.git" || { echo "missing DuckDB checkout: $(DUCKDB_SOURCE)"; exit 1; } + @test "$$(git -C "$(DUCKDB_SOURCE)" rev-parse HEAD)" = "$(UPSTREAM_COMMIT)" || { \ + echo "DuckDB checkout is not pinned to $(UPSTREAM_COMMIT)"; exit 1; } + @git -C "$(DUCKDB_SOURCE)" diff --quiet && git -C "$(DUCKDB_SOURCE)" diff --cached --quiet || { \ + echo "DuckDB checkout must be clean before contract generation"; exit 1; } + @test "$$(git -C "$(DUCKDB_SOURCE)" ls-files | wc -l | tr -d ' ')" = "15476" || { \ + echo "DuckDB tracked-file count differs from pinned baseline"; exit 1; } + +contracts: check-upstream + @mkdir -p contracts/generated + "$(JERBOA)" "$(GENERATOR)" "$(DUCKDB_SOURCE)" "$(INVENTORY)" + +contracts-check: check-upstream + @tmp=$$(mktemp); trap 'rm -f "$$tmp"' EXIT INT TERM; \ + "$(JERBOA)" "$(GENERATOR)" "$(DUCKDB_SOURCE)" "$$tmp"; \ + cmp -s "$$tmp" "$(INVENTORY)" || { \ + echo "$(INVENTORY) is stale; run make contracts"; \ + diff -u "$(INVENTORY)" "$$tmp" || true; exit 1; } + +unit: + @fail=0; for test_file in $(UNIT_TESTS); do \ + echo "== unit: $$test_file =="; \ + "$(JERBOA)" "$$test_file" "$(INVENTORY)" || fail=1; \ + done; test $$fail -eq 0 || { echo "UNIT FAILURES"; exit 1; } + +test: unit + +check-docs: + @test -s Project.md + @test -s README.md + @! LC_ALL=C grep -n '[[:blank:]]$$' Project.md README.md + @test "$$(grep -c '^# ' Project.md)" -eq 1 + @test "$$(grep -c '^# ' README.md)" -eq 1 + @echo "documentation hygiene: ok" + +verify: contracts-check unit check-docs + +clean: + find . \( -name '*.so' -o -name '*.wpo' \) -delete 2>/dev/null || true + +help: + @echo "jerboa-duckdb Phase 0" + @echo "" + @echo " make contracts Regenerate the pinned upstream inventory" + @echo " make contracts-check Verify the generated inventory is current" + @echo " make unit Run current Jerboa unit tests" + @echo " make check-docs Check Markdown hygiene" + @echo " make verify Run all current acceptance gates" new file mode 100644 --- /dev/null +++ b/Project.md @@ -0,0 +1,1089 @@ +# Jerboa DuckDB Equivalence Project + +## Document status + +- Status: implementation handoff; no implementation is claimed complete. +- Target repository: `/Users/user/mine/jerboa-duckdb` +- Upstream reference checkout: `/Users/user/duckdb` +- Upstream commit: `117e1a46be1c903c5a36ee3c881c125597f93c60` +- Upstream description: `v1.5.4-9213-g117e1a46be` +- Snapshot date: 2026-07-19 +- Upstream license: MIT, copyright 2018-2026 Stichting DuckDB Foundation. +- Primary objective: implement a DuckDB-compatible analytical database in Jerboa, with DuckDB used as a development oracle but not linked into the final engine. + +This file is the project contract. Update it whenever scope, architecture, generated +contracts, milestone status, compatibility exceptions, or release evidence changes. +Checkboxes stay unchecked until the named acceptance evidence exists in the target +repository. + +## Executive directive + +"Full equivalence" is not satisfied by accepting similar SQL. The completed product +must reproduce the observable behavior exposed by this pinned DuckDB source tree: + +1. SQL syntax, binding, types, casts, scalar/aggregate/window/table functions, result + values, NULL behavior, ordering, and errors. +2. Catalog, DDL, DML, transactions, prepared statements, settings, pragmas, system + tables, profiling, logging, secrets, and attached database behavior. +3. Vectorized and parallel execution, spill behavior, interruption, streaming results, + resource limits, and optimizer controls. +4. Durable storage, WAL recovery, MVCC, checkpoints, indexes, compression, encryption, + and supported historical storage versions. +5. CSV, JSON, Parquet, Arrow, multi-file, glob, import/export, and extension behavior. +6. Public C ABI, extension ABI, Arrow interfaces, C++ compatibility shim, shell, and the + client packages present in this repository. +7. Build, test, fuzz, benchmark, platform, packaging, and security behavior. + +The implementation may use reviewed C or Rust kernels where Jerboa cannot express the +required ABI, SIMD, compression, atomics, OS I/O, or numeric operation efficiently. +Jerboa remains the owner of parsing, binding, catalogs, planning, orchestration, +transactions, lifecycle policy, and public semantic behavior. Native code must not +quietly become a second database implementation. + +Using `libduckdb` behind a Jerboa wrapper is allowed only for the differential oracle +and temporary bootstrap tools. It is not an acceptable production implementation. +Likewise, `(std db sqlite)` may help prototype a test harness, but SQLite cannot back +production storage or execution because its types, concurrency, NULL, optimizer, and +file behavior are not DuckDB-equivalent. + +## Baseline inventory + +The pinned checkout contains 15,476 tracked files. The main implementation has 3,292 +files under `src`, including 2,386 C++ translation units and 2,615 C/C++ headers across +the repository. The highest-value machine-readable contracts are: + +| Contract | Baseline size | Source of truth | +|---|---:|---| +| Statement kinds | 33 | `src/include/duckdb/common/enums/statement_type.hpp` | +| Logical operator kinds | 66 | `src/include/duckdb/common/enums/logical_operator_type.hpp` | +| Physical operator kinds | 86 | `src/include/duckdb/common/enums/physical_operator_type.hpp` | +| Logical type IDs | 54 | `src/include/duckdb/common/types.hpp` | +| Vector representations | 6 | `src/include/duckdb/common/enums/vector_type.hpp` | +| Optimizer passes | 42 plus invalid sentinel | `src/include/duckdb/common/enums/optimizer_type.hpp` | +| Generated setting definitions | 171 | `src/include/duckdb/main/settings.hpp` | +| C API functions | 548 unique symbols | `src/include/duckdb.h` and C API JSON manifests | +| C API manifest groups | 42 | `src/include/duckdb/main/capi/header_generation/functions` | +| Function manifests | 38 files, 452 top-level entries | `src/function/**/functions.json`, `extension/core_functions/**/functions.json` | +| SQLLogicTest files | 5,207 | `test/**/*.test`, `test/**/*.test_slow` | +| Optimizer implementation files | 127 C++ files | `src/optimizer` | +| Execution implementation files | 215 C++ files | `src/execution` | +| Storage implementation files | 134 C++ files | `src/storage` | + +These counts are snapshot checks, not estimates. A generated contract update must fail +CI if the upstream counts change without reviewed generated diffs and a baseline bump. + +## Definition of equivalence + +Track compatibility in separate dimensions. A feature is only "done" when every +applicable dimension passes. + +### E1: SQL surface equivalence + +- The same input is accepted or rejected. +- Accepted input has the same resolved schema, logical types, values, row count, and + deterministic row order. +- Floating point comparison follows the upstream test runner's tolerances and special + handling for NaN, infinities, and signed zero. +- Errors match category, SQLSTATE where exposed, statement state, and upstream regex or + stable message fragment. Do not weaken tests to "any error". +- Prepared and direct execution agree, including parameter inference and rebinding. +- `EXPLAIN` output and profiling JSON have stable compatibility tests. Exact physical + plan identity may be waived only when results and the performance gate pass, and every + waiver is recorded. + +### E2: API equivalence + +- C enum values, type layouts, symbol names, signatures, ownership, NULL sentinels, + callback lifetime, and thread behavior match `duckdb.h`. +- The extension API version and function table layout are ABI checked. +- Arrow C Data and C Stream interfaces round-trip without copies where DuckDB promises + that behavior. +- C++ source compatibility is supplied by a narrow C++ header/library shim over the C + ABI; do not expose Scheme internals to C++ callers. + +### E3: storage equivalence + +- Jerboa DuckDB can open, query, update where supported, checkpoint, close, and reopen + DuckDB database files for every claimed storage version. +- DuckDB can open files written by Jerboa DuckDB at the same compatibility version. +- WAL replay and failure atomicity match under process kill, torn/truncated WAL, I/O + errors, out-of-memory, and failed commit/checkpoint injection. +- Block checksums, encryption metadata, catalog serialization, statistics, row groups, + indexes, and compression streams are byte-compatible where the format requires it. + +### E4: extension equivalence + +- Install/load/autoload, repository selection, platform tags, signatures, metadata, + settings, secrets, filesystem registration, and extension callbacks match. +- Both statically linked and dynamically loaded extension paths are exercised. +- A compatible C extension API is required before claiming third-party extension + compatibility. Reimplementing only bundled SQL functions is not enough. + +### E5: operational equivalence + +- Interrupt, progress, pending results, streaming, task scheduling, query timeout, + memory limits, spilling, temp cleanup, logging, profiling, and concurrent connection + behavior are tested. +- Release artifacts work on the supported target matrix and do not depend on a source + checkout or ambient shared objects. + +### E6: performance equivalence + +- Correctness always takes precedence over a faster wrong answer. +- Every benchmark records DuckDB and Jerboa DuckDB versions, build flags, hardware, + dataset checksum, cold/warm state, wall time, CPU time, peak RSS, allocations, and I/O. +- Full release signoff requires an agreed performance envelope for scan, filter, + projection, group-by, join, sort/window, ingest, checkpoint, CSV, and Parquet suites. +- Until an envelope is approved, publish ratios instead of calling performance parity + complete. Independently gate regressions against the last Jerboa DuckDB release. + +## Compatibility ledger + +Create `contracts/compatibility.sexp` as the machine-readable ledger. Each row needs: + +- stable feature ID; +- upstream file and symbol/test references; +- E1-E6 applicability; +- implementation module; +- focused tests; +- status: `missing`, `partial`, `compatible`, or `waived`; +- upstream commit last checked; +- waiver owner, reason, expiry, and replacement plan. + +Generate the human-readable matrix at `docs/compatibility.md`. Never hand-maintain two +independent lists. CI must reject `compatible` rows with missing tests or stale source +references. + +## Architecture + +### Query path + +```text +SQL text + -> tokenizer and generated PEG/packrat parser + -> parsed statement/expression/table-reference records + -> transformer and normalizer + -> binder, catalog lookup, overload/cast resolution + -> typed bound expressions and logical operator tree + -> rule, statistics, and cost optimizers + -> physical plan and pipeline builder + -> bounded task scheduler + -> vectorized operators and native kernels + -> streaming or materialized result +``` + +### Storage path + +```text +catalog transaction + table transaction + -> local row groups / updates / delete vectors / index changes + -> undo and WAL records + -> commit visibility publication + -> checkpoint writer + -> metadata, row groups, column segments, compression blocks + -> single-file block manager and buffer pool +``` + +### Ownership boundaries + +- Jerboa records own semantic objects and immutable plan nodes. +- Query-local mutable state is scoped to an executor or pipeline task. +- Database-global mutable state is behind explicit locks or actor-owned mailboxes. +- Native buffers and handles are owned by a native registry using opaque generation IDs, + never forgeable raw pointer integers. +- Every resource has deterministic `close`/`destroy` and unwind cleanup. Guardians are + a leak safety net only. +- A close operation rejects new FFI leases, waits for active calls, then destroys the + native resource exactly once. +- Blocking native I/O never occupies an actor scheduler worker without an explicit + blocking pool and cancellation path. + +### Verified Jerboa capability map + +The following modules and exports were checked through the Jerboa MCP against the local +toolchain on the snapshot date. Recheck them when upgrading Jerboa. + +| Need | Verified Jerboa facility | Intended database use | +|---|---|---| +| General language/runtime | `(jerboa prelude)` | Records/structs, matching, vectors, hash tables, results, resource guards, binary ports, paths, CSV/JSON tooling | +| Safe FFI declarations | `(jerboa ffi)` | `c-lambda`, `define-c-lambda`, `c-declare`, `begin-ffi`, type mapping, controlled shared-object loading | +| Native library policy | `(std native-loader)` | Canonical library/directory validation, clean environment checks, controlled system/libc symbols | +| CPU service scheduling | `(std actor)` | Fixed worker scheduler, supervision, registry, CPU count, task submission; not per-row actors | +| Async orchestration | `(std async)` | `run-async/workers`, async tasks/promises, sleeps, and async channel get/put | +| Backpressure | `(std misc channel)` | Bounded channels, try operations, select, close, queue depth | +| Local queues | `(std misc queue)` | Executor-ready queues and nonconcurrent local work lists | +| Structured cancellation | `(std concur structured)` | `with-task-scope`, scoped spawn, await, cancel, parallel, and race | +| Timing/allocation profiling | `(std profile)` | `with-profile`, `profile-stats`, `allocation-count`, `with-timing`, and `time-it` | +| Prototype/reference DB only | `(std db sqlite)` | Harness experiments only; its export introspection currently needs the optional `jsqlite` dependency | + +Start implementation work by retrieving the relevant verified cookbook recipes rather +than reconstructing their edge cases: + +- `columnar-typed-columns-fxvector-flvector` for unboxed homogeneous columns; +- `jerboa-structured-concurrency` for scoped worker cancellation; +- `jerboa-profile` for wall/CPU/allocation evidence; +- `binary-file-content-addressed-store` and `checked-bytevector-length-parser` for + resource-safe binary I/O and bounded decoding; +- `ffi-synchronized-handle-active-lease` and `guardian-reaped-ffi-handles` for native + lifetime safety; +- `standalone-binary-register-ffi-symbols` for WPO/static binaries; +- `jerboa-cdylib-opaque-handle-wrapper` for native module boundaries; +- `typed-llvmir-whole-program-strings-records` only for the experimental restrictions + described in Workstream 15. + +## Proposed repository layout + +```text +AGENTS.md +Project.md +Makefile +contracts/ + upstream-baseline.sexp + compatibility.sexp + generated/ +docs/ + compatibility.md + architecture/ +lib/jerboa/duckdb/ + api/ + parser/ + planner/ + optimizer/ + execution/ + storage/ + catalog/ + transaction/ + parallel/ + function/ + extension/ + common/ +support/ + generate/ + oracle/ + abi/ +native/ + include/ + src/ +test/ + sqllogic/ + api/ + storage/ + recovery/ + differential/ + fuzz/ +benchmark/ +``` + +All user-facing Jerboa sources are `.ss`. Do not create hand-written `.sls` files. +Keep imports narrow outside ordinary application modules, especially for FFI and +threading modules, so WPO builds do not acquire ambiguous bindings. + +## Workstream 0: reproducible bootstrap and contract generation + +- [x] Add `contracts/upstream-baseline.sexp` with commit, describe string, dirty-state + requirement, source path override, counts above, and license metadata. +- [ ] Add generators that parse structured JSON and C/C++ enums; do not scrape C++ with + fragile line substitutions where a JSON manifest or compiler AST is available. +- [ ] Generate statement, expression, table-ref, query-node, logical operator, physical + operator, type, vector, compression, optimizer, catalog, relation, setting, metric, + WAL, and C API enumerations. +- [ ] Import the 38 function manifests, aliases, variants, examples, categories, and + generated registration order. +- [ ] Import all 42 stable C API manifest groups and the unstable v1 API manifests. +- [ ] Generate a complete test manifest with tags, required extensions, environment, + storage versions, and slow-test status. +- [x] Add `make contracts`, `make contracts-check`, and a diff report that names newly + added, removed, or renumbered upstream contracts. +- [ ] Preserve upstream MIT notices in generated or substantially derived files. +- [x] Make all generation deterministic and verify a second run is byte-identical. + +Acceptance: a clean target checkout regenerates identical contracts from the pinned +DuckDB checkout; changing the upstream commit causes a reviewed, categorized diff. + +## Workstream 1: SQLLogicTest runner and differential oracle + +Build this before the engine so every later feature lands behind executable evidence. + +- [ ] Parse `statement ok/error/maybe`, `query`, expected rows, regex errors, hashes, + named results, sort modes, labels, skip/only conditions, and result types. +- [ ] Implement `loop`, `foreach`, `concurrentloop`, `concurrentforeach`, `endloop`, + `continue`, loop variable substitution, named connections, reconnect/restart, load, + mode, test environment, `require`, and `require-env`. +- [ ] Match DuckDB result normalization for NULL, empty strings, floats, booleans, + blobs, dates/times, nested values, and row ordering. +- [ ] Emit structured per-test and per-statement events, duration, seed, configuration, + skip reason, and minimized failure artifact. +- [ ] Run each eligible case against the pinned DuckDB binary and Jerboa DuckDB in + isolated directories, compare structured outcomes, and retain both logs. +- [ ] Support expected divergence only through expiring ledger waivers. +- [ ] Add delta debugging for SQL text and input rows without changing the failure. +- [ ] Partition and shard deterministically for local and CI parallelism. + +Acceptance: the runner self-tests against DuckDB's runner semantics and can execute a +representative file from every `test/sql` top-level group before the new engine exists. + +## Workstream 2: common types, values, vectors, and serialization + +Implement all 54 logical IDs, including binder-only and internal types, not just common +SQL values: + +- scalar/null: SQLNULL, UNKNOWN, ANY, UNBOUND, TEMPLATE, TYPE; +- boolean and signed/unsigned integers through 128 bits; +- BIGNUM, DECIMAL, FLOAT, DOUBLE; +- DATE, TIME/TIME_NS/TIME_TZ, timestamps at second/millisecond/microsecond/nanosecond + precision with and without timezone, and INTERVAL; +- CHAR, VARCHAR, BLOB, BIT, UUID, POINTER, VALIDITY, GEOMETRY; +- STRUCT, TUPLE, LIST, fixed ARRAY, MAP, ENUM, UNION, VARIANT, TABLE, LAMBDA, and + aggregate state. + +Required representation work: + +- [ ] Define logical type metadata, equality, aliasing, child types, decimal width/scale, + enum dictionaries, array size, collation, extension type info, and serialization. +- [ ] Implement values, strict conversions, hashing, comparison, SQL rendering, nested + access, min/max/sentinel values, and round-trips. +- [ ] Implement validity bitmaps, selection vectors, unified vector format, and the six + vector representations: flat, FSST, constant, dictionary, sequence, and shredded. +- [ ] Implement `DataChunk`, cardinality, slicing, referencing, flattening, ownership, + verification, hashing, copying, and Arrow conversion. +- [ ] Use `fxvector`/`flvector` only where their numeric ranges exactly match the SQL + physical type. A Chez fixnum is not a general signed 64-bit container. +- [ ] Use packed bytevectors or aligned native-owned buffers for full-width integers, + decimals, 128-bit values, validity words, offsets, and SIMD kernels. +- [ ] Store strings/blobs as offsets plus length and an owned heap; specify inline-string + rules and lifetime before optimizing them. +- [ ] Bounds-check every offset/length before bytevector or native access and reject + trailing data where a frame requires exact consumption. +- [ ] Add checked arithmetic for buffer sizes, row counts, offsets, and nested lengths. + +Performance rules: avoid per-row records and generic method dispatch in hot loops. +Compile a bound expression once to a type-specialized vector procedure. Use boxed Jerboa +objects at control boundaries, not for every fixed-width cell. + +Acceptance: type/cast/value/vector tests pass under default and 512-row vector sizes, +with randomized slice/dictionary/constant/NULL composition and Arrow round-trips. + +## Workstream 3: parser, AST, formatter, and autocomplete + +The upstream parser is PEG/packrat based. Treat its grammar and transformer as one +versioned unit. + +- [ ] Generate Jerboa grammar tables from `src/parser/peg/*.gram` and + `src/parser/peg/inlined_grammar.gram`. +- [ ] Implement a memoized packrat runtime with bounded memory, source spans, error + context, cancellation, and configurable maximum expression depth. +- [ ] Port tokenizer rules, keywords, quoted identifiers, strings, dollar quoting, + comments, numeric literals, parameters, Unicode, and invalid UTF-8 behavior. +- [ ] Model every parsed expression class, result modifier, table reference, query node, + constraint, column definition, create/alter info record, and statement node. +- [ ] Implement all statement kinds: SELECT, INSERT, UPDATE, CREATE, DELETE, PREPARE, + EXECUTE, ALTER, TRANSACTION, COPY, ANALYZE, variable SET, CREATE FUNCTION, EXPLAIN, + DROP, EXPORT, PRAGMA, VACUUM, CALL, SET, LOAD, RELATION, EXTENSION, LOGICAL PLAN, + ATTACH, DETACH, MULTI, COPY DATABASE, UPDATE EXTENSIONS, MERGE INTO, CONNECT, and + DISCONNECT. +- [ ] Implement CTEs including recursive/materialized forms; pivot/unpivot; qualify; + grouping sets, cube, rollup; samples; windows; lambdas/list comprehensions; lateral, + positional, as-of, natural, semi, anti, mark, and dependent joins. +- [ ] Implement parser extension hooks, statement copy/equality/serialization, + query-to-string, SQL formatting, tokenization, and autocomplete catalog callbacks. +- [ ] Differentially fuzz parse success, AST serialization, `ToString` reparse, error + location, and stack/memory bounds. + +Acceptance: all parser and PEG parser test groups pass, statement round-trips are stable, +and malformed input fuzzing stays within configured time and memory. + +## Workstream 4: catalog, binder, and function resolution + +- [ ] Implement database/catalog/schema namespaces, `main`, `temp`, and `system`, search + path, identifier case preservation, temporary objects, OIDs, and catalog versions. +- [ ] Implement entries for tables, views, schemas, sequences, indexes, types, macros, + scalar/aggregate/table/copy/pragma/window functions, collations, secrets, triggers, + external resources, and extensions. +- [ ] Implement dependency tracking, create conflict modes, cascading/restricted drops, + view/macro dependencies, invalidation, and transactional catalog changes. +- [ ] Bind scopes, aliases, stars/COLUMNS expressions, USING keys, rowid/row number, + generated columns, defaults, constraints, correlated references, lateral references, + CTEs, subqueries, lambdas, grouping, windows, pivot, and RETURNING.