Document local mine repository authority
ober
146ba94271be94e5b39b9c7e7e205ddd7b59f85d
--- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ *.wpo dist/ tmp/ +vendor/ --- a/AGENTS.md +++ b/AGENTS.md @@ -1,3 +1,9 @@ +## STOP: Local Mine Repositories Are Authoritative + +Always commit development changes only in the matching `/Users/user/mine/jerboa-*` repository. Never commit development changes in `vendor/jerboa-*`; vendor checkouts are read-only mirrors for reference and dependency inspection only. + +Before starting work that compares against or uses a vendor checkout, update that vendor checkout to the latest revision and verify it matches exactly with the corresponding `/Users/user/mine/jerboa-*` repository state. If they differ, stop and sync from `/Users/user/mine/jerboa-*` before making or committing changes. The `/Users/user/mine/jerboa-*` worktree must always represent the latest authoritative version. + ## STOP: Editing `.ss`/`.sls` Files — Mandatory Rules These rules exist because local-model sessions have lost **hours** fighting new file mode 100644 --- /dev/null +++ b/FULL_COMPATIBILITY_PLAN.md @@ -0,0 +1,472 @@ +# Full DuckDB Compatibility Plan + +This plan describes how to remove the remaining compatibility gaps between +Jerboa DuckDB and the pinned DuckDB contract inventory. It extends the current +Phase 1 implementation documented in `docs/compatibility.md` and the broader +workstreams in `Project.md`. + +The target is behavioral and operational compatibility, not a line-for-line +port of DuckDB internals. Every claimed feature must have an executable +contract, an oracle comparison where applicable, and a resource/error test. + +## 1. Baseline and Target + +Current status is partial in all six compatibility dimensions: + +| Dimension | Current state | Full target | +|---|---|---| +| E1 SQL | Broad but bounded SQL subset | Full supported DuckDB SQL surface, including nested types, advanced joins, subqueries, windows, DML, pragmas, and extensions | +| E2 API | Jerboa Scheme API | Stable Scheme API plus C ABI, Arrow interfaces, extension ABI, and client-facing compatibility layers | +| E3 storage | Jerboa snapshots | Durable, recoverable, versioned storage with WAL/MVCC and documented DuckDB file compatibility policy | +| E4 extensions | In-process metadata registry | Native load/install, extension registration, ABI/version checks, secrets, and remote resources | +| E5 operations | Single-connection sequential execution | Concurrent connections, parallel execution, scheduling, cancellation, progress, streaming, spilling, and profiling | +| E6 performance | Focused in-memory benchmark | Reproducible benchmark suite across workloads, thread counts, vector sizes, cache states, and spill modes | + +The pinned upstream inventory remains the contract denominator. No percentage +or compatibility claim may exclude unsupported cases without listing the +exclusion and its reason. + +## 2. Execution Rules + +1. Preserve the current native Jerboa implementation boundary. DuckDB may be + used as an oracle and contract source, but production execution must not + link to `libduckdb`. +2. Land work in dependency order. A later phase may prototype against an + earlier phase, but it cannot be declared complete until the earlier phase's + gates pass. +3. Add the failing test or contract before, or in the same change as, the + implementation. Differential tests must include success, NULL, coercion, + error, cancellation, restart, and resource-limit cases where relevant. +4. Keep compatibility behavior behind explicit capability boundaries when a + feature cannot be implemented atomically. Unsupported behavior must return + a stable categorized error, never silently produce an approximation. +5. Record every intentional deviation in `contracts/compatibility.sexp` and + `docs/compatibility.md`, with an upstream reference and a removal issue. +6. Keep the upstream source reproducible through `vendor/duckdb` and the + pinned commit. Do not make build files depend on sibling checkouts. + +## 3. Phase A: Contract and Measurement Foundation + +### A1. Freeze the compatibility denominator + +Deliverables: + +- Generate and validate all upstream inventories: statements, logical and + physical operators, logical types, vectors, optimizers, compression, + settings, functions, C API, manifests, and SQLLogicTest files. +- Split the ledger into `implemented`, `partial`, `missing`, `waived`, and + `incompatible` rows with stable IDs. +- Add a machine-readable feature owner, dependency, test command, and removal + criterion to every non-compatible row. + +Gate: + +- A clean checkout fetches the pinned source, regenerates identical contracts, + and fails on drift. + +### A2. Build the conformance harness + +Deliverables: + +- A DuckDB oracle runner with version, commit, platform, settings, and schema + capture. +- Result comparison that handles NULL, numeric tolerances, timestamps, + decimals, blobs, nested values, error categories, SQLSTATE, and row order. +- Seeded randomized and metamorphic query generation. +- Structured JSONL artifacts for every statement, plan, error, timeout, and + resource measurement. + +Gate: + +- The harness can reproduce every existing differential failure from a saved + seed and classify it as engine, oracle, harness, or unsupported behavior. + +### A3. Establish safety budgets + +Deliverables: + +- Query text, AST depth, expression depth, row/column count, allocation, + memory, temporary storage, file descriptor, decompression, recursion, and + thread limits. +- Fail-closed behavior for malformed input, corrupt files, invalid extension + manifests, and exhausted resources. + +Gate: + +- Fuzz-smoke tests terminate within bounded time and memory and never leak a + resource after an error or cancellation. + +Current progress: the public API rejects SQL text over a conservative 1 MiB +character budget before tokenization, token streams whose structural nesting +exceeds a conservative depth of 256 before recursive parsing, and public +result materialization over a conservative 500,000 rows. The focused +differential harness also writes a replayable JSONL mismatch artifact when +`JERBOA_DUCKDB_DIFF_ARTIFACT` is set. The focused harness also compares coarse +parser, catalog, binder, conversion, and JSON error categories. Remaining A2/A3 +work includes seeded generation, broad resource budgets, timeout artifacts, and +SQLSTATE-aware error classification. + +## 4. Phase B: SQL and Type Completeness + +Prerequisite: Phase A. + +### B1. Complete the type system + +Implement and test: + +- Nested LIST, MAP, STRUCT, UNION, ARRAY, and VARIANT values and validity + propagation. +- Full DECIMAL precision/scale rules, HUGEINT/UUID, temporal time-zone + variants, collations, ENUM, user-defined types, and aliases. +- Type equality, hashing, comparison, serialization, casting, implicit casts, + assignment casts, constant folding, and NULL typing for every supported + type pair. +- Full vector representations, including FLAT, CONSTANT, DICTIONARY, + SEQUENCE, FSST/compressed, and SHREDDED/nested vectors. + +Current progress: aggregate window functions `count`, `sum`, `avg`, `min`, and +`max` now support explicit `ROWS BETWEEN` frames with partitioning, NULL-aware +behavior, `DISTINCT`, and `FILTER`. `RANGE`, `GROUPS`, named windows, and +streaming frame execution remain deferred. + +The planner now exposes deterministic structural fingerprints for bound query +plans. The fingerprint includes operator kinds, schemas, scan identity, and +bound expressions, and is covered by repeatability and plan-change tests. It +is a regression/cache key rather than a collision-free cryptographic identity; +full plan serialization and optimizer-pass fingerprints remain deferred. + +The optimizer now fuses adjacent logical LIMIT/OFFSET nodes into one equivalent +plan node. Broader limit pushdown, cardinality estimation, and optimizer passes +remain deferred. + +The public Scheme API now applies a conservative 1 MiB query-text budget before +tokenization and a structural AST-nesting depth budget of 256 before recursive +parsing. The limits and measured helpers are exported, and over-budget query, +prepare, and plan-fingerprint calls fail closed with a stable `Invalid Input +Error` category. Public query materialization also fails closed above 500,000 +rows and 1,024 columns without truncation. Byte-oriented limits, configurable +budgets, and broader resource governance remain deferred. + +Logical `LIMIT 0` now folds to the same typed empty plan used by impossible +literal filters, avoiding source execution while retaining the bound schema. +Broader limit pushdown and optimizer passes remain deferred. + +Literal `WHERE TRUE` predicates now fold away, while literal `WHERE FALSE` and +`WHERE NULL` predicates become typed empty plans before execution. General +expression folding and broader optimizer passes remain deferred. + +The storage slice now uses a versioned checksum envelope with generation +metadata, validates the temporary snapshot before rename, and recovers a valid +temporary snapshot when the committed file is missing or invalid. WAL, fsync +durability, MVCC, and DuckDB file-format compatibility remain deferred. + +The operations slice now exposes connection-scoped `duckdb-interrupt!` with +cooperative cancellation checks before query execution and at pull-operator +chunk boundaries. Preemptive thread interruption, scheduler integration, +progress reporting, resource limits, parallel execution, and streaming remain +deferred. + +CSV I/O now parses quoted records character-by-character, preserving embedded +newlines and escaped quotes, handling LF and CRLF record terminators, and +retaining configurable delimiters and NULL text. The strict read path now +rejects unterminated quoted fields and non-delimiter text after a closing quote, +matching DuckDB's fixed-dialect strict parser; permissive malformed-row +ingestion, broader DuckDB COPY options, and streaming I/O remain deferred. + +The JSON slice now covers `json_valid`, `json_array_length`, `json_type`, +bounded `json_keys`, the bounded `json_array(...)` constructor, the bounded +`json_object(...)` constructor, one-argument `json_quote(...)`, and +one-argument `json_pretty(...)`, bounded `json_extract(json, path)`, bounded +`json_extract_string(json, path)`, and bounded `json_exists(json, path)`. +The constructors support their +documented scalar arguments, JSON string escaping, and SQL NULL values +rendered as JSON `null`; `json_object(...)` additionally requires an even +key/value argument count and VARCHAR-compatible keys. `json_quote(...)` +JSON-serializes scalar values and propagates SQL NULL. `json_pretty(...)` +formats valid JSON/VARCHAR scalar, array, and object values and propagates +SQL NULL. `json_extract(...)` and `json_exists(...)` support only root, +object-field, and array-index paths: `$`, `$.key`, and `$[n]`. +`json_extract_string(...)` returns unquoted scalar strings and compact JSON +object/array text; `json_exists(...)` returns true for an existing JSON null. +Nested JSON values, broader paths, mutation, and the broader JSON function +family remain deferred. + +The DML slice now supports bounded `RETURNING` select-lists for INSERT, UPDATE, +and DELETE, with focused DuckDB-oracle comparisons for each statement kind. +Qualified names, `RETURNING *`, CTE/MERGE forms, conflict clauses, and broader +DML syntax remain deferred. + +The join slice covers `JOIN ... USING (column, ...)` with merged unqualified +key output, qualified source-key access, SQL NULL non-matches, and stable +missing-left/missing-right column diagnostics. Broader join syntax and +semantics remain tracked in B2. + +Acceptance: + +- Every upstream logical type has metadata, parser spelling, binder behavior, + vector representation, serialization, and explicit unsupported diagnostics + where execution is not yet possible. +- Property tests cover vector flattening/slicing/composition and validity + equivalence across all representations. + +### B2. Complete parser and binder coverage + +Implement and test: + +- Full statement grammar, parser diagnostics, source spans, formatting, + statement serialization, and parser extension hooks. +- Nested and lateral subqueries, correlated scalar/EXISTS/IN forms at all + supported nesting levels, quantified comparisons, recursive CTE semantics, + materialization hints, and advanced set-operation clauses. +- Full JOIN syntax and binding: NATURAL, USING, ASOF, POSITIONAL, SEMI, + ANTI, lateral, and multi-source ambiguity rules. +- Window frames, named windows, multiple window orders, window chaining, + advanced QUALIFY, and all registered window functions. +- Complete DDL/DML, constraints, macros, prepared statements, pragmas, + settings, COPY, EXPLAIN, and system-table syntax. + +Acceptance: + +- The non-I/O SQLLogicTest corpus runs without unexplained skips. +- Parser errors match category and source position; binder errors identify + the same ambiguous, missing, or invalid object as the oracle. + +### B3. Complete expression and function semantics + +Implement and test: + +- The full scalar and aggregate function manifests, overload resolution, + variadic functions, lambdas, macros, table functions, and function metadata. +- Exact overflow, rounding, NaN/infinity, collation, timezone, interval, + decimal, and NULL behavior. +- Aggregate state correctness across chunks, empty input, DISTINCT, FILTER, + ordering-sensitive aggregates, and nested/complex values. + +Acceptance: + +- Every registered function has an arity/type/null/error test and an oracle + result test or an explicit compatibility waiver. + +## 5. Phase C: Relational Engine and Optimizer + +Prerequisite: B1-B3. + +### C1. Logical and physical plan completeness + +Deliverables: + +- A complete logical-plan IR with stable explain output. +- Physical operators for scans, joins, aggregates, windows, set operations, + table functions, nested values, DML, COPY, and exchange/materialization + boundaries. +- Correct cardinality, type, ordering, and nullability propagation. +- Plan serialization and deterministic plan fingerprints for prepared-cache + and regression testing. + +### C2. Optimizer passes + +Implement, measure, and independently toggle: + +- Binding and constant folding. +- Filter, projection, and limit pushdown. +- Join reordering and algorithm selection. +- Aggregation and distinct rewrites. +- Common-subexpression elimination and CTE/materialization decisions. +- Top-N/order optimizations, zonemap/statistics pruning, and index selection. +- Cost model, statistics collection, and adaptive/runtime filters. + +Acceptance: + +- Optimized and unoptimized plans return identical results and errors. +- Each pass has a before/after plan test, a correctness differential test, and + a performance measurement; disabling a pass is deterministic. + +## 6. Phase D: Concurrency, Parallelism, and Operations + +Prerequisite: C1; D1 may begin after A3. + +### D1. Execution runtime + +Deliverables: + +- Task scheduler, worker pools, pipeline breakers, exchange operators, and + explicit connection/transaction ownership. +- Parallel scans, joins, aggregates, sorts, windows, and file I/O. +- Thread-count settings with deterministic one-thread and many-thread modes. +- Backpressure, bounded queues, memory accounting, and spill-to-disk. + +### D2. Control and observability + +Deliverables: + +- Cancellation, interrupts, deadlines, progress callbacks, profiling, + operator timing, cardinality counters, memory/spill metrics, and structured + logs. +- Streaming result fetch with bounded buffers and correct cleanup on early + close, cancellation, or consumer failure. + +Acceptance: + +- Results and errors are stable across 1/N threads. +- Cancellation tests stop CPU, I/O, worker tasks, and temporary files within + a bounded deadline. +- No connection, statement, worker, file, or buffer leaks under repeated + success and failure cycles. + +## 7. Phase E: Transactions, MVCC, and Durable Storage + +Prerequisite: D1 ownership and resource accounting. + +### E1. Transaction manager + +Implement: + +- Transaction IDs, snapshots, read/write sets, isolation levels, conflict + detection, savepoints, nested statement rollback, and concurrent catalog + ownership. +- MVCC versions for table rows, indexes, sequences, views, settings, and + catalog entries. +- Atomic multi-table DDL/DML, failed-commit recovery, and serializable conflict + tests. + +### E2. Storage engine + +Implement: + +- Buffer manager, page/block layout, free lists, metadata blocks, compression, + checksums, encryption policy, versioning, and migration. +- WAL records, ordering, fsync policy, checkpointing, replay, truncation, + crash recovery, and corruption handling. +- Persistent indexes, statistics, zonemaps, temporary files, spill files, and + vacuum/compaction. + +Acceptance: + +- Restart after every WAL phase yields the same committed state and no + uncommitted state. +- Fault injection covers torn writes, partial records, invalid checksums, + missing pages, disk-full, and interrupted checkpoints. +- The storage compatibility policy is explicit: byte-compatible with DuckDB, + import/export-compatible, or intentionally separate with a migration tool. + +## 8. Phase F: Data Ecosystem and Extensions + +Prerequisite: B1, D2 resource handling, and E2 durable/temp storage as needed. + +### F1. File and data formats + +Implement and test: + +- Complete CSV parsing/writing, dialect detection, quoting, encoding, errors, + COPY, and parallel ingestion/export. +- JSON scalar/nested values, JSON functions, serialization, and newline/multi- + file ingestion. +- Parquet metadata, statistics, compression codecs, nested definition/repetition + levels, projection/filter pushdown, partition discovery, and corruption + handling. +- Arrow C Data/C Stream import/export, ownership callbacks, dictionary arrays, + nested arrays, and zero-copy paths where valid. +- Glob, HTTP/object-store paths, secrets, filesystem policies, and remote I/O + retries/limits. + +Acceptance: + +- Round-trip and corruption suites pass for every claimed type/codec/path. +- Results match DuckDB for local, multi-file, compressed, nested, and empty + inputs. + +### F2. Extension system + +Implement: + +- Native `LOAD`/`INSTALL`, repository resolution, manifests, signatures or + trust policy, ABI/version checks, dependency resolution, and unload rules. +- Registration for scalar/table functions, types, casts, collations, macros, + storage formats, and replacement scans. +- Sandboxing, path canonicalization, symlink/TOCTOU defenses, and disabled- + network behavior. + +Acceptance: + +- First-party extensions load, query, unload, and fail safely across version + mismatches; third-party ABI compatibility is tested with a fixture SDK. + +## 9. Phase G: Public APIs and Clients + +Prerequisite: stable execution, errors, streaming, types, and storage policy. + +Implement and version independently: + +- Complete C ABI: opaque handles, allocators, result ownership, appender, + prepared parameters, streaming fetch, configuration, cancellation, + callbacks, error strings/categories, and thread-safety rules. +- Extension ABI and C++ relation/shim APIs. +- Arrow and ADBC interfaces. +- Shell behavior, command modes, output formats, history, completion, and + diagnostics. +- Swift, Julia, and other supported client examples with CI fixtures. + +Acceptance: + +- ABI symbol/layout checker passes for every claimed release symbol. +- C/C++/Arrow/ADBC/client examples build and run without linking DuckDB. +- Ownership, double-close, NULL, cancellation, and allocator-failure tests + pass under sanitizers. + +## 10. Phase H: Security, Packaging, and Release Evidence + +Prerequisite: all runtime phases that introduce external input or native code. + +Deliverables: + +- Parser, binder, serializer, storage, WAL, format, extension, and API fuzz + corpora with stable seeds and resource limits. +- Security/import-policy/resource-leak/blocking-FFI/NULL-safety/static-symbol + audits, with no unresolved high-severity findings. +- Reproducible debug/release/WPO builds, static/shared libraries, shell, + extension SDK, SBOM, license bundle, checksums, signatures, and upgrade/ + rollback notes. +- Hermetic, seedable, sharded, resumable CI artifacts for unit, differential, + API, storage, recovery, fuzz-smoke, benchmark, security, and release gates. + +Acceptance: + +- A clean machine can fetch, build, test, package, install, run, and uninstall + the project without a source-tree dependency. +- A second build is reproducible within the documented toolchain constraints. +- Sanitizers, corruption tests, fuzz-smoke, and resource-leak tests are clean. + +## 11. Convergence and Removal Workflow + +For each ledger row: + +1. Add an upstream reference and a minimal failing fixture. +2. Classify the gap: parser, binder, type, executor, optimizer, storage, API, + extension, harness, or performance. +3. Implement the smallest complete vertical slice, including errors and + resource cleanup. +4. Add unit, oracle, randomized, and restart/concurrency tests appropriate to + the feature. +5. Run the phase gates and update evidence paths. +6. Move the ledger row from `missing` to `partial` only when a bounded subset + is honestly documented; move it to `compatible` only when the complete + scoped contract passes. +7. Remove the gap-specific test only when the general contract supersedes it; + retain the regression as part of the permanent suite. + +## 12. Final Definition of Done + +Full gap removal is achieved only when all of the following are true: + +- Every upstream contract row is `compatible` or has an approved, current + waiver with a documented rationale. +- The full SQLLogicTest corpus, API suites, storage/recovery suites, fuzz-smoke, + and differential suites pass. +- Optimized/unoptimized, one/many thread, cold/warm cache, prepared/direct, + in-memory/spill, and restart/corruption matrices pass. +- C ABI, extension ABI, Arrow/ADBC, and supported client compatibility checks + pass. +- Security, sanitizer, resource-limit, leak, packaging, and reproducibility + gates pass. +- E1 through E6 are reported with machine-readable results, exclusions, and + historical regressions; no unsupported behavior is hidden in an aggregate. new file mode 100644 --- /dev/null +++ b/IMPLEMENTATION_HANDOFF.md @@ -0,0 +1,493 @@ +# Implementation Handoff — Jerboa DuckDB, Phase 1 Vertical Slice + +**Audience:** the local LLM (or human) implementing the next phase. +**Date:** 2026-07-27. **Jerboa:** 0.2.6 (Chez 10.4.0). **Upstream oracle:** `duckdb` CLI v1.5.1 at `/opt/homebrew/bin/duckdb`; pinned source tree at `vendor/duckdb` (commit `117e1a46be`, see `Project.md`). + +This document is the output of a research + validation pass. The hard +architectural choices are **made and machine-verified**: the four foundation +modules under `lib/jerboa/duckdb/common/` are implemented, compile cleanly, +and pass kernel smoke tests. Everything below tells you exactly what exists, +what to build next, in what order, with which interfaces, and which traps +will waste your time if you ignore them. + +--- + +## 1. Mission and scope + +Build the **Phase 1 in-memory scalar vertical slice** defined in +`Project.md`: a DuckDB-semantics analytical engine in pure Jerboa Scheme +(no `libduckdb` FFI in production code — the CLI is a *test oracle only*, +mirroring the jsqlite working rule). + +In scope for this slice: + +- All 54 logical type IDs as metadata; full execution support for scalar + types: BOOLEAN, all signed/unsigned ints, HUGEINT (boxed exact), FLOAT, + DOUBLE, VARCHAR, BLOB; DECIMAL/DATE/TIME/TIMESTAMP/INTERVAL represented. +- Columnar in-memory storage (row groups of typed vectors), validity bitmaps, + FLAT/CONSTANT/DICTIONARY/SEQUENCE vector representations. +- Vectorized, type-specialized expression kernels (this is the performance + core — already built). +- SQL: `CREATE TABLE`, `INSERT` (VALUES and INSERT..SELECT), `VALUES`, + `SELECT` with `WHERE`, `DISTINCT`, `GROUP BY` + `HAVING`, aggregates + (`count`/`sum`/`avg`/`min`/`max`), `ORDER BY` (with `NULLS FIRST/LAST`, + ordinals, aliases), `LIMIT`/`OFFSET`, expressions with full precedence, + `CASE`, `CAST` + `::`, `BETWEEN`, `IN`, `LIKE`, `IS [NOT] NULL`, + boolean `AND/OR/NOT`, scalar function subset. +- API: `duckdb-open`/`duckdb-connect`/`duckdb-query`/`duckdb-prepare`/ + `duckdb-execute`, materialized columnar results, upstream error categories. + +Explicitly **out of scope** (later phases; do not gold-plate): joins, +subqueries/CTEs, set operations, window functions, nested-type execution, +persistent storage/WAL/MVCC, parallelism (`(std actor)` scheduling is +Phase 3), optimizer passes, CSV/JSON/Parquet I/O, the C ABI. + +## 2. Ground rules (repo conventions — verified empirically) + +1. **Library modules** are `.ss` files containing `(library (jerboa duckdb + ...) ...)` forms, consumed via libdirs — this is how jerboa's own stdlib + (`lib/std/*.ss`) and jsqlite (`src/jsqlite/*.ss`) both work. The repo + `AGENTS.md` "NEVER write `(library ...)` forms" rule targets *scripts*; + libraries are the exception, and there is no other multi-module mechanism. +2. **Scripts/tests** are plain `.ss` with `(import (jerboa prelude) ...)` at + top — no `(library ...)`. Run with `jerbuild exec --libdirs ...` (NOT + `jerboa run` — that subcommand does not exist in jerboa 0.2.6; `jerboa + file.ss` works for stdlib-only scripts but ignores `JERBOA_LIBDIRS`). +3. **Build/run commands** (from repo root): + ```sh + JH=$(jerbuild --jerboa-home) + jerbuild compile --libdirs "lib:$JH/lib" lib/jerboa/duckdb/common/types.ss + jerbuild exec --libdirs "lib:$JH/lib" test/unit/engine.ss + ``` + Add a `build` target compiling `$(find lib -name '*.ss')` and switch the + `unit` target to `jerbuild exec --libdirs "$(ROOT)/lib:$(JERBOA_HOME)/lib"` + (the existing `make unit` uses plain `jerboa` and must keep passing the + inventory path arg to `test/unit/upstream-contracts.ss`). +4. **Editing `.ss` files:** use `jerboa_write_file` / `jerboa_balanced_insert` + / `jerboa_balanced_replace` per the repo AGENTS.md. **Caveat found this + session:** `jerboa_write_file verify:true` rejects `(library ...)` modules + ("Exception in cadr: incorrect list structure (defstruct)") because the + MCP verifier expands forms without the library import environment. Write + without `verify`, then verify with `jerbuild compile` (the ground truth). + Run `jerboa_check_balance` after every edit. +5. **Imports in library modules:** import `(except (chezscheme) ...)` with the + prelude-conflict exclusions, then `(except (jerboa prelude) meta atom?)` — + exactly the jsqlite pattern (see header of any existing module). Hot code + uses raw Chez primitives; jerboa sugar (`def`, `defstruct`, `match`, + `try`, hash fns) comes via the prelude import. +6. **`defstruct` exports:** it generates `make-X`, `X?`, `X-field`, + `X-field-set!` — there is NO binding named `X` itself; do not list the + bare type name in `(export ...)`. +7. **Never touch sibling repos** (`~/mine/jerboa-sqlite` etc.) — read-only + reference. Never reference sibling checkouts from build files. +8. **`(def ...)` after an expression in a body is invalid** — internal + defines must come first in a body (repo AGENTS.md rule 5). +9. Pre-commit on macOS: `make binary` must pass (per AGENTS.md); do not + commit unless the user asks. + +## 3. Research findings + +### 3.1 Upstream contract facts (from `contracts/generated/upstream-inventory.sexp`) + +- Pinned checkout: 15,476 tracked files; 33 statement kinds; 66 logical + operators; 86 physical operators; **54 logical type IDs**; 6 vector types + (FLAT, FSST, CONSTANT, DICTIONARY, SEQUENCE, SHREDDED); 42+1 optimizer + types; 17 compression types; 171 settings; 548 C API symbols; 452 function + manifest entries; 5,207 sqllogictest files. +- `make contracts-check` verifies this inventory stays in sync; CI gates on it. + +### 3.2 The jsqlite contrast (why this architecture) + +`~/mine/jerboa-sqlite` is a from-scratch SQLite engine in Jerboa: ~44k LOC, +26 modules, **1.2–1.4× slower than C SQLite** on the canonical +sqlite.org/speed.html workload at N=10000. Its perf pass (`perf-pass.md`) +proves the levers that matter for us: + +- **Algorithmic wins dominate.** Unsafe O3 buys only 6–25%; GC is 0–4 ms even + on a 180 MB workload. "Do less work" beats "tune codegen". +- **Its biggest deficit (B1) was per-row full-record decode of untouched + columns.** A *columnar* engine eliminates this by construction — untouched + columns are never read. This is the single strongest argument for DuckDB's + architecture over a row store in Scheme. +- **Finding A: per-statement literal-SQL lifecycle cost** (re-tokenize per + statement) was 71% of indexed-select time. Lesson: `duckdb-prepare` must + parse+bind ONCE and cache compiled plans; `duckdb-query` should cache by + SQL text keyed on catalog generation. +- VDBE-style per-row dispatch is the wrong model. The right model is + **vectorized batch execution**: operators consume/produce ~2048-row data + chunks; per-cell cost is one unboxed access + one op, no dispatch. + +### 3.3 DuckDB semantics — verified against the CLI oracle + +Re-verify with `duckdb -csv -c "..."` before relying on anything else: + +| Semantics | Verified behavior | +|---|---| +| `5/2` → `2.5` DOUBLE; `5//2` → `2` INT32 | `/` on ints is DOUBLE division; `//` is integer division | +| `1/0` → `inf`; `1//0` → `NULL` | float div is IEEE; int div/mod by zero → NULL (not error) | +| `9223372036854775807 + 1` | **Out of Range Error** (int64 overflow errors; does NOT promote to HUGEINT) | +| `SELECT 1 WHERE 1` | works — non-zero numbers implicitly cast to BOOLEAN in boolean context | +| `SELECT 1 WHERE NULL` | 0 rows (NULL is not true) | +| ORDER BY default | NULLS LAST for ASC, NULLS FIRST for DESC (NULL = largest value; `default_null_order=nulls_last`) | +| `GROUP BY` | NULL keys group together | +| Empty-set aggregates | `count(*)`/`count(x)` → 0; `sum`/`avg`/`min`/`max` → NULL | +| Aggregate types | `sum(int)` → HUGEINT; `avg` → DOUBLE; `count(*)` → BIGINT; `min`/`max` keep input type | +| Literal typing | `typeof(5)`=INTEGER, `typeof(5000000000)`=BIGINT; `'2'+1` → **Binder Error** (ambiguous literal+literal); `1='1'` → true (STRING_LITERAL coerces to the other side's type) | +| `'abc' LIKE 'a%'`, `BETWEEN`, `IN`, `CASE`, `coalesce`, `nullif` | standard | + +### 3.4 Jerboa toolchain facts + +- `(library (foo) ...)` in `.ss` resolves via libdirs (verified). +- `(jerboa core)` exports `def defstruct defrule match try catch finally + displayln hash-* make-hash-table ...`; the prelude is a superset. +- Cookbook recipe `columnar-typed-columns-fxvector-flvector` (measured + ~2.4× on unboxed double reductions) is the basis of the storage classes. +- Chez fixnum is ~60 bits on aarch64 — **not** a full int64 container + (Project.md warns about this explicitly). + +## 4. Hard decisions (already made — do not relitigate) + +1. **Full native conversion.** No `libduckdb` FFI except in `test/`/`tools/` + as a differential oracle. (`Project.md` executive directive.) +2. **Columnar storage from day one.** Tables = list of row groups; each row + group is a `data-chunk` of FLAT typed column vectors + per-column validity + bitmap. Untouched columns are never touched (kills jsqlite B1). +3. **Four storage classes** (`types.ss`): `bool` → u8 bytevector; `fx` → + fxvector (INTEGER/SMALLINT/TINYINT + date/time/timestamp); `fl` → + flvector (FLOAT/DOUBLE); `boxed` → generic vector (BIGINT and beyond, + VARCHAR, BLOB, DECIMAL, INTERVAL, nested). Rationale: fixnum ≈60 bits + can't hold int64; exact bignums give HUGEINT semantics for free in + accumulators (sum promotes fixnum→bignum naturally = HUGEINT). +4. **Chunk-level shared selection vector.** Filters never copy columns; they + wrap every column in a DICTIONARY vector sharing the backing store with + ONE selection (`chunk-slice-chunk` composes nested selections). This + collapses the kernel matrix: every binary kernel takes a single `sel` + serving both inputs. Kernels come in direct/indirect loop shapes from one + macro. This is THE key perf decision. +5. **Type-specialized kernels generated by macros at bind time** + (`kernels.ss`). Dispatch `(op lstorage rstorage)` → specialized closure + ONCE per expression; per-chunk call is a tight `do` loop with + `fx+`/`fl+`/`fx<` etc. Generic boxed fallback for BIGINT/VARCHAR/etc. + Constant operands get `-rc`/`-lc` scalar×vector variants (upstream's + flat/constant executor split). +6. **Booleans are u8 + validity** (no three-state data); 3VL in AND/OR/NOT + kernels: FALSE dominates AND, TRUE dominates OR, else NULL. +7. **Validity = bit-packed bytevector, `#f` = all-valid.** Allocate on first + NULL; fast path (no NULLs) allocates nothing and branches once per chunk. +8. **Literal types** STRING_LITERAL/INTEGER_LITERAL exist as binder-only + types with coercion rules (§3.3), matching upstream's literal resolution. +9. **Errors** are raised as `(error category message irritants)` with + category symbols `parser`/`binder`/`catalog`/`execution`/`out-of-range`, + message text prefixed upstream-style ("Binder Error: ..."). The API layer + catches and optionally returns `(err ...)` (prelude result type). +10. **Pull model** (chunk-at-a-time `next()` closures) for the slice; + DuckDB's push/scheduler model is Phase 3 with `(std actor)` + + `(std misc channel)` per `Project.md` Workstream 7. + +## 5. What already exists (compiled + smoke-tested) + +``` +lib/jerboa/duckdb/common/types.ss (jerboa duckdb common types) +lib/jerboa/duckdb/common/value.ss (jerboa duckdb common value) +lib/jerboa/duckdb/common/vector.ss (jerboa duckdb common vector) +lib/jerboa/duckdb/common/kernels.ss (jerboa duckdb common kernels) +``` + +- **types.ss** — `logical-type` record (id name storage width scale + children); all 54 IDs (`*logical-type-ids*`); canonical type values + (`type-integer`, `type-double`, ...); predicates + (`numeric-type?` `integer-type?` `string-type?` `temporal-type?` + `literal-type?`); `implicit-cast-cost` (numeric hierarchy rank + + literal rules); `common-type`; `sql-name->type` (parser's type names + + aliases). +- **value.ss** — `sql-null` sentinel; `interval`/`decimal` records; + `sql-truthy` (numbers → boolean, error on VARCHAR per upstream); + `sql-and/or/not` (3VL); `value-compare` (cross-numeric, string BINARY, + bytevector, interval); `null-safe-compare` (NULL = largest); + `sql-hash-key`; `display-value`/`value->csv-string` (NULL→"NULL", + booleans lowercase — matches oracle CSV output). +- **vector.ss** — validity bitmap ops (`make-validity`, `validity-valid?`, + `validity-set-null!`, `combine-validity`); storage-class + allocate/ref/set!/length; `dvector` record (vtype storage data validity + count aux); FLAT/CONSTANT/DICTIONARY/SEQUENCE constructors; + `vector-value-ref` (generic), `vector-flat-view` → `(values data valid + sel n)` for kernels, `vector-flatten`, `vector-slice`; `data-chunk`; + **`chunk-slice-chunk`** (zero-copy filter: wraps columns in DICTIONARY + with shared sel, composes nested sels); `rows->chunk` (INSERT pivot); + `*standard-vector-size*` (parameter, default 2048; the 512 config is an + acceptance variant). +- **kernels.ss** — the macro system and kernel table: + - `define-syntax def-bin-kernel / def-bin-kernel-rc / def-bin-kernel-lc` + generate direct+indirect specialized loops; per-cell ops inline. + - `arith-kernel[-rc/-lc] (op ls rs)`: specializations for + `+ - * / // %` over (fx,fx) (fx,fl) (fl,fx) (fl,fl); `/`→fl always; + `//`/`%` zero→NULL hand-written; boxed generic fallback. + - `cmp-kernel[-rc/-lc]`: `<` specializations + `value-compare` generic + for `= <> < <= > >=`; outputs u8 bool. + - `and-kernel`/`or-kernel` (3VL), `not-kernel`, `is-null-kernel`. + - `where-select` → `(values fxvector count)` of TRUE view indices. + - `cast-kernel`/`elementwise-unary-kernel` + `cast-elt-proc` (numeric, + boolean, varchar casts with upstream error text). + - `like-match?` (direct %/_ matcher), `in-list-kernel`. + - Aggregates: `make-agg-state`, `agg-step` (per-kind per-storage tight + loops; sum uses generic `+` on fixnums → natural HUGEINT promotion), + `agg-finalize`, `agg-state-null?` (empty sum/avg/min/max → NULL, + count → 0). + +**Smoke-test evidence** (all passed this session): fx+fx with NULLs and +with selection; `<` specialization; sum with NULL skip = 70; empty sum → +NULL; where-select count; fx→fl cast. Reproduce: + +```sh +JH=$(jerbuild --jerboa-home) +jerbuild compile --libdirs "lib:$JH/lib" lib/jerboa/duckdb/common/kernels.ss +# then jerbuild exec a script that imports (jerboa duckdb common kernels) +# and calls (arith-kernel '+ 'fx 'fx) etc. — see §8. +``` + +Empty dirs `catalog/ parser/ planner/ execution/` exist under +`lib/jerboa/duckdb/` matching `Project.md`'s layout. + +## 6. Remaining work — module-by-module plan + +Build in this order; each module gets a compile + focused smoke test before +moving on (§8). Keep every file's closer-runs ≤ 4 (use local helper defines; +see `kernels.ss` for the pattern — this matters, see pitfall P2). + +### 6.1 `catalog/catalog.ss` — `(jerboa duckdb catalog catalog)` + +- `database` record wrapping a catalog of schemas ("main", "temp") → + hash tables, keyed by `string-downcase` (DuckDB folds identifiers for + lookup but preserves display case). +- `table` record: name, columns (vector of names), column-types (vector of + logical-types), storages (cached vector of storage classes), row-groups + (prepend list of data-chunks), total-rows. +- API: `make-database`, `catalog-create-table!` (error: "Catalog Error: + table already exists"), `catalog-find-table` (schema-qualified `s.t`, + temp fallback; error: "table does not exist"), `catalog-drop-table!`, + `table-insert-chunk!`, `table-scan-chunks` (insertion order), + `table-column-index` (case-insensitive). +- Generation counter on the catalog for the prepared-statement cache key + (jsqlite Finding A: invalidate cached plans on DDL). + +### 6.2 `parser/tokenizer.ss` — `(jerboa duckdb parser tokenizer)` + +- Hand-coded, fixnum-indexed scan over the SQL string (no per-char + allocations; jsqlite's tokenizer profile shows this is worth doing right). +- Tokens: identifiers (+ keyword table, case-insensitive), `"quoted idents"` + (double-quote escape), `'strings'` (`''` escape), numbers (int / decimal / + exponent — keep the raw text, classify at parse time), `$1..$n` and `?` + parameters, multi-char operators `>= <= <> != || :: // ~~ !~~`, + punctuation `( ) , ; . *`, comments `--` and `/* */`, EOF with source + offset for error positions. +- Output: vector of `(kind value offset)` — kind ∈ + `ident keyword number string param op lparen rparen comma semi dot star eof`. + +### 6.3 `parser/ast.ss` + `parser/parser.ss` + +- AST as tagged lists with constructors (`(e-lit value type-tag)`, + `(e-ref name table?)`, `(e-binary op l r)`, `(e-call name args)`, + `(e-case ...)`, `(e-cast e type)`, `(e-star table?)`, `(e-between)`, + `(e-in)`, `(e-is-null e not?)`); statements as records: + select-statement / create-table-statement / insert-statement. +- Recursive descent, precedence (low→high): OR < AND < NOT < + IS/IN/BETWEEN/LIKE/comparison < `+ -` < `* / % //` < unary < `::` cast < + postfix/primary. Parenthesized exprs, function calls, `COUNT(*)`, + `CASE WHEN .. THEN .. [ELSE ..] END`, `CAST(x AS T)`, `x::T`. +- SELECT: `SELECT [DISTINCT] select-list FROM table-ref [alias] + [WHERE e] [GROUP BY exprs] [HAVING e] [ORDER BY items [ASC|DESC] + [NULLS FIRST|LAST]] [LIMIT n [OFFSET m]]`; select items + `expr [AS alias]`, `*`, `t.*`; FROM: single table for the slice (joins + are Phase 2). Also `VALUES (..),(..)..` as a statement and table source. +- Errors: "Parser Error: ..." with offset. + +### 6.4 `planner/binder.ss` + +Resolves AST → bound expression tree (`bx` record: kind type a b c) + plan +tree against the catalog: + +- Column resolution against the FROM namespace; ambiguity → "Binder Error: + Ambiguous column reference"; unknown → " Binder Error: column not found". +- Type inference bottom-up; implicit casts inserted as `cast` bx nodes via + `implicit-cast-cost`/`common-type`; literal coercion (§3.3): a + STRING_LITERAL next to a non-string type re-parses into that type; + INTEGER_LITERAL adopts the numeric context (default INTEGER, BIGINT when + out of int32 range). +- Arithmetic: result type per `kernel-out-storage` semantics (`/`→DOUBLE; + int `+ - *` keep type; `// %` keep int type). Boolean context inserts + truthiness cast (numbers → BOOLEAN). +- Aggregates: `count/sum/avg/min/max` recognized; `count(*)` special; + aggregate inputs are arbitrary bound exprs; result types per §3.3. + Detection of aggregates switches the plan to the aggregate operator; + `GROUP BY` keys + `HAVING` validated (post-agg exprs reference group + keys/aggregates only — enforce with "Binder Error: column must appear in + GROUP BY"). +- Star expansion against the catalog; select-list aliases registered for + ORDER BY resolution; ORDER BY ordinals (`ORDER BY 1`). +- Output plan nodes: `scan-plan` (table, column idxs, types), `values-plan`, + `filter-plan`, `project-plan`, `agg-plan` (group-bxs, agg-bxs), `order-plan` + (keys + asc/nulls flags), `limit-plan`, `distinct-plan`. + +### 6.5 `planner/compiler.ss` + +`compile-expr bx → (lambda (chunk) → dvector)`: + +- `colref` → the chunk's column dvector directly (reference, no copy). +- `lit` → `make-constant-vector`. +- `cast` → `cast-kernel` applied to child output (flatten child first if it + has no flat view — CONSTANT flattens; DICTIONARY passes through with the + chunk's shared sel). +- `arith`/`cmp` → `arith-kernel`/`cmp-kernel` dispatch on the children's + storage classes; constant child → `-rc`/`-lc` variant; both-constant → + constant-fold at compile time. +- `and/or/not/isnull` → corresponding kernels. +- `case` → build per-branch bool vectors then select (or elementwise with + pre-computed branch predicates; correctness first). +- Scalar functions (`abs upper lower length sqrt round floor ceil greatest + least coalesce nullif concat || like in`) — registry: name → + `(arg-types) → (values result-type kernel-builder)`; vectorized via + `elementwise-unary-kernel` or hand kernels; `coalesce` = first-valid + merge of validity maps. +- Compiled exprs return FLAT vectors (materialize results) except colref + (pass-through) — this is the fused-gather point. + +### 6.6 `execution/operators.ss` + +Each operator = closure `(lambda () → chunk-or-#f)` (pull model): + +- **scan**: yields the table's row-group chunks in order. +- **filter**: compile predicate; per chunk, get bool vector + `where-select`, + then `chunk-slice-chunk` (zero-copy). +- **project**: evaluate compiled exprs → new FLAT columns chunk. +- **limit/offset**: count rows across chunks, slice boundary chunk. +- **order**: materialize all rows (flatten key columns), build index + fxvector `0..n-1`, `vector-sort!` with comparator = lexicographic + `null-safe-compare` per key with per-key asc/desc inversion and NULLS + FIRST/LAST flags; emit sorted chunks. (Comparator over typed storages: + specialize single-fx-key and single-fl-key fast paths — they dominate + benchmarks.) +- **hash-aggregate**: group-key columns flattened once; per row build key + (single fixnum key → eqv? hashtable fast path; else key list → + equal?-hash hashtable); per group a vector of agg states; step via + `agg-step` per chunk (tight loops, no per-row records); finalize via + `agg-finalize`; output chunk built with `rows->chunk` semantics. + Ungrouped aggregate = one implicit group (empty input still emits one row: + count→0, others→NULL). +- **distinct**: equal?-hash set on materialized row keys. + +### 6.7 `api.ss` + umbrella `(jerboa duckdb)` + +- `duckdb-open` → database; `duckdb-connect db` → connection record + (db + statement cache keyed on `(sql . catalog-generation)`). +- `duckdb-query conn sql` → `query-result` record (names, types, columns + vector-of-vectors-of-values, row-count) — raises on error; + `duckdb-query/result` → `(ok result)`/`(err message)` (prelude results). +- `duckdb-prepare` (parse+bind+compile once), `duckdb-execute stmt . params` + (params substitute `$1..`/`?` as literals before binding — bind-time + constant vectors). +- DDL/DML statements return an empty result with a "Count" of affected rows + where upstream does (INSERT returns count). +- Umbrella `lib/jerboa/duckdb.ss` re-exports api + types via + `(export (import (jerboa duckdb api)) (import (jerboa duckdb common types)))`. + +## 7. Pitfalls register (each one cost real time — read before coding) + +- **P1 — `case` never matches list keys.** `case` compares clause datums + with `eqv?`; a freshly consed `(list op ls rs)` never matches `(+ fx fx)`. + All kernel triple-dispatch MUST use `(assoc key table)` (equal?) or + nested `cond` with `eq?`. Symptom: silent fallthrough to the boxed + generic path, then `vector-ref: #vfx(...) is not a vector`. +- **P2 — Paren repair makes *balanced-but-wrong* files.** Dropping/adding a + closer can glue top-level forms together; the file then "compiles" with + exports missing (defs became internal). Verify structure, not just + balance: read the file as data and check the library body form count and + that every `(export ...)` name has a body-top-level `(def ...)`. Keep + closer-runs ≤ 4 with local helper defines (see `def-bin-kernel` in + kernels.ss) so this class of bug can't happen. +- **P3 — syntax-rules hygiene breaks kernel macros.** If a macro's use + sites write `(fxvector-ref ld j)` and `(fx+ a b)` referencing identifiers + the template binds, `defrule` (syntax-rules) renames them → "unbound + identifier a". Fix: `define-syntax` + `syntax-case` + + `(datum->syntax #'name 'a)` capture for exactly `a b ld rd j` (pattern in + place in kernels.ss — copy it). +- **P4 — `jerboa_write_file verify:true` rejects `(library ...)` modules** + (MCP verifier bug, "incorrect list structure (defstruct)"). Write without + verify; compile with jerbuild. +- **P5 — `defstruct` has no bare-name export.** Export `make-X`, `X?`, + accessors only. +- **P6 — `jerboa run` doesn't exist; `JERBOA_LIBDIRS` is ignored.** Use + `jerbuild exec --libdirs`. +- **P7 — Empty-set aggregates.** sum/avg/min/max over zero valid rows → + NULL, count → 0. Track valid-row counts in states (`sum` state is + `(vector (box acc) (box nvalid))`). +- **P8 — Integer `/` is DOUBLE** (`5/2`=2.5), `//` integer; int div/mod by + zero → NULL; int64 `+ - *` overflow → raise `out-of-range` (fixnum ops + trap at safe optimize levels — let the trap propagate as the error). +- **P9 — Internal defines must lead a body.** `(def ...)` after any + expression in a body → "invalid context for definition" (or worse, a + glued form; see P2). +- **P10 — Chez fixnum ≈ 60 bits.** Never store BIGINT in fxvector; boxed + exact integers give correct HUGEINT accumulation for free. + +## 8. Verification workflow (per module, mandatory) + +1. Write file (`jerboa_write_file`), then `jerboa_check_balance`. +2. Compile: `jerbuild compile --libdirs "lib:$JH/lib" <file>`. Fix before + proceeding — never stack uncompiled modules. +3. Focused smoke script in `/var/folders/01/7797pkc13nq0x1fl7wxp1s0c0000gn/T/opencode/` + run via `jerbuild exec --libdirs "lib:$JH/lib"`, asserting with `assert!`. +4. Structural check after any paren-level incident (P2): read file as data, + `(cdddr lib-form)` body length > 2, every export has a top-level def. +5. When behavior is in doubt, ask the oracle: `duckdb -csv -c "<sql>"` and + match it. Record the verified semantics in this doc's §3.3 table format. +6. Save non-trivial discoveries: `jerboa_howto_add` (recipes), + `jerboa_error_fix_add` (P1–P4 above are candidates if not already saved), + `jerboa_anti_pattern_add` for failed strategies. + +## 9. Tests, differential oracle, benchmark + +- **`test/unit/*.ss`** — one per layer: `vector.ss` (validity/slice/flatten/ + chunk-slice composition), `kernels.ss` (port the §5 smoke test), + `types.ss`, `parser.ss` (AST shapes + error positions), `engine.ss` + (end-to-end SQL cases with expected results from the oracle). +- **`test/diff/run.ss`** — deterministic query list over a fixed generated + dataset; run through jerboa engine AND `duckdb -csv -c` into a temp file; + compare CSV text byte-for-byte (value.ss CSV rendering already matches + oracle formatting: NULL, true/false, integral doubles without `.0`). + Make target `diff` with `DUCKDB_CLI ?= duckdb`. +- **`benchmark/bench.ss`** — build a 1M-row table + `(a INTEGER, b DOUBLE, c VARCHAR)` via direct chunk construction (not + SQL INSERT — measure the engine, not ingestion); time: + scan+filter (`WHERE a < 900000 AND b > 0.5`), `count(*)`/`sum(a)`/`avg(b)`, + `GROUP BY` on a 1k-cardinality int key, `ORDER BY a DESC LIMIT 100`. + Print ms per query, plus the `duckdb` CLI time for the same queries, plus + the ratio. Per `Project.md` E6: publish ratios, never claim parity. +- **Perf expectation setting:** jsqlite is 1.2–1.4× of C SQLite; the + vectorized columnar model should do *relatively* better on scan/aggregate + workloads (tight unboxed loops, no per-row dispatch) and relatively worse + where boxed/string columns dominate. Measure before tuning; unsafe O3 is + a last resort (6–25% in jsqlite's measurements). +- **Make targets to add:** `build` (compile all `lib/**/*.ss`), `unit` + (switch to jerbuild exec + libdirs), `diff`, `bench`. Keep + `contracts-check`, `check-docs`, `security` intact. + +## 10. When the slice is done — bookkeeping (required, not optional) + +1. Update `contracts/compatibility.sexp`: flip the relevant E-dimensions to + `partial`; add feature rows (types-values-vectors, sql-parser-subset, + binder, vectorized-execution, public-api) with `status: partial` and + `evidence` file lists. +2. Update `Project.md`'s status header (what works, with evidence paths); + do NOT check acceptance boxes whose named gates don't fully pass. +3. Update `README.md` command list (`build`, `diff`, `bench`). +4. Run `make verify` (contracts-check + unit + check-docs) green, plus + `make binary` (macOS pre-commit rule) before any commit the user asks for. +5. Save discoveries (§8.6) — the kernel-macro pattern (P3), the case/eqv? + dispatch trap (P1), and the balanced-but-wrong repair trap (P2) are all + reusable across every future Jerboa project. + +--- + +*Session artifacts: the four `common/` modules compile clean on jerboa +0.2.6; kernel smoke tests passed 2026-07-27; all §3.3 semantics were +verified against duckdb CLI v1.5.1 on that date.* --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -JERBUILD ?= $(if $(wildcard ../jerboa/dist/jerbuild),$(abspath ../jerboa/dist/jerbuild),jerbuild) +JERBUILD ?= jerbuild JERBOA ?= jerboa JERBOA_HOME := $(shell "$(JERBUILD)" --jerboa-home 2>/dev/null) ifeq ($(JERBOA_HOME),) @@ -6,19 +6,39 @@ $(error jerbuild not found on PATH or --jerboa-home failed) endif ROOT := $(abspath .) -LIBDIRS := $(ROOT):$(JERBOA_HOME)/lib -DUCKDB_SOURCE ?= /Users/user/duckdb