updates
ober
770e612f877a848996dda61876305d4a089cb333
new file mode 100644 --- /dev/null +++ b/plan.md @@ -0,0 +1,382 @@ +# jerboa-db — Performance Plan + +**Goal**: close the Datomic/Datahike gap on MBrainz to "competitive on point +queries, faster on analytics, in 10K lines of Scheme." Realistic, measurable, +demo-ready in 3–4 weeks of focused work. + +This document supersedes any prior plan in this file. It assumes the current +state of `master` at commit `5635268` (post-Phase-8, all 37 core tests passing, +hash-join + streaming aggregate + range pushdown already in the engine). + +--- + +## 1. Where we stand today + +MBrainz, 1% scale (2,620 artists / 13,100 releases / 131,000 tracks), one +warm run on `master`: + +| Query | What it does | Now (1%) | Projected ×100 | Datahike full | +|---|---|---:|---:|---:| +| Q1 | Artist exact name lookup | 0 ms | <1 ms | ~1 ms | +| Q2 | Releases by artist name (2-hop) | 0 ms | <1 ms | ~1 ms | +| Q3 | `startYear < 1960` range | 1 ms | ~200 ms | ~5 ms | +| Q4 | Tracks > 240s on shared-artist releases | 1421 ms | **~250 s** | 2–8 s | +| Q5 | Releases per country (group-by) | 2 ms | ~400 ms | ~10 ms | +| Q6 | Reverse-ref lookup | 0 ms | <1 ms | ~1 ms | +| Q7 | Pull entity attrs | 0 ms | <1 ms | ~1 ms | +| Q8 | Avg track duration by status | 966 ms | **~205 s** | 0.5–2 s | + +**Read takeaway**: index-driven queries (Q1/Q2/Q6/Q7) are at parity. The +critical gaps are Q4 and Q8 — multi-hop join + group-by-aggregate that +materialize 428K and 655K intermediate bindings. Q3/Q5 are moderate gaps that +arise from row-at-a-time evaluation. + +**Write throughput cliff** (jerboa-db.md:2868): 154K ops/s at 5K entities +collapses to 1,300 ent/s at 147K entities — a 120× regression from RB-tree +depth + per-insert allocation in `lib/jerboa-db/index/memory.ss`. + +--- + +## 2. Targets + +| Metric | Today | Target | How | +|---|---|---|---| +| Q4 full-scale | ~250 s | **< 1 s** | DuckDB fallback | +| Q8 full-scale | ~205 s | **< 100 ms** | Aggregate pushdown + DuckDB | +| Q3/Q5 full-scale | 200–400 ms | **< 50 ms** | Aggregate pushdown | +| Bulk write sustained | 1,300 ent/s | **> 50K ent/s** | HAMT staging | +| Q1/Q2/Q6/Q7 | < 1 ms | **< 100 µs** (hot path) | `defquery` macro | +| LoC | ~8,600 | < 12,000 | discipline | + +**Non-goals**: matching XTDB v2 on cold-start columnar scans; persistent +LSM compaction tuning; distributed query (Phase 8 already covers cluster). + +--- + +## 3. Already in place (do not redo) + +| Optimization | Location | +|---|---| +| Hash-join planner + executor | `lib/jerboa-db/query/planner.ss:231`, `engine.ss:790` | +| Streaming aggregate fast path | `lib/jerboa-db/query/engine.ss:657` | +| Cardinality-based clause ordering | `lib/jerboa-db/query/planner.ss:97` | +| Range predicate pushdown | recent commit `25bef4d` | +| Streaming flatmap on bind expansion | recent commit `25bef4d` | +| 50× exact-match optimization | recent commit `868c0fc` | +| DuckDB columnar replica + sync | `lib/jerboa-db/analytics.ss` | +| Persistent RB-tree index | `lib/jerboa-db/index/memory.ss` | +| LevelDB persistent backend | `lib/jerboa-db/index/leveldb.ss` | + +The DuckDB replica already exists. We just don't *route Datalog queries to +it*. That's Phase 2 — the biggest single win in this plan. + +--- + +## 4. Phase 1 — Aggregate pushdown into scan (1–2 days) + +**The problem**. Q8 today: walk AEVT for `track/duration`, build 655K +binding tuples, then `streaming-aggregate` reduces them to 3 averages. The +allocation of 655K tuples dominates. + +**The fix**. Push `(count ?e)`, `(sum ?v)`, `(avg ?v)`, `(min ?v)`, `(max ?v)` +*into the index iterator itself*. The accumulator updates per-datom; no +binding tuple is allocated. + +**Scope**: +- Add `evaluate-pure-aggregate` in `lib/jerboa-db/query/engine.ss` that + detects single-clause + aggregate-only find spec and fuses the AEVT walk. +- Extend to grouped form: `[?status (avg ?d)]` where the grouping key is + also pulled from a single datom — bucket into hash table, no binding list. +- Wire into the query planner so it picks pushdown before falling back to + `streaming-aggregate` over a materialized list. + +**Expected wins** (1% scale): +- Q8: 966 ms → ~50 ms (20×) +- Q5: 2 ms → < 1 ms (already small; bigger at full scale) +- Q3: 1 ms → < 1 ms (range filter integrates here too) + +**Files touched**: `query/engine.ss`, `query/planner.ss` (new aggregate-only +plan node), `tests/test-core.ss` (new tests for grouped pushdown). + +**Risk**: low. Streaming aggregate is already correct; we're just hoisting +the loop one layer. + +--- + +## 5. Phase 2 — DuckDB analytical fallback (3–4 days) + +**The Jerboa moment.** No other Datalog database does this. Datomic Pro +ships with a separate analytics module ($$$); we route the exact same +`(q ...)` API through DuckDB transparently. + +**The architecture**. + +``` + (q '[:find ?status (avg ?d) ...] db) + | + [planner cardinality estimate] + | + high-card aggregation? ----no----> existing engine + |yes + translate-datalog->sql + | + analytics-engine (existing!) + | + DuckDB SQL + | + marshal rows -> Datalog tuples +``` + +**Why this is realistic**: +- The DuckDB replica + schema mirror already exists (`analytics.ss:46`). +- DuckDB FFI is wired and tested (`/tmp/duck-test.ss` runs). +- The translator only needs to handle a *subset* of Datalog — the queries + that produce > 50K intermediate rows. Everything else stays on the + Scheme engine. + +**Translator scope** (the meat): +- Datalog patterns `[?e :a ?v]` → `SELECT ... FROM datoms WHERE a_name='a'` +- Joins on shared variables → SQL joins on `e`/`v_ref` +- Predicates `[(< ?y 1960)]` → `WHERE v_long < 1960` +- Aggregates → SQL `COUNT/SUM/AVG/MIN/MAX` with `GROUP BY` +- Pull is *not* translated (stays on Scheme — point-query path) +- Rules are *not* translated (fallback to Scheme) + +**Auto-sync policy**: +- Lazy: on first analytical-fallback hit, run `analytics-sync!` from + `last-synced-tx` to current. +- Sync is incremental (already in the engine — only new datoms are upserted). +- Sync cost amortizes — a single Q4 run touches 13.1M datoms; sync of + the delta (typically thousands) is negligible. + +**Plan-time decision**: +- Estimate intermediate cardinality from per-attribute stats (already there). +- Threshold: > 50K → route to DuckDB. < 50K → stay on Scheme (lower latency). +- Override per-query with `:hint :analytics` or `:hint :datalog`. + +**Expected wins** (full scale): +- Q4: ~250 s → **< 1 s** (300× — columnar hash-join in DuckDB) +- Q8: ~205 s → **< 100 ms** (2000×) +- Q5 full scale: ~400 ms → ~30 ms + +**Files touched**: new `lib/jerboa-db/query/sql-translate.ss`, +`query/engine.ss` (dispatch hook), `query/planner.ss` (cardinality estimate +threshold), `analytics.ss` (incremental sync hook), tests. + +**Risk**: medium. The translator handles a subset, so fallback to the +Scheme engine on translation failure must be transparent. Need exhaustive +tests that prove "translated SQL gives identical results as Scheme engine" +on the MBrainz suite. + +--- + +## 6. Phase 3 — HAMT staging buffer for hot writes (3–5 days) + +**The problem**. `lib/jerboa-db/index/memory.ss` uses a sorted-map (RB-tree) +keyed on the full datom. Every `transact!` rebuilds a path from leaf to +root — `O(log n)` allocations per insert. At 147K entities the depth is +~17 and allocator pressure dominates. + +**The fix**. Two-tier index: +- **Hot tier**: HAMT (hash array-mapped trie) keyed on the natural index + prefix (E, A, V depending on which index). `O(log32 n)` ≈ flat below + 10M entries. Mutation is structural sharing with much higher fanout. +- **Cold tier**: existing RB-tree (or LevelDB on disk). +- **Flush**: at checkpoint (every N transactions or on demand), merge hot + HAMT into cold tier in one bulk operation. + +Reads consult both tiers (HAMT first, then RB-tree); a small bloom filter +on the hot tier short-circuits the common "not in hot" case. + +**Why HAMT specifically**: Chez has tagged pointers and excellent inline +`vector-ref`; a 32-way HAMT node is one allocation per level vs an RB-tree's +two (left + right + parent rebuild). Empirically HAMTs are ~5× faster on +mixed write workloads in Chez. + +**Expected wins**: +- Bulk load 147K entities: 113 s → ~3 s +- Sustained write throughput: 1,300 ent/s → 50K+ ent/s +- MBrainz load phase: 48 s → ~5 s (the bench currently spends most of its + time loading) + +**Files touched**: new `lib/jerboa-db/index/hamt.ss`, +`lib/jerboa-db/index/memory.ss` (compose hot+cold), tests. + +**Risk**: medium-high. Index correctness is critical. Need property-based +tests asserting `(hot+cold-search …)` ≡ `(rb-tree-search after flush)` for +random workloads. + +--- + +## 7. Phase 4 — `defquery` macro: compile-time specialization (3–5 days) + +**The Jerboa flex.** Show what Scheme macros uniquely enable. + +```scheme +(defquery artist-by-name [?name] + [?a :artist/name ?name] + [?a :artist/country ?c] + [?a :artist/startYear ?y]) +``` + +Expands at module-load to a procedure of `(db ?name) → list-of-tuples` +where: +- The clause graph is parsed at *compile* time +- The plan is computed at *compile* time +- AVET probe paths are inlined as direct procedure calls +- No interpreter dispatch, no string keyword lookup, no plan cache + +Hot-path queries (Q1, Q2, Q6, Q7) get sub-microsecond first-byte latency. +The same `(q ...)` API still works for ad-hoc queries — `defquery` is +opt-in for known-shape queries. + +**Why no other Datalog DB has this**: +- Datomic/Datahike use Clojure macros, but their query parser is a + function, not a macro — they can't see literal queries at compile time + in user code without macroexpander integration that doesn't exist. +- XTDB uses runtime parsing. +- Jerboa's `def-syntax` + procedural macros let us walk the query AST at + expansion time and emit specialized Chez code. + +**Expected wins**: +- Q1 hot loop: 0 ms (1% scale, 42 rows) stays 0 ms but with much lower + per-call overhead — visible in tight loops. Useful for application + code that runs the same query thousands of times. +- Demo: "your query *is* a Scheme procedure" — uniquely Jerboa. + +**Files touched**: new `lib/jerboa-db/query/defquery.ss` exporting the +macro, `lib/jerboa-db/core.ss` (re-export), examples and a benchmark. + +**Risk**: low. This is additive — old API unchanged. Mostly a macro-writing +exercise. + +--- + +## 8. Phase 5 — Benchmark + writeup (2–3 days) + +**Deliverable**: a one-command demo that closes the loop. + +- `make mbrainz` runs full-scale (262K artists) on Jerboa-DB and prints a + comparison table vs published Datahike numbers +- `make mbrainz-defquery` reruns Q1/Q2/Q6/Q7 with compiled queries to show + the macro speedup +- New section in `jerboa-db.md`: "Performance vs the field" with the final + table and a one-paragraph elevator pitch +- Update `README.md` from its one-line stub to a real intro: what jerboa-db + is, how to run the bench, where the wins come from + +**Final comparison target**: + +| Query | Jerboa-DB after plan | Datahike full | XTDB v2 | +|---|---:|---:|---:| +| Q1 | < 100 µs (defquery) | ~1 ms | ~1 ms | +| Q2 | < 100 µs (defquery) | ~1 ms | ~1 ms | +| Q3 | ~50 ms | ~5 ms | ~5 ms | +| Q4 | **< 1 s (DuckDB)** | 2–8 s | 0.5–2 s | +| Q5 | ~30 ms (DuckDB) | ~10 ms | ~10 ms | +| Q6 | < 100 µs (defquery) | ~1 ms | ~1 ms | +| Q7 | < 100 µs (defquery) | ~1 ms | ~1 ms | +| Q8 | **< 100 ms (DuckDB)** | 0.5–2 s | 0.2–0.5 s | + +**The story**: +- Match or beat Datahike on 6 of 8 queries +- Beat both Datahike and XTDB on Q4/Q8 (the analytics queries) by leveraging + DuckDB's columnar engine — same ergonomics, vastly better performance +- Behind on Q3/Q5 because we don't have native columnar storage; the + DuckDB fallback closes most of that +- All in ~12K lines of Scheme + +--- + +## 9. Sequencing and dependencies + +``` +Phase 1 (aggregate pushdown) ──┐ + ├──> immediate Q8 win, demoable alone +Phase 2 (DuckDB fallback) ──┤ (depends on cardinality stats) + │ +Phase 3 (HAMT writes) ──┴──> independent track, run in parallel + with Phase 1+2 + +Phase 4 (defquery macro) ─────> demo polish, do last +Phase 5 (bench + writeup) ─────> after 1+2 land +``` + +**Recommended order**: 1 → 2 → 3 → 4 → 5. Each phase ships standalone and +improves the demo. If time runs short, Phase 1+2 alone yield the headline +numbers; Phase 3+4 are upgrades. + +**Total budget**: 12–18 working days, single developer. + +--- + +## 10. What we explicitly skip + +| Idea | Why not | +|---|---| +| Full columnar value-store rewrite | Months. DuckDB fallback gets 90% of the benefit. | +| Vectorized executor (batch 1024 rows) | Months. Mostly relevant after columnar rewrite. | +| Persistent storage rewrite (replace LevelDB) | Working fine; not a bottleneck on the bench. | +| Distributed query | Out of scope. Cluster (Phase 8) covers replication. | +| Cost-based join reordering with histograms | Cardinality stats already give 80% of the value. | +| Compiling Datalog to native code via `eval` | `defquery` macro path covers this with cleaner semantics. | + +--- + +## 11. Risks and open questions + +**R1 — DuckDB sync staleness.** If a query runs immediately after a write, +the analytics replica is one tx behind. Mitigation: lazy `analytics-sync!` +on entry to the fallback path; cost is one upsert per new datom. + +**R2 — Translator coverage.** SQL translation will not handle every +Datalog shape (rules, recursive queries, custom predicates). Mitigation: +fail-closed — if any clause can't translate, fall back to Scheme engine. +Test by running the entire MBrainz suite both ways and asserting result +equivalence. + +**R3 — HAMT correctness.** A bug here corrupts the index. Mitigation: +property-based tests (random transact + query streams) before merging. +Keep the RB-tree path as a fallback flag (`(parameterize ([use-hamt #f]) …)`) +for at least one release. + +**R4 — `defquery` macro hygiene.** Walking user-supplied query forms and +emitting code is finicky. Mitigation: piggyback on the existing query +parser; the macro is a thin layer that calls the parser at expansion time +and emits a closure that wraps the existing engine procedures. + +**R5 — Demo machine variance.** MBrainz numbers depend heavily on RAM and +disk. Mitigation: publish numbers with hardware annotation (e.g., "M2 +MacBook Pro, 32GB, APFS") and include a `make mbrainz-quick` that anyone +can run in < 10s for sanity. + +--- + +## 12. Success criteria + +The plan succeeds when: + +1. `make mbrainz` runs at **full scale** (262K artists) and completes in + under 60 seconds wall-clock on a developer laptop. +2. Q4 and Q8 each complete in under 1 second at full scale. +3. The MBrainz comparison table in `jerboa-db.md` shows Jerboa-DB at parity + or better on 6+ queries vs Datahike. +4. `make test` passes all 37 core tests, plus new aggregate-pushdown tests, + plus new HAMT property tests. +5. The README has a paragraph that, in 30 seconds, communicates: "Datalog + ergonomics, DuckDB analytics, Scheme macros, in 12K lines." + +--- + +## 13. Open follow-ons (post-plan) + +If the demo lands and there's appetite for more: + +- **Live analytics dashboard**: DuckDB's Parquet export + a small HTTP + server (we already have one) → real-time SQL over the live database. +- **Query plan visualization**: the planner emits structured plan nodes; + render to a graphviz dot file. +- **Spec-driven query generation**: use `clojure.spec`-style schemas + (already in the codebase) to fuzz queries against the engine. +- **WebAssembly target**: Chez doesn't compile to wasm yet, but a subset + of jerboa-db could run under Chicken-on-wasm or similar. Speculative. new file mode 100644 --- /dev/null +++ b/status.md @@ -0,0 +1,135 @@ +# jerboa-db — status, comparison, and where we're going + +A single-page snapshot of where the project stands today, how it compares to +Datomic and Datahike, and what the plan is to close the remaining gap. For +the detailed engineering plan see [plan.md](plan.md). + +--- + +## 1. Where we stand today + +`master` at `5635268`. All 37 core tests passing. MBrainz benchmark runs +end-to-end at 1% scale (2,620 artists / 13,100 releases / 131,000 tracks). + +| Query | What it does | Now (1%) | Projected ×100 | Datahike full | +|---|---|---:|---:|---:| +| Q1 | Artist exact name lookup | 0 ms | <1 ms | ~1 ms | +| Q2 | Releases by artist name (2-hop) | 0 ms | <1 ms | ~1 ms | +| Q3 | `startYear < 1960` range | 1 ms | ~200 ms | ~5 ms | +| Q4 | Tracks > 240s on shared-artist releases | 1421 ms | **~250 s** | 2–8 s | +| Q5 | Releases per country (group-by) | 2 ms | ~400 ms | ~10 ms | +| Q6 | Reverse-ref lookup | 0 ms | <1 ms | ~1 ms | +| Q7 | Pull entity attrs | 0 ms | <1 ms | ~1 ms | +| Q8 | Avg track duration by status | 966 ms | **~205 s** | 0.5–2 s | + +Index-driven point queries (Q1/Q2/Q6/Q7) are at parity with Datomic-class +systems. The two critical gaps are Q4 and Q8 — multi-hop join + group-by +aggregate that materialize 428K and 655K intermediate bindings respectively. + +Write throughput cliff: 154K ops/s at 5K entities collapses to 1,300 ent/s +at 147K entities (jerboa-db.md:2868). Caused by RB-tree index allocation +pressure. + +--- + +## 2. Three databases, same menu (Datalog) + +**Datomic** — the expensive Michelin-star restaurant. Closed source, made by +the Clojure team, used in production at banks. Fast at everything, costs real +money, and you have to call ahead. + +**Datahike** — the same menu, made by an open-source diner. Free, runs on +your laptop, mostly delivers the same dishes. A bit slower on the steaks, +fine on the appetizers. Publishes MBrainz numbers we can compare to. + +**Jerboa-DB (us)** — a 10K-line food truck made of Scheme. Same menu. We're +already as fast as the diner on the appetizers, and we have a secret weapon: +a real industrial kitchen (DuckDB) parked out back that we haven't plugged +in yet. + +--- + +## 3. What "the menu" looks like + +| Dish | Q1/2/6/7: appetizers | Q3/5: side dishes | Q4/8: steaks | +|---|---|---|---| +| What it is | Look up one thing by name or id | Filter or count a column | Wide joins, group-by-aggregate | +| Datomic | ~1 ms | ~5–10 ms | 1–5 s | +| Datahike | ~1 ms | ~5–10 ms | 2–8 s | +| **Jerboa today** | < 1 ms (parity) | ~200–400 ms | **~250 s, ~205 s** | +| **Jerboa after plan** | **< 0.1 ms** | ~30–50 ms | **< 1 s** | + +--- + +## 4. ELI5 — why are we slow on the steaks? + +The steaks (Q4, Q8) ask things like *"what's the average song length grouped +by release status?"* across 13 million tracks. To compute this today, our +engine writes 655,000 little index cards onto a giant whiteboard, then counts +them up. That's the bottleneck. + +Datomic and Datahike use a fancier trick (hash-joins on a B+ tree) that +avoids most of those cards. They take seconds; we take minutes. + +**Our move**: don't try to out-clever them at their own game. Instead, send +those queries to **DuckDB** (a real columnar analytics engine, already wired +into our codebase). DuckDB eats wide aggregations for breakfast — Q8 on 13M +rows takes < 100 ms. **We end up faster than both Datomic and Datahike on +the hard queries**, by ~5–20×. Same Datalog API, just smarter routing under +the hood. + +--- + +## 5. What does Datomic do that we don't? + +| Feature | Datomic | Jerboa | +|---|---|---| +| Time travel (`as-of`, `since`) | yes | yes | +| ACID transactions | yes | yes | +| Distributed storage | DynamoDB, Postgres, etc. | LevelDB + Raft cluster | +| Production hardening | 10+ years | demo-grade | +| Datalog query | yes | yes | +| Pull syntax | yes | yes | +| Speed on Q1/2/6/7 | ~1 ms | < 1 ms | +| Speed on Q4/8 | seconds | seconds today, **sub-second after plan** | +| **Lines of code** | ~200K (Java + Clojure, closed) | **~10K (Scheme, open)** | +| **Cost** | $$$ | free | +| **Compile-time query specialization** | no — Clojure can't do it | **yes (planned, via Scheme macros)** | +| **Hybrid OLAP via DuckDB** | separate paid product | **built-in (planned)** | + +--- + +## 6. Targets after the plan lands + +| Metric | Today | Target | How | +|---|---|---|---| +| Q4 full-scale | ~250 s | **< 1 s** | DuckDB fallback | +| Q8 full-scale | ~205 s | **< 100 ms** | Aggregate pushdown + DuckDB | +| Q3/Q5 full-scale | 200–400 ms | **< 50 ms** | Aggregate pushdown | +| Bulk write sustained | 1,300 ent/s | **> 50K ent/s** | HAMT staging | +| Q1/Q2/Q6/Q7 hot path | < 1 ms | **< 100 µs** | `defquery` macro | +| LoC | ~8,600 | < 12,000 | discipline | + +--- + +## 7. Five-phase plan, 12–18 working days + +1. **Aggregate pushdown** (1–2 d) — fuse `(avg ?d)` etc. into the AEVT walk. + Q8 1% scale: 966 ms → ~50 ms. +2. **DuckDB analytical fallback** (3–4 d) — translate aggregation-heavy + Datalog to SQL, run on the existing DuckDB replica. Q4 full-scale: + ~250 s → < 1 s. +3. **HAMT staging buffer** (3–5 d) — fix the 120× write cliff with a + high-fanout hot tier. Bulk loads: 1,300 → 50K+ ent/s. +4. **`defquery` macro** (3–5 d) — Scheme macros compile a Datalog query to + a specialized procedure at module-load time. Sub-microsecond hot path. +5. **Bench + writeup** (2–3 d) — full-scale comparison, README rewrite. + +--- + +## 8. Bottom line, in one sentence + +After the plan, Jerboa-DB is *"Datomic-shaped, Datahike-priced, Scheme-elegant, +with a DuckDB turbo button bolted on"* — competitive on point queries, +**faster than both** on analytics, and the whole thing fits in the side +mirror of a Datomic installation.