plan: record round 2 (Datomic-internals port) + correct stale facts
ober
d687bc98efa13429b0c664eceae79ca8257c2f8a
--- a/plan.md +++ b/plan.md @@ -8,6 +8,11 @@ This document supersedes any prior plan in this file. It assumes the current state of `master` at commit `5635268` (post-Phase-8, all 37 core tests passing, hash-join + streaming aggregate + range pushdown already in the engine). +> **Update:** the original plan (Phases 1–5) shipped, and a second round then +> rebuilt the engine internals by mining Datomic Pro's decompiled source — see +> **§0a** for the round-2 results and what they changed. `master` is now at +> `b4010b4`, **52 core tests** passing. + --- ## 0. Results — what actually shipped @@ -102,10 +107,69 @@ sorted (e,a,v,tx) order and keeping only the highest-tx datom per --- +## 0a. Round 2 — Datomic-internals port (shipped) + +After the original plan landed, a second round mined **Datomic Pro's decompiled +internals** (full 15-item analysis in [docs/datomix-idea.md](docs/datomix-idea.md); +measured head-to-head in [docs/datomic-comparison.md](docs/datomic-comparison.md)). +Headline: jerboa-db is now **competitive with Datomic across the board** at 1% +scale — faster on load + OLTP, and **faster than Datomic on the Q8 join+aggregate +in plain Datalog** (428 vs 566 ms), with Q4 the lone holdout (~2.2×, DuckDB-covered). + +### Datalog query times moved (1% scale) + +| Query | Round 1 | Round 2 | What changed | +|---|---:|---:|---| +| Q4 (multi-join) | 1416 ms | **811 ms** | alist bindings + per-query scan cache | +| Q8 (join + agg) | 958 ms | **428 ms** | alist bindings + scan cache (now beats Datomic's 566 ms) | + +Q1–Q3/Q5–Q7 stayed sub-/low-ms; load ~1 s; DuckDB fallback unchanged (Q4 ~283 ms, Q8 ~11 ms). + +### Phases shipped (round 2) + +| # | Phase | Commits | Win | +|---|---|---|---| +| 6 | B+-tree memory index (Datomic `btset`) + real-cardinality planner | `635c5c2`,`042a197` | 2.6× scans; resolved the write cliff | +| 7 | Lazy cursors + `:limit` streaming early-exit | `696cdd7`,`3dc3ac0` | top-N O(log n + N) | +| 8 | Columnar segments + native group-by + `q` auto-route | `92deeb9`,`bb573b4`,`0e1cd69` | 19× smaller; Q8-shape 5.6× | +| 9 | Durable segment-tree backend (`:segtree:`) | `3062a42`,`607bffc` | snapshots, trace GC, columnar leaves | +| 10 | Correctness: component cascade, ident immutability/unique, ref values | `9b5e501`,`5004d5b`,`7300ea6` | Datomic-parity semantics | +| 11 | Join engine: scan cache + alist bindings + single-pass result relation | `965dac1`,`fa7acd3`,`c3f068b` | Q4/Q8 ~halved; Q8 beats Datomic | +| 12 | Measured Datomic head-to-head + bench scripts | `9db9d6b`,`b601142`,`b4010b4` | `docs/datomic-comparison.md` | + +### What we learned (round 2) + +- **The write cliff WAS partly the RB-tree after all.** Round 1 fixed the + fulltext O(n²) hot spot; replacing the red-black tree with a width-32 + **B+-tree** (Datomic's `btset`) then gave 2.6× faster ordered scans and + removed the depth/allocation pressure §1 flags. This supersedes the Phase-3 + HAMT plan — the B+-tree was the simpler, Datomic-proven choice. +- **The Datalog join gap was representational, not algorithmic.** Bindings were + hashtables copied on every extension; switching to O(1)-extend **alists** + (contained to `engine.ss`) roughly halved Q4/Q8 and made Q8 beat Datomic. A + dedup-disabled probe showed Q4's residual ~2.2× is intermediate-binding + *materialization* (~435k rows), not the join algorithm. +- **Datomic itself offloads OLAP to Presto** — confirming the round-1 DuckDB + strategy; the `datomic-pro` distribution ships a Presto server for exactly the + Q4/Q8 shapes. + +### Still open (round 2) + +- **Tuple/column intermediate relations** for Q4 (the one query where Datomic + still leads ~2.2×) — a large representational change, and Q4 is already + DuckDB-covered. +- **Disk-backed segment store** to make `:segtree:` a real persistent backend + (the store is currently in-memory; the `:memory:` default stays the fast + B+-tree, since serializing has no payoff without disk). +- `defquery` (Phase 4) still deferred. + +--- + ## 1. Where we stand today -MBrainz, 1% scale (2,620 artists / 13,100 releases / 131,000 tracks), one -warm run on `master`: +MBrainz, 1% scale (2,620 artists / 13,100 releases / 131,000 tracks). *This is +the round-1 snapshot that motivated the plan; current Datalog numbers are in +§0a (Q4 811 ms, Q8 428 ms).* | Query | What it does | Now (1%) | Projected ×100 | Datahike full | |---|---|---:|---:|---:| @@ -124,8 +188,9 @@ materialize 428K and 655K intermediate bindings. Q3/Q5 are moderate gaps that arise from row-at-a-time evaluation. **Write throughput cliff** (jerboa-db.md:2868): 154K ops/s at 5K entities -collapses to 1,300 ent/s at 147K entities — a 120× regression from RB-tree -depth + per-insert allocation in `lib/jerboa-db/index/memory.ss`. +collapsed to 1,300 ent/s at 147K entities — a 120× regression from RB-tree +depth + per-insert allocation. **Resolved in round 2 (§0a):** the memory index +is now a width-32 B+-tree (Datomic's `btset`) in `lib/jerboa-db/index/memory.ss`. --- @@ -156,7 +221,7 @@ LSM compaction tuning; distributed query (Phase 8 already covers cluster). | Streaming flatmap on bind expansion | recent commit `25bef4d` | | 50× exact-match optimization | recent commit `868c0fc` | | DuckDB columnar replica + sync | `lib/jerboa-db/analytics.ss` | -| Persistent RB-tree index | `lib/jerboa-db/index/memory.ss` | +| Persistent **B+-tree (`btset`)** index — round 2; was RB-tree | `lib/jerboa-db/index/memory.ss` | | LevelDB persistent backend | `lib/jerboa-db/index/leveldb.ss` | The DuckDB replica already exists. We just don't *route Datalog queries to @@ -264,6 +329,10 @@ on the MBrainz suite. ## 6. Phase 3 — HAMT staging buffer for hot writes (3–5 days) +> **Superseded in round 2 (§0a).** The write cliff was resolved by switching the +> memory index to a width-32 B+-tree (Datomic's `btset`) — simpler than a HAMT +> hot/cold tier and Datomic-proven. The analysis below is kept for context. + **The problem**. `lib/jerboa-db/index/memory.ss` uses a sorted-map (RB-tree) keyed on the full datom. Every `transact!` rebuilds a path from leaf to root — `O(log n)` allocations per insert. At 147K entities the depth is @@ -453,8 +522,9 @@ The plan succeeds when: 2. Q4 and Q8 each complete in under 1 second at full scale. 3. The MBrainz comparison table in `jerboa-db.md` shows Jerboa-DB at parity or better on 6+ queries vs Datahike. -4. `make test` passes all 37 core tests, plus new aggregate-pushdown tests, - plus new HAMT property tests. +4. `make test` passes all core tests (now **52**, up from 37), plus the + round-2 tests (B+-tree, cursors/`:limit`, columnar segments, native group-by, + segment-tree, component cascade, ident immutability, durable backend). 5. The README has a paragraph that, in 30 seconds, communicates: "Datalog ergonomics, DuckDB analytics, Scheme macros, in 12K lines."