status: update with engine-rebuild results + measured Datomic head-to-head
ober
5e3d862d4426026d6894f1b8e99c176b7893299e
--- a/status.md +++ b/status.md @@ -8,22 +8,28 @@ the detailed engineering plan see [plan.md](plan.md). ## 1. Where we stand today -`master` at `bdb00d5`. All 37 core tests passing. MBrainz benchmark runs -end-to-end at 1% and 5% scale, with a DuckDB fallback that auto-routes -analytical queries. +`master` at `88b261e`. All 52 core tests passing. MBrainz benchmark runs +end-to-end at 1% and 5% scale; a DuckDB fallback auto-routes analytical +queries, and group-by-aggregates auto-route to a native columnar operator. +Since the original snapshot the engine internals were rebuilt by mining +Datomic Pro's decompiled source — see §1a for the measured head-to-head and +[docs/datomic-comparison.md](docs/datomic-comparison.md). ### 1% scale (2,620 artists / 13,100 releases / 131,000 tracks) -| Query | What it does | Now (1%) | DuckDB fallback | Datahike full | +| Query | What it does | Datalog | DuckDB fallback | Datahike full | |---|---|---:|---:|---:| | Q1 | Artist exact name lookup | 0 ms | — | ~1 ms | | Q2 | Releases by artist name (2-hop) | 0 ms | — | ~1 ms | | Q3 | `startYear < 1960` range | 1 ms | — | ~5 ms | -| Q4 | Tracks > 240s on shared-artist releases | 1416 ms | **281 ms** | 2–8 s | +| Q4 | Tracks > 240s on shared-artist releases | 811 ms | **283 ms** | 2–8 s | | Q5 | Releases per country (group-by) | 1 ms | — | ~10 ms | | Q6 | Reverse-ref lookup | 0 ms | — | ~1 ms | | Q7 | Pull entity attrs | 0 ms | — | ~1 ms | -| Q8 | Avg track duration by status | 958 ms | **9 ms** | 0.5–2 s | +| Q8 | Avg track duration by status | 428 ms | **11 ms** | 0.5–2 s | + +The Datalog Q4/Q8 figures are after the join-engine rebuild (alist bindings + +per-query scan cache): Q4 1885 → 811 ms, Q8 1314 → 428 ms. ### 5% scale (13,100 artists / 65,500 releases / 655,000 tracks) @@ -32,8 +38,46 @@ analytical queries. | Q4 (2.14M rows) | 10018 ms | **1467 ms** | 6.8× | | Q8 | 6407 ms | **31 ms** | 207× | +### 1a. Head-to-head vs Datomic (measured, 1% scale, in-memory) + +Same machine, equivalent dataset, jerboa `:memory:` vs Datomic Pro `mem` peer +(method + reproduction in [docs/datomic-comparison.md](docs/datomic-comparison.md)): + +| | jerboa Datalog | jerboa + DuckDB | Datomic | +|---|---:|---:|---:| +| Load (~147k entities) | **1011 ms** | — | 1745 ms | +| Q1–Q3, Q5–Q7 | 0–1 ms | — | 0.4–9 ms | +| Q4 (multi-join, 428k rows) | 811 ms | **283 ms** | 368 ms | +| Q8 (join + aggregate) | **428 ms** | 11 ms | 566 ms | + +jerboa loads ~1.7× faster, is at parity or faster on every point/index query, +**now beats Datomic on Q8** (join + aggregate), and trails only on Q4 (~2.2×, a +428k-row result that DuckDB covers at 283 ms). The old "can't out-clever them on +joins" caveat no longer holds — switching bindings from copied hashtables to +O(1)-extend alists closed most of that gap. Q4's residual cost is materialising +~435k intermediate bindings; closing it needs tuple/column intermediate +relations, weighed against DuckDB already covering it. + ### What changed +**Engine rebuild (mined from Datomic Pro's decompiled internals):** +- Memory index is now a width-32 **B+-tree** (Datomic's `btset`), replacing the + red-black tree — 2.6× faster ordered scans at 500k datoms. +- Real per-(attribute,value) **cardinality** now drives clause ordering. +- **Columnar segments** (`TransposedData`-style) + a durable immutable + **segment-tree** backend (`:segtree:`): 19× smaller serialized, O(log n) + range counts, structural-sharing snapshots, trace GC. +- **Native columnar group-by** that `q` auto-routes to (single-hop + ref-hop), + alongside the DuckDB fallback. +- **Join engine:** per-query scan cache + bindings as O(1)-extend alists + + single-pass result relation — Q8 1314 → 428 ms, Q4 1885 → 811 ms; jerboa now + beats Datomic on Q8 (see §1a). +- `:limit` with streaming early-exit (top-N is O(log n + N)). +- Correctness: `:db/retractEntity` component cascade, `:db/ident` immutability + + unique/identity (idempotent schema), tempid/nested-map/lookup ref values. +- Test suite 37 → 52, zero regressions. + +**Original snapshot:** - Index-driven point queries (Q1/Q2/Q6/Q7) are at parity with Datomic-class systems. - Q3 and Q5 hold parity at 1% scale (≤ 1 ms) and remain in the low-ms @@ -60,10 +104,11 @@ money, and you have to call ahead. your laptop, mostly delivers the same dishes. A bit slower on the steaks, fine on the appetizers. Publishes MBrainz numbers we can compare to. -**Jerboa-DB (us)** — a 10K-line food truck made of Scheme. Same menu. We're -already as fast as the diner on the appetizers, and we have a secret weapon: -a real industrial kitchen (DuckDB) parked out back that we haven't plugged -in yet. +**Jerboa-DB (us)** — a 10K-line food truck made of Scheme. Same menu. As fast +as the diner on the appetizers, the DuckDB "industrial kitchen out back" is now +plugged in for the steaks — and after rebuilding our own kitchen from the +Michelin restaurant's recipes (decompiled Datomic), we now out-cook it on one +steak (Q8) even without DuckDB. --- @@ -74,27 +119,28 @@ in yet. | What it is | Look up one thing by name or id | Filter or count a column | Wide joins, group-by-aggregate | | Datomic | ~1 ms | ~5–10 ms | 1–5 s | | Datahike | ~1 ms | ~5–10 ms | 2–8 s | -| **Jerboa today** | < 1 ms (parity) | ~200–400 ms | **~250 s, ~205 s** | -| **Jerboa after plan** | **< 0.1 ms** | ~30–50 ms | **< 1 s** | +| **Jerboa today (Datalog)** | < 1 ms (parity) | **≤ 1 ms** | **~0.8 s / ~0.4 s** | +| **Jerboa + DuckDB** | — | — | **~0.3 s / ~0.01 s** | --- -## 4. ELI5 — why are we slow on the steaks? +## 4. ELI5 — the steaks, revisited -The steaks (Q4, Q8) ask things like *"what's the average song length grouped -by release status?"* across 13 million tracks. To compute this today, our -engine writes 655,000 little index cards onto a giant whiteboard, then counts -them up. That's the bottleneck. +The steaks (Q4, Q8) ask things like *"what's the average song length grouped by +release status?"* — they build hundreds of thousands of intermediate rows. That +used to take us *minutes*, because every intermediate row copied a whole +hashtable. -Datomic and Datahike use a fancier trick (hash-joins on a B+ tree) that -avoids most of those cards. They take seconds; we take minutes. +We fixed the representation: each intermediate row is now a cheap O(1) `cons` +(an alist), not a copied map. So our plain Datalog now does these in **under a +second — and we're actually faster than Datomic on Q8** (428 vs 566 ms). We no +longer need to out-clever anyone there; we just stopped doing the expensive +thing. -**Our move**: don't try to out-clever them at their own game. Instead, send -those queries to **DuckDB** (a real columnar analytics engine, already wired -into our codebase). DuckDB eats wide aggregations for breakfast — Q8 on 13M -rows takes < 100 ms. **We end up faster than both Datomic and Datahike on -the hard queries**, by ~5–20×. Same Datalog API, just smarter routing under -the hood. +For the very heaviest aggregations we still hand off to **DuckDB** (a columnar +engine wired in behind the same Datalog API) — Q8 on 13M rows in ~30 ms. Fast on +our own now, faster still with the turbo button — the same way Datomic leans on +Presto for these shapes. --- @@ -109,11 +155,11 @@ the hood. | Datalog query | yes | yes | | Pull syntax | yes | yes | | Speed on Q1/2/6/7 | ~1 ms | < 1 ms | -| Speed on Q4/8 | seconds | seconds today, **sub-second after plan** | +| Speed on Q4/8 | seconds | **sub-second** (Datalog; Q8 beats Datomic) / **< 100 ms** via DuckDB | | **Lines of code** | ~200K (Java + Clojure, closed) | **~10K (Scheme, open)** | | **Cost** | $$$ | free | | **Compile-time query specialization** | no — Clojure can't do it | **yes (planned, via Scheme macros)** | -| **Hybrid OLAP via DuckDB** | separate paid product | **built-in (planned)** | +| **Hybrid OLAP via DuckDB** | separate paid product | **built-in (shipped)** | --- @@ -143,7 +189,14 @@ the hood. | 2.3 | Auto-route via `current-analytics-engine` parameter | `66ba53c` | transparent 5×/88× on Q4/Q8 | | 3 | LSM staging + tempid hashtable + fulltext O(n²) fix | `6207c2d` | 47× ingest | | 4 | `defquery` macro | — | deferred (auto-routing covers it) | -| 5 | Final benchmark + writeup | `bdb00d5` + this | parity story | +| 5 | Final benchmark + writeup | `bdb00d5` | parity story | +| 6 | B+-tree memory index + real-cardinality planner | `635c5c2`,`042a197` | 2.6× scans, better clause ordering | +| 7 | Lazy cursors + `:limit` streaming early-exit | `696cdd7`,`3dc3ac0` | top-N O(log n + N) | +| 8 | Columnar segments + native group-by + `q` auto-route | `92deeb9`,`bb573b4`,`0e1cd69` | 19× smaller; Q8-shape 5.6× | +| 9 | Durable segment-tree backend (`:segtree:`) | `3062a42`,`607bffc` | snapshots, trace GC, columnar leaves | +| 10 | Correctness: component cascade, ident immutability/unique, ref values | `9b5e501`,`5004d5b`,`7300ea6` | Datomic-parity semantics | +| 11 | Join engine: scan cache + alist bindings + result relation | `965dac1`,`fa7acd3`,`c3f068b` | Q8 1314→428 (beats Datomic), Q4 1885→811 | +| 12 | Measured Datomic head-to-head | `9db9d6b`,`b601142` | [docs/datomic-comparison.md](docs/datomic-comparison.md) | See [plan.md](plan.md) §0 for the full results table. @@ -152,6 +205,7 @@ See [plan.md](plan.md) §0 for the full results table. ## 8. Bottom line, in one sentence Jerboa-DB is *"Datomic-shaped, Datahike-priced, Scheme-elegant, with a DuckDB -turbo button bolted on"* — at parity on the appetizers, **faster on the -steaks**, ingest 47× faster than the start of the plan, and the whole thing -still fits in the side mirror of a Datomic installation. +turbo button"* — and in a measured head-to-head it **loads faster than Datomic, +ties or beats it on every query except one wide join** (which DuckDB covers), +and even **beats it on the Q8 join+aggregate in plain Datalog**. All in ~10K +lines of open Scheme that still fits in the side mirror of a Datomic install.