docs: add datomix-idea report (Datomic internals -> jerboa-db)
ober
8875fcb865368c3b0fa23034845f36011cfdc846
new file mode 100644 --- /dev/null +++ b/docs/datomix-idea.md @@ -0,0 +1,432 @@ +# Datomix ideas — what jerboa-db can learn from Datomic Pro internals + +> Reverse-engineered from **Datomic Pro 1.0.7622** (`peer-1.0.7622.jar`), +> decompiled with **CFR** from Clojure AOT bytecode. Source map and method are in +> the appendix. This is a design-mining report, not a port plan. + +**Read this first.** jerboa-db is already a mature Datomic-class system. A +subsystem-by-subsystem comparison shows it *already* implements, correctly, most +of what makes Datomic fast and correct: four covering indexes, sortable byte +codec (big-endian ids + sign-flipped doubles), predicate pushdown into range +scans, live-variable projection between clauses, streaming mutable-accumulator +aggregation, index selection by bound components, `not-join` scoping, unique +upsert via AVET, `:db/cas`, cardinality-one auto-retract with a tx-local cache, +lazy per-attribute entity loading, ident interning, and reverse-refs via VAET. +Those are called out in §6 so nobody re-does them. + +This report concentrates on the **genuine deltas** — places where Datomic's +internal design differs from jerboa-db's current code and the difference is worth +copying. Confidence is flagged per item; "verified" means I checked the relevant +jerboa-db source. + +--- + +## TL;DR — ranked + +| # | Idea | jerboa-db today | Impact | Effort | +|---|------|-----------------|--------|--------| +| 1 | Memory index = **B+-tree**, not red-black | RB-tree (`index/memory.ss`) | High (GC, scans, bulk load) | Med | +| 2 | Index reads = **lazy cursors**, not lists | `dbi-range`/`dbi-seek` return lists | High (wide joins Q4/Q8) | Med | +| 3 | Durable segments = **columnar (transposed)** | per-datom FASL in LevelDB | High (analytics gap, compression) | High | +| 4 | **Range cardinality from segment metadata** | `dbi-count` scans | High (planner quality) | Med | +| 5 | **Immutable id-addressed segments + snapshots** | per-datom LevelDB put | High (snapshots, GC, as-of) | High | +| 6 | **Byte-aware, sizing-based cache** | count-limited LRU | Med (predictable memory) | Low | +| 7 | **Hash-join execution** | planner detects, engine doesn't run it | Med (multi-unbound joins) | Med | +| 8 | **Cardinality-aware clause deferral** (`reqcnt`) | selectivity-only scoring | Med (avoids blow-ups) | Low | +| 9 | **Pull batching** (kill N+1) | per-ref recursive index scans | Med (pull-heavy reads) | Med | +| 10 | **Component cascade retraction** | *not implemented* (verified) | Correctness | Low | +| 11 | **Partitioned, bit-packed entity ids** | global counter | Med (locality, sharding) | Med | +| 12 | **Reified tx + log as a datom source** | emits `:db/txInstant`; log yields tx-reports | Med (audit, CDC, replication) | Med | +| 13 | **External sort for bulk index build** | none | Med (restore/migrate at scale) | Med | +| 14 | **Pluggable storage SPI** | LevelDB hardcoded | Low (testing, backends) | Low | +| 15 | **Ident-immutability enforcement** | not enforced | Correctness/safety | Low | + +--- + +## 0. The architecture in one picture + +Each of Datomic's four indexes (EAVT / AEVT / AVET / VAET — internally `:raet`) +is **two layers, merged at read time**: + +``` + query → seek-datoms(index, components) + → normalize components to one [e a v t] tuple + per-index comparator + → MERGE-ITERS( memory-index-cursor , durable-tree-cursor ) ; lazy k-way + → windowed( ... ) ; streaming as-of/t cut + assert/retract dedup + → datoms +``` + +- **Memory layer**: a `btset` (B+-tree) of the datoms added since the last + indexing job. Small, fast, fully in RAM. +- **Durable layer**: a shallow immutable tree stored as **segments in a KV + store** — `RootNode → DirNode → leaf segment` — each node referencing children + by **segment id**. Reading = binary-search root keys → load dir → binary-search + dir keys → load segment → binary-search segment. +- **Indexing job** (background): merge-sorts the memory layer into the durable + tree, writes *new* immutable segments, swaps in a new root, resets the memory + layer. Old segments are immutable and shared; GC reclaims unreferenced ones. + +Two consequences worth internalizing: (a) a `db` value is just *(durable root ids ++ memory btset + basisT)* — so snapshots, `as-of`, `since`, and speculative +`with` are O(1) and share structure; (b) history is not a separate store — it's +the same indexes read through a different filter. + +jerboa-db today writes datoms straight into LevelDB per-datom and keeps a memory +RB-tree; items 1–5 below close the gap to this architecture. + +--- + +## 1. Memory index: B+-tree (`btset`), not red-black · *high* · verified + +**Datomic** — `datomic.btset` (`btset/BTSet.java`, `BTSetLeaf.java`, +`BTSetBranch.java`). A persistent **B+-tree**, width 16: +- `BTSetLeaf{cnt, cmp, ks=Object[]}` — sorted keys, ≤16. +- `BTSetBranch{cmp, nks=Object[]}` — interleaved `[child0, sep0, child1, sep1,…]`; + separators are routing keys (min-key of the right child). +- Insert path-copies one root→leaf path; overflow **splits** (half = 8) and + bubbles a `BTSetSplit{left, k, right}` upward; a root split grows height. Set + semantics: `compare == 0` ⇒ no-op (`conjoin` returns `this`). +- **Append fast-path** (`BTSetLeaf.conjoin`): appending at the end of a *full* + leaf returns `BTSetSplit(this, k, newLeaf[k])` — a fresh right sibling with **no + array copy**. In-order/sorted inserts become near-linear. +- Iteration via explicit path-link cursors (`BTSetIter` / `BTSetIterLink` = + node + index + parent link) ⇒ O(1)-amortized `next`/`prev`, no recursion. + +**jerboa-db** — `lib/jerboa-db/index/memory.ss`: an Okasaki **red-black tree**, +node `#(color left key val right)`, fanout 2. + +**Why switch.** At 1M datoms an RB-tree is ~40 deep with 2-way fanout; a width-32 +B+-tree is ~4 deep. Each insert copies a handful of *flat arrays* (cache-friendly, +sequential) instead of many 5-slot vectors plus rotation nodes — fewer +allocations and far better locality (the "RB allocation pressure" status.md +flagged). The append fast-path, combined with **sorting each tx batch into index +order before insert**, makes bulk load cheap. + +**Recommendation.** Port `BTSetLeaf`/`BTSetBranch`/`BTSet` to `index/memory.ss` +(width 32). Chez `vector`/`fxvector` map directly onto the `Object[]` nodes. +Closest reference implementation: Tonsky's `persistent-sorted-set` (the +open-source twin of this exact design; Datahike uses it). Keep the datom +comparator you already have. While here, add a per-node subtree-count for item 4. + +## 2. Index reads are lazy cursors, not lists · *high* · verified + +**Datomic** — `datomic.iter`: `Iter` is a cursor (`get`/`next`/`prev`). +`merge-iters` composes binary `MergeIter`s into a k-way merge (`least-index` +compares the two heads, advances the smaller). `filter`/`take-while`/ +`drop-while`/`map` are all streaming. `seek-datoms` returns this lazy iterable; +nothing materializes until consumed. + +**jerboa-db** — `lib/jerboa-db/index/protocol.ss`: `dbi-range` and `dbi-seek` +return **fully materialized lists**. + +**Why switch.** Materializing every clause's matches into a list is the cost +driver on wide joins and group-bys — exactly Q4 (2.14M rows) and Q8, the queries +you currently divert to DuckDB. Lazy cursors let `query/engine.ss` do streaming +merge-joins and honour `LIMIT` / early-exit without building intermediate lists. + +**Recommendation.** Change the index protocol to return ordered **cursors** +(Chez: a struct with `peek`/`advance!`/`at-end?`, or a generator closure). Have +`dbi-range`/`dbi-seek` produce cursors; provide a `cursor->list` for callers that +still want eager results. Then add a streaming k-way `merge-cursors` (port +`merge-iters` + `least-index`) and use it in the engine's join loop. This pairs +naturally with item 1 (B+-tree path-link cursors are the cursor implementation). + +## 3. Durable segments are columnar (transposed) · *high* · verified + +**Datomic** — leaf segments and node key-data are **`TransposedData`** +(`index/TransposedData.java`, `index/ITransposeData.java`): column arrays, not row +structs — `eas` (entity+attr packed), `vs` (values), plus t/op columns, with +**type-specialized primitive accessors** `getLongV` / `getIntV` / `getDoubleV` / +`getFloatV` and `isAssertion(i)`. Homogeneous columns avoid boxing entirely. + +**jerboa-db** — `index/leveldb.ss`: each datom is individually FASL-encoded +(`datom->fasl-bytevector`) and stored as its own LevelDB value. + +**Why switch.** Column layout gives (a) strong compression — sorted runs of +identical `e`/`a` and monotone `t` compress with delta/RLE; (b) fast scans over a +single attribute's values; (c) **cardinality and range stats without decoding +values** (item 4). This is the columnar property you currently rent from DuckDB; +owning it in the segment format shrinks that gap and feeds the planner. + +**Recommendation.** Introduce a segment as the unit of durable storage (item 5), +and encode each segment column-transposed: parallel arrays for e, a, v +(primitive-typed per attribute value-type), t, op; plus a small header (count, +min/max key, per-column codec). Store the whole segment as one value. Keep FASL +only for heterogeneous fallback columns. + +## 4. Range cardinality straight from the directory · *high* · verified + +**Datomic** — `DirNode{keydata, segids, offsets[], counts[], segs}` carries per- +segment `offsets`/`counts`, so Datomic computes the size of any key-range +(`bounded-count`, `seg-count-cumsum`) **without loading a single segment**. + +**jerboa-db** — has `dbi-count`, but on the RB-tree it walks the range; the +planner (`query/planner.ss`, `stats.ss`) leans on heuristics. (Agent note: verify +how much real statistics the planner consumes today.) + +**Why it matters.** Correct clause ordering is most of query performance, and it +needs *true* selectivity: count of `[?e :a _]`, count of `[?e :a ?v]` over a +value range. Get it in O(1)–O(log n) from metadata instead of scanning. + +**Recommendation.** Two complementary sources: (a) make the memory B+-tree an +*order-statistic* tree — store subtree size per node (free during item 1), giving +O(log n) rank/count for any range; (b) read counts from columnar segment headers +(item 3) for the durable layer. Feed both into `planner.ss` clause scoring, +replacing heuristics with measured cardinality. + +## 5. Immutable, id-addressed segments → cheap snapshots & GC · *high* + +**Datomic** — the durable tree is segments addressed by id in a KV store with a +tiny SPI (`kv_store__init.java`): `get`, `put` (with an optional `:ensure` +compare-and-set precondition), `delete`, `close`. Segments are immutable blobs; +nodes reference children by id. A `db` value is a set of root ids + memory index, +so every snapshot shares all unchanged segments. Unreferenced segments are +reclaimed by a trace/mark GC (`kv_cluster$mark_pod_garbage.java`). + +**jerboa-db** — `index/leveldb.ss` does per-datom `leveldb-put`/`delete`; there's +no segment/snapshot abstraction; `gc.ss` does logical retraction cleanup (scans +EAVT, drops fully-retracted `(e,a,v)` groups), not physical segment reclamation. + +**Why switch.** Immutable id-addressed segments are what make O(1) `db` snapshots, +`as-of`/`since`, and speculative `with` possible with structural sharing — and +they turn GC into "sweep segment ids no live root references" instead of scanning +datoms. It also decouples you from LevelDB's per-key model. + +**Recommendation.** Layer a segment store over LevelDB: segments keyed by id +(uuid/content-hash), each holding a transposed datom batch (item 3); index nodes +reference children by id. Persist a `db` as its root ids + basisT. Split GC into +logical (retraction cleanup, today) and **physical** (mark segment ids reachable +from any root retained within the history window; sweep the rest). Use LevelDB's +atomic write for the root swap; if you later want lock-free multi-writer, mirror +Datomic's `:ensure` CAS precondition on the root pointer. + +## 6. Byte-aware, sizing-based cache · *med* · verified + +**Datomic** — segment/object caches are **byte-budgeted**; `datomic.memory_size` +recursively estimates object footprints (arrays, strings, keywords, collections) +to enforce the ceiling, and there's a two-tier story (decoded object cache over a +compressed segment cache). + +**jerboa-db** — `cache.ss` is an LRU limited by **entry count** (`capacity`), with +no size awareness. + +**Why switch.** Entry-count limits make memory unpredictable when value sizes vary +(a cache of 10k tiny idents vs 10k large text values differ by orders of +magnitude). Byte budgeting gives a predictable footprint and better hit rates +under a fixed RAM target. + +**Recommendation.** Track an approximate byte size per entry (you already encode +to bytevectors — reuse that length; sample 1% for variable values) and evict by +total bytes. Optional second tier: when the decoded cache is full, fall back to a +compressed segment cache (pairs with item 3) before hitting storage. + +## 7. Hash-join execution · *med* + +**Datomic** — builds relations and hash-joins on shared variables +(`datalog$create_join_maps`, `join_project_coll`) rather than always nesting. + +**jerboa-db** — `planner.ss` has a `hash-join-plan` that *detects* joinable pairs, +but `engine.ss` executes clauses as a sequential `evaluate-data-pattern` → +flatmap pipeline (effectively nested-loop). (Agent finding; worth confirming.) + +**Recommendation.** Wire the planner's hash-join decision into the engine: when +two clauses share an unbound variable and neither side is a cheap point lookup, +materialize the smaller side into a hashtable keyed by the shared var, then probe +while streaming the larger side. Biggest win when both entities are unbound +(`[?e1 :a ?x] [?e2 :b ?x]`). + +## 8. Cardinality-aware clause deferral (`reqcnt`) · *med* + +**Datomic** — `datalog$sched_in_order` attaches a `reqcnt` ("required count") +per clause and treats a clause as *underbound* if its free vars aren't bound and +its expected row count is high — deferring it so it doesn't multiply rows. + +**jerboa-db** — `planner.ss` scores by selectivity (entity-bound, unique-attr, +value-bound, …) but doesn't penalize a high-cardinality clause that is currently +unbound; it can pick a row-exploding clause too early. + +**Recommendation.** Add a growth term to clause scoring: if a clause is unbound +and its attribute is cardinality-many or high-count (use item 4's real counts), +heavily discount it until a binding arrives. Cheap to add to the existing +scoring loop, and it prevents accidental Cartesian blow-ups. + +## 9. Pull: batch index lookups to kill N+1 · *med* + +**Datomic** — the pull engine prefetches: it groups ref attributes and resolves +constituents in a batched pass (`pull$index_pull`, `prefetch-constituents` / +`prefetch-identity` in `db__init`) instead of fetching each ref one at a time. + +**jerboa-db** — `query/pull.ss` recurses per ref attribute, each triggering its +own EAVT `dbi-range`; a pattern with N refs over D levels is ~O(N·D) index ops. + +**Recommendation.** At each depth, collect the entities needing the same ref +attribute and do one grouped scan (AEVT for "all `(e,a)` of an attr", or batched +EAVT seeks), then distribute results before recursing. Memoize within a single +pull. Pairs well with cursors (item 2). + +## 10. Component cascade retraction · *correctness* · verified missing + +**Datomic** — `builtins$component_es_set.java` walks `:db/isComponent` refs +transitively; retracting a component entity **cascade-retracts** its sub-tree. + +**jerboa-db** — `:db/isComponent` is stored and used to flatten nested maps on +assert (`tx.ss`), but there is **no cascade on retract** (grep for cascade/ +component-retract in `tx.ss` returns nothing). Retracting a parent orphans its +components. + +**Recommendation.** Implement `component-entity-set db eid` (transitive walk over +component attrs via VAET/EAVT) and, when retracting an entity, also retract the +reachable component sub-tree. Mirrors Datomic semantics; small and self-contained. + +## 11. Partitioned, bit-packed entity ids · *med* · verified absent + +**Datomic** — entity ids encode a **partition** in the high bits plus a per- +partition sequence (`db$eid->part`, `db$default-partition`, `db$calc-tempids`); +`:db.part/db` (schema), `:db.part/tx` (transactions), `:db.part/user` (default). +Co-locates related entities in index order and namespaces allocation. + +**jerboa-db** — entity ids are a global monotonic counter (`core.ss`); encoding.ss +bit-packs the *retract bit* (bit 63 of t) and sortable doubles, but ids carry no +partition. (Verified: no partition logic in `encoding.ss`/`core.ss`.) + +**Recommendation.** Reserve high bits of the eid for a partition tag and allocate +per partition. Immediate benefit: schema (`:db.part/db`) and tx entities +(`:db.part/tx`, item 12) cluster together and away from user data, improving +EAVT/AEVT locality; longer-term it's the natural seam for sharding. Keep ids +bytewise-sortable (partition in the high bits preserves order). + +## 12. Reified transactions + log as a datom source · *med* · partial today + +**Datomic** — every transaction is itself an entity (in `:db.part/tx`) carrying +`:db/txInstant` (and you can add `:db/txUser`, etc.); the log is an append-only +sequence of `{t, datoms}` queryable as a first-class datom source (`tx-range`, +`tx-ids`), enabling audit, CDC, and replication without rebuilding indexes. + +**jerboa-db** — already emits `:db/txInstant` (`tx.ss`), and `log-tx-range` +exists — but it returns **tx-reports**, not a `[e a v t op]` datom stream, and the +tx entity isn't a queryable schema entity. (Verified.) + +**Recommendation.** (a) Allocate tx ids in a `:db.part/tx` partition (item 11) and +treat the tx entity as a normal entity so `:db/txInstant` is queryable in Datalog. +(b) Add a datom-level `tx-range`/`tx-since` that streams `[e a v t op]` from the +log (lazily, hot segments cached) for replication/CDC and point-in-time replay. + +## 13. External sort for bulk index builds · *med* · absent + +**Datomic** — `external_sort$file_system_sorter.java`: chunk an unsorted datom +stream, sort each chunk in memory (bounded by `:max-chunk-size`, using an +`:item-sizer`), spill to temp files, then k-way merge — so index builds and big +merges never OOM. + +**jerboa-db** — indexes are populated incrementally via `transact!`; no bulk +build/external-sort path. + +**Recommendation.** Add an external-sort-backed bulk loader for restore, schema +migration, and initial benchmark loads (mbrainz). Bound memory by chunk size, +sort chunks into index order, k-way merge into segments (items 3/5). Complements +the B+-tree append fast-path (item 1) for the in-memory portion. + +## 14. Pluggable storage SPI · *low* · verified + +**Datomic** — a 4-op KV SPI (`get`/`put`+`:ensure`/`delete`/`close`) with many +backends (in-mem, DDB, SQL, Cassandra). jerboa-db hardcodes `(std db leveldb)` in +`index/leveldb.ss`. + +**Recommendation.** Define a storage protocol mirroring those four ops (including +a CAS precondition for the root pointer) and make LevelDB one implementation. +Enables an in-memory backend for tests, and future RocksDB/LMDB/S3 backends. + +## 15. Enforce ident immutability · *correctness* · likely absent + +**Datomic** — `db$prevent_ident_retarget_BANG_.java` rejects transactions that +re-point an existing `:db/ident` to a different entity. jerboa-db's +`schema-intern-attr!` caches ident→id but doesn't block reassignment/retraction. + +**Recommendation.** In the tx pipeline, reject datoms that change/retract a +`:db/ident` on an entity that already has one. Cheap guard against schema +corruption. + +--- + +## 6'. Already as good as (or better than) Datomic — don't redo + +These were checked and are solid; listed so they're not re-litigated: + +- **Sortable byte codec** — big-endian e/a/t + sign-flipped doubles + content-hash + for var-length values (`encoding.ss`). Equivalent to Datomic's key encoding. +- **Four covering indexes** EAVT/AEVT/AVET/VAET with correct orders + (`index/protocol.ss`); reverse-refs via VAET (`pull.ss`). +- **Predicate pushdown** into range scans, **live-variable projection** between + clauses, **index selection** by bound components, **`not-join` scoping** + (`engine.ss`, `planner.ss`). +- **Streaming aggregation** with mutable accumulators — O(1) alloc per group + (`engine.ss`; the explicit Q8 path). +- **Unique upsert** via AVET + tx-local value cache, **`:db/cas`**, + **cardinality-one auto-retract** with tx-local cache (`tx.ss`). +- **Lazy per-attribute entity loading** with caching (`entity.ss`); **ident + interning** (`schema.ss`). + +Minor polish noted by the review (low priority): tighten VAET range bounds in +reverse-ref pull; add explicit `avet-eligible?`/`vaet-eligible?` flags to the +attribute struct instead of recomputing; consider a tempid namespace/prefix for +safe batch replay; collect all `:db/cas` failures into one error rather than +failing on the first. + +--- + +## Suggested sequencing + +1. **Foundations that unlock the rest** — item 1 (B+-tree, with subtree counts) + and item 2 (lazy cursors). Contained to `index/memory.ss` + `index/protocol.ss`; + together they enable streaming joins and item 4. +2. **Planner quality** — item 4 (real cardinality) + item 8 (deferral) + item 7 + (hash-join). This is where Q4/Q8 improve without DuckDB. +3. **Durable redesign** — items 3 + 5 (columnar immutable segments, snapshots, + physical GC), then item 6 (byte-aware cache) and item 13 (external sort) ride + on the segment format. +4. **Semantics/correctness** — item 10 (component cascade), item 15 (ident + immutability), then items 11–12 (partitions, reified tx + log-as-datoms). +5. **Plumbing** — item 14 (storage SPI) whenever a second backend is wanted. + +--- + +## Appendix — method & source map + +**Decompile.** Datomic is Clojure AOT bytecode. `brew install openjdk` (keg-only; +`export PATH=/opt/homebrew/opt/openjdk/bin:$PATH JAVA_HOME=/opt/homebrew/opt/openjdk`). +Use **CFR**, not jadx — jadx folds Clojure's `$`-munged function classes as Java +inner classes and silently drops them. CFR matches **dotted FQNs**: + +``` +java -jar cfr.jar peer-1.0.7622.jar \ + --jarfilter 'datomic\.(btset|index|db|iter|datalog|query|pull|log|transaction|kv_.*|valcache|fressian|codec|external_sort|builtins|memory.*)\..*' \ + --outputdir cfr-core --silent +``` + +Decompiled core lives at `~/mine/datomix/work/cfr-core/datomic/` (2008 files). A +shorter findings note is at `~/mine/datomix/DATOMIC-FINDINGS.md`. Clojure naming: +`a.b` → `a/b$fn.java` (fn names survive; `_QMARK_`=`?`, `_BANG_`=`!`, +`__GT_`=`->`, `_STAR_`=`*`), types/protocols → `a/b/Type.java`, ns init → +`a/b__init.class` (grep these for the namespace inventory). + +**Reading map** (`cfr-core/datomic/`): +- B+-tree: `btset/BTSetLeaf.java`, `btset/BTSetBranch.java`, `btset/BTSet.java` +- Lazy merge: `iter$merge_iters.java`, `iter$least_index.java`, `iter/MergeIter.java` +- Read path: `db$seek_datoms.java`, `db$seek_datoms$fn__12075.java` (`windowed`) +- Durable tree: `index/RootNode.java`, `index/DirNode.java`, + `index/TransposedData.java`, `index/ITransposeData.java` +- Indexing job: `index$merge_one_index.java`, `index$build_one_seg.java`, + `index$merge_db_STAR_.java`, `external_sort$file_system_sorter.java` +- Db / schema: `db/IndexSet.java`, `db/Db.java`, `db/Attribute.java`, + `builtins$component_es_set.java`, `db$prevent_ident_retarget_BANG_.java` +- Tx / ids: `db$with_tx.java`, `db$resolve_lookup_ref.java`, `db$calc_tempids.java`, + `builtins$compare_and_swap.java`, `codec$*.java` +- Storage / cache: `kv_store__init.java`, `kv_cluster$mark_pod_garbage.java`, + `valcache__init.java`, `memory_size__init.java`, `fressian__init.java` + +**Confidence.** Items 1–6 are from direct reading of the decompiled source and +verified against jerboa-db. Items 7–15 combine decompiled evidence with a +subsystem review; where a jerboa-db behavior is asserted as missing it was grep- +verified (10, 11, and the cache/log specifics). Datomic's exact `:ensure` +isolation semantics and the valcache wire protocol were not fully recoverable from +bytecode — treat those details as approximate.