docs: comprehensive jerboa-db.md update — Phase 6.3 complete + benchmark gap analysis
ober
5c1a2f9b2f33253a078d8e916d1db3b639f86ff8
--- a/jerboa-db.md +++ b/jerboa-db.md @@ -4,10 +4,12 @@ persistent index storage and DuckDB for analytics (planned). Single binary, embeddable, with a Datalog query engine and immutable time-travel over all data. -**Status:** 2026-04-13 — **Phases 1–3 fully implemented and tested (34/34 tests -pass). Phase 2 (LevelDB persistence) is wired and production-ready. Phases 4–6 -are functional stubs awaiting future work. MBrainz benchmark harness complete -(all 8 queries verified at 1% scale).** +**Status:** 2026-04-13 — **Phases 1–3, 5, 6 (in-process + TCP transport), 7–8 +fully implemented and tested (52/52 tests pass; 34 core + 6 cluster + 12 transport). +Phase 2 (LevelDB persistence) is production-ready. Phase 4 (DuckDB) is +structurally complete but build-dependent. MBrainz benchmark harness complete +(all 8 queries verified at 1% scale). Three critical `(std raft)` bugs found +and fixed during Phase 6.3 development.** Implementation lives in `lib/jerboa-db/` (Jerboa library files using `#!chezscheme` + `(library ...)` form). Built from scratch — no `(std mvcc)`, `(std datalog)`, or @@ -29,8 +31,8 @@ synthetic data at configurable scale. `make mbrainz-quick` for smoke test, | 2 | LevelDB persistence (`connect("path")`, 4-index LevelDB backend, FASL encoding) | ✅ Complete | | 3 | Query engine (planner, predicates, aggregates, rules, streaming) | ✅ Complete | | 4 | DuckDB analytics (SQL over datoms, Parquet export/import) | ✅ Core done, Parquet 🚧 | -| 5 | Server mode (HTTP API, WebSocket tx-stream, remote peer) | 🚧 Stub | -| 6 | Raft HA (distributed consensus, read replicas) | 📋 Planned | +| 5 | Server mode (HTTP API, WebSocket tx-stream, remote peer) | ✅ Functional | +| 6 | Raft HA (in-process + TCP transport, read replicas) | ✅ Complete (TLS 🚧) | | 7 | Polish (backup/restore ✅, GDPR excision ✅, schema migration 🚧) | ✅ Mostly done | | 8 | Advanced (fulltext ✅, GC ✅, entity specs ✅, composite tuples ✅) | ✅ Complete | @@ -103,10 +105,11 @@ they are unimplemented (❌), implemented (✅), or planned stubs (🚧). |---|---|---| | Single-node embedded | ✅ Done | Core use case | | LevelDB persistent storage | ✅ Done | `connect("path/to/db")` | -| HTTP server | 🚧 Stub | `server.ss` skeleton; not runnable yet | -| Remote peer client | ❌ Missing | Post-Phase-5 | -| Raft consensus / HA | 📋 Planned | `replication.ss` is a structural stub | -| Read replicas | ❌ Missing | Requires Phase 5+6 | +| HTTP server | ✅ Done | `server.ss` — all REST + WebSocket endpoints; CLI entrypoint 🚧 | +| Remote peer client | ✅ Done | `peer.ss` — transact, query, pull, multi-URL failover | +| Raft consensus / HA (in-process) | ✅ Done | `replication.ss` + `cluster.ss`; 6/6 tests | +| Raft consensus / HA (TCP transport) | ✅ Done | `transport.ss`; 12/12 tests; plain TCP (TLS 🚧) | +| Read replicas | ✅ Done | Follower apply fiber, ~50ms lag | ### Analytics @@ -805,10 +808,10 @@ the connection registry handle multiple named databases. **Files:** `lib/jerboa-db/replication.ss`, `lib/jerboa-db/cluster.ss`, `lib/jerboa-db/peer.ss` -The in-process Raft layer and the HTTP peer client are complete. Multi-node -replication works inside a single OS process using `(std raft)` channels. -What remains is a TCP/TLS transport adapter to span processes or machines -(Phase 6.3) and local db-value caching in the peer client (Phase 6.4). +The in-process Raft layer, TCP transport, and HTTP peer client are all complete. +Multi-node replication works both inside a single OS process (in-process channels) +and across separate OS processes over TCP (Phase 6.3). Local db-value caching +in the peer client (Datomic-style segment cache) is the remaining item (Phase 6.4). #### What's implemented @@ -830,10 +833,15 @@ What remains is a TCP/TLS transport adapter to span processes or machines #### Architecture notes -All nodes share the same apply path: every node's fiber reads the **cluster -leader's** Raft log rather than the node's own local log. This works around -a limitation in `(std raft)` where the AppendEntries handler can truncate -earlier entries from a follower's local log view. +**In-process clusters** (`start-local-db-cluster`): all nodes read the +**cluster leader's** Raft log in their apply fiber. This simplifies the +in-process case and was originally a workaround for a log-truncation bug. + +**TCP transport nodes** (`start-transport-db-node!`): each node's apply fiber +reads its **own** committed log via `replication-for-each-committed!`. This +is correct for multi-process clusters where there is no shared memory. This +path depends on the `(std raft)` AppendEntries log-truncation fix (≤ instead of +<) being in place — without it, followers drop earlier log entries. For **1-node clusters**, `(std raft)` never fires `try-advance-commit-index!` (it's triggered by AppendEntriesResponse, which requires peers). The apply @@ -849,9 +857,11 @@ fiber detects this case and treats the last log entry as effectively committed from the leader's log. Serves `cluster-db` reads (eventually consistent, typically < 100 ms behind the leader). -- **Transport (Phase 6.3, not yet started):** `(std raft)` uses in-process - channels. A future transport adapter will bridge each node's inbox channel - to a TCP/TLS socket, enabling multi-process and multi-host clusters. +- **Transport (Phase 6.3, complete):** `transport.ss` bridges each node's + `(std raft)` channels to TCP sockets using length-framed FASL messages. + Outbound proxy fibers serialise and write; inbound accept-loop fibers + deserialise and deliver. Reconnect backoff (100ms → 5s) handles transient + failures. API: `start-transport-db-node!`, `transport-node-add-peer!`. #### Peer caching (the real Datomic secret) @@ -889,7 +899,7 @@ trade-offs: | DataScript | Clojure/JS | In-memory | Yes | No | No | Yes (JS) | | Datahike | JVM | Pluggable | Yes | Partial | No | No | | Datalevin | JVM | LMDB | Yes | No | No | No | -| **Jerboa-DB** | **Native** | **LevelDB (persistent) + in-memory RB-trees; DuckDB planned** | **Yes** | **Yes** | **Planned (stub)** | **Yes** | +| **Jerboa-DB** | **Native** | **LevelDB (persistent) + in-memory RB-trees; DuckDB planned** | **Yes** | **Yes** | **Yes (Raft, TCP)** | **Yes** | Jerboa-DB fills a gap that doesn't exist today: a **native, embeddable, distributed, time-traveling Datalog database** that ships as a single binary and @@ -912,7 +922,7 @@ make this project tractable in a way it wouldn't be in Go, Rust, or Python: | Serialization | `(std fasl)` — Chez native binary format | **Used** — FASL encoding of datom values in LevelDB | | Server | `(std net fiber-httpd)` — fiber-native HTTP | **Used in stub** — `server.ss` imports it; not functional yet | | Analytical queries | `(std db duckdb)` — columnar OLAP engine | **Used in stub** — `analytics.ss` imports it; DuckDB planned | -| Distributed consensus | `(std raft)` — leader election + log replication | **Available** — Phase 6 (not started) | +| Distributed consensus | `(std raft)` — leader election + log replication | **Used** — `replication.ss` + `cluster.ss` + `transport.ss`; 3 bugs fixed in `(std raft)` itself | | Actor supervision | `(std actor)` — OTP-style fault tolerance | **Available** — Phase 5+ | | EDN wire format | `(std text edn)` — EDN parser | **Used in server stub** — for HTTP request/response | @@ -1479,10 +1489,11 @@ No `(std mvcc)`, `(std datalog)`, or other stdlib modules are used in the core p ## Implementation Plan -> **Current state (2026-04-13):** Phases 1–3 and 7–8 are fully implemented and -> tested (34/34 tests pass). Phase 2 (LevelDB) is wired and production-ready. -> Phases 4–6 are stubs. The sections below describe the original design intent; -> they serve as reference documentation. +> **Current state (2026-04-13):** Phases 1–3, 5, 6.1–6.3, 7–8 are fully +> implemented and tested (52/52 tests pass: 34 core + 6 in-process cluster + +> 12 TCP transport). Phase 2 (LevelDB) is wired and production-ready. Phase 4 +> (DuckDB analytics) is structurally complete but FFI-build-dependent. The +> sections below describe design intent and serve as reference documentation. ### Phase 1: Core — In-Memory Datomic ✅ IMPLEMENTED @@ -2128,9 +2139,11 @@ Note: operations are server-side round-trips. Local db-value caching **Goal:** Multi-node Jerboa-DB with automatic failover. One transactor (leader), multiple read replicas (followers). -**Current state:** The in-process layer is **complete and tested**. All six -integration tests pass. The outstanding item is Phase 6.3 — a TCP/TLS -transport so nodes can run in separate OS processes. +**Current state:** Complete. Both the in-process layer (6/6 tests) and the TCP +transport layer (12/12 tests) are implemented and passing. Each `start-transport-db-node!` +call returns a `(values transport-node replicated-conn)` pair compatible with the +full cluster API. The remaining items are TLS wrapping for production use and +local db-value caching in the peer client (Phase 6.4). #### 6.1 Raft-Based Transactor ✅ Complete (in-process) @@ -2182,11 +2195,11 @@ transport so nodes can run in separate OS processes. (cluster-db/consistent leader) ;; read-latest — leader only, always current ``` -#### 6.3 TCP/TLS Transport 🚧 Not started +#### 6.3 TCP/TLS Transport ✅ Complete -`(std raft)` communicates over in-process Scheme channels. A transport adapter -must bridge each node's inbox channel to a TCP/TLS socket so nodes can run in -separate OS processes or on separate machines. +`transport.ss` bridges each `(std raft)` node's inbox/peer channels to TCP +sockets so nodes can run in separate OS processes or on separate machines. +12/12 integration tests pass (`tests/test-transport.ss`, `make test-transport`). ##### Architecture @@ -2232,86 +2245,54 @@ Messages (Raft vector messages, as-is from (std raft)): `client-propose` is always local (client → leader via local channel). Only the four Raft RPC message types cross the wire. -##### Implementation plan +##### Actual Implementation ```scheme -;; lib/jerboa-db/transport.ss (new file) +;; lib/jerboa-db/transport.ss (import (jerboa prelude) (rename (only (chezscheme) make-time) (make-time chez-make-time)) - (std net tcp) - (std net tls) ;; optional — TLS wrapping + (std net tcp) ;; tcp-listen, tcp-connect-binary, tcp-accept-binary + (std fasl) ;; fasl->bytevector, bytevector->fasl (std misc channel) - (jerboa-db replication)) - -;; --- Outbound proxy fiber --- -;; Reads from a local channel and writes length-framed FASL to a TCP connection. -(def (start-outbound-proxy! proxy-ch tcp-conn) - (fork-thread - (lambda () - (let loop () - (let ([msg (channel-get proxy-ch)]) - (unless (eof-object? msg) - (let* ([bytes (fasl-write-to-bytevector msg)] - [len (bytevector-length bytes)] - [frame (make-bytevector (+ 4 len))]) - (bytevector-u32-set! frame 0 len 'big) - (bytevector-copy! bytes 0 frame 4 len) - (guard (exn [#t (void)]) ;; peer disconnected - (tcp-write tcp-conn frame)) - (loop)))))))) - -;; --- Inbound listener fiber --- -;; Reads length-framed FASL from a TCP connection and puts to node inbox. -(def (start-inbound-listener! tcp-conn local-inbox) - (fork-thread - (lambda () - (let loop () - (let ([len-buf (tcp-read tcp-conn 4)]) - (when (and len-buf (= (bytevector-length len-buf) 4)) - (let* ([len (bytevector-u32-ref len-buf 0 'big)] - [body (tcp-read tcp-conn len)]) - (when (and body (= (bytevector-length body) len)) - (let ([msg (fasl-read-from-bytevector body)]) - (channel-put local-inbox msg)) - (loop))))))))) - -;; --- Transport-aware cluster startup --- -(def (start-transport-cluster node-id peers data-path port . tls-opts) - ;; peers: list of (id host port) — other cluster members - ;; 1. Make raft-node, wire peer proxy channels - ;; 2. Start TCP listener on `port` for inbound connections - ;; 3. Dial each peer, start outbound proxy + inbound listener - ;; 4. Start Raft - ...) -``` - -##### Planned API - -```scheme -;; lib/jerboa-db/replication.ss (extension) - -;; TCP cluster config — extends new-replication-config -(def cfg (new-replication-config - 'node-0 - '((node-1 "host-b" 7001) ;; peer descriptors - (node-2 "host-c" 7002)) - "/var/lib/jerboa-db/node-0" - :port 7000 - :tls #t)) ;; enable TLS - -;; Starts Raft + TCP transport: -(def rconn (replicated-connect "/var/lib/jerboa-db/node-0" cfg)) + (std raft) ;; raft-node-*, raft-start!, raft-node-add-peer! + (jerboa-db replication) + (jerboa-db core) + (jerboa-db cluster)) + +;; Convenience: one call creates Raft node + TCP transport + DB connection +(def-values (tnode rconn) + (start-transport-db-node! 'node-0 '() "/var/db/node-0" 7000)) + +;; Staged setup (ports only known after start): +(def-values (ta ra) (start-transport-db-node! 'a '() ":memory:" 0)) +(def-values (tb rb) + (start-transport-db-node! 'b `((a "127.0.0.1" ,(transport-node-listen-port ta))) + ":memory:" 0)) +(transport-node-add-peer! ta 'b "127.0.0.1" (transport-node-listen-port tb)) ;; Same cluster API as in-process: -(cluster-transact! rconn schema-tx) -(cluster-db rconn) -(cluster-status rconn) +(cluster-transact! ra schema-tx) +(cluster-db ra) +(cluster-status ra) + +;; Clean shutdown: +(stop-transport-node! ta) +(stop-transport-node! tb) ``` -##### TLS +Key implementation details: +- **Wire protocol:** 4-byte big-endian uint32 length header + FASL body +- **Uni-directional connections:** node A dials B for A→B traffic; node B dials A for B→A traffic +- **Reconnect backoff:** 100ms → 5s (doubles on failure) to handle startup races +- **`client-propose` dropped:** carries a live reply channel; never serialised over wire +- **`raft-node-add-peer!`:** thread-safe; initialises `next-index`/`match-index` if node is already leader (prevents `send-heartbeats!` crash on new peer) + +##### TLS (future) -Optional TLS via `(std net tls)`: +The current TCP transport is plain text — suitable for localhost/loopback +clusters and trusted private networks. Production cross-datacenter deployments +should add mTLS via `(std net tls)`: ```scheme ;; Mutual TLS (mTLS) for inter-node auth: @@ -2321,15 +2302,11 @@ Optional TLS via `(std net tls)`: :key "/etc/jerboa-db/node.key" :ca "/etc/jerboa-db/ca.crt")) -;; Outbound: (tls-connect host port tls-cfg) instead of tcp-connect -;; Inbound: (tls-accept tcp-server tls-cfg) instead of tcp-accept +;; Outbound: (tls-connect host port tls-cfg) instead of tcp-connect-binary +;; Inbound: (tls-accept tcp-server tls-cfg) instead of tcp-accept-binary ``` -For development clusters on a trusted network, plain TCP is fine. For -production or cross-datacenter deployments, mTLS is recommended. - -**Estimated effort:** 2–4 weeks. The consensus algorithm is complete; the -transport is purely I/O plumbing. +For development and single-host clusters, plain TCP is fine. #### 6.4 Peer Client (`peer.ss`) ✅ Functional (HTTP-based) @@ -2413,13 +2390,27 @@ incremental updates without a full re-download. (set! db (apply-tx-report db tx-report)))) ;; incremental update ``` +#### `(std raft)` Bugs Fixed During Phase 6 + +Three bugs were found and fixed in `~/mine/jerboa/lib/std/raft.sls` while +building the TCP transport layer. These affect both in-process and TCP clusters. + +| Bug | Symptom | Root cause | Fix | +|---|---|---|---| +| AppendEntries log truncation | Follower always had only the latest single log entry; previous entries were lost | `(< entry-index prev-idx)` instead of `(<= entry-index prev-idx)` in the keep-existing-entries filter — dropped everything with index < prev-idx instead of ≤ prev-idx | Changed `<` to `<=` in AppendEntries handler | +| Synchronised election timers | Leader election failed ~50% of runs with 2+ nodes | `(random n)` in Chez Scheme is not thread-safe; concurrent threads get identical PRNG values → identical election timeouts → perpetual split-vote | `election-timeout-ms` now takes a `node` argument and adds a node-id-derived constant offset, making timeouts deterministically distinct per node | +| `send-heartbeats!` crash on new peers | Raft message loop thread died silently when a peer was added after `become-leader!` | `(cdr (assv peer-id next-index))` = `(cdr #f)` when the peer was absent from `next-index` | Added `raft-node-add-peer!` (thread-safe, initialises `next-index`/`match-index` if leader); added per-peer `guard` + safe `ni` fallback in `send-heartbeats!`; added `guard` around main message loop dispatch | + +These fixes are committed to `~/mine/jerboa` (commit c73d576). + **Phase 6 success criteria:** - ✅ Leader elected within 500ms in 3-node cluster (in-process) - ✅ `cluster-transact!` commits and returns tx-report (1-node and 3-node) - ✅ Follower connections converge to leader state within 200ms (in-process) - ✅ `cluster-status` returns Raft fields + db basis-tx - ✅ `peer.ss` HTTP client — transact, query, pull with failover (multi-URL) -- 🚧 TCP/TLS transport for multi-process/multi-host clusters (Phase 6.3) +- ✅ TCP transport for multi-process/multi-host clusters (Phase 6.3) — 12/12 tests pass +- 🚧 TLS wrapping for production inter-node traffic - 🚧 WebSocket tx-stream in peer client - 🚧 Local db-value caching in peer client (Datomic-style segment cache) @@ -2585,7 +2576,8 @@ project and a multi-year one. > All source lives in `lib/jerboa-db/`. Core engine built from scratch (no > `(std mvcc)`, `(std datalog)`, etc. used). LevelDB FFI via `(std db leveldb)`. -> All 34 integration tests pass as of 2026-04-13. +> 52/52 integration tests pass as of 2026-04-13 (34 core + 6 in-process cluster +> + 12 TCP transport). ### Core (Phase 1) — ✅ COMPLETE @@ -2669,9 +2661,10 @@ Files: `lib/jerboa-db/server.ss` | `register-cluster!` hook | ✅ Done | Wires cluster status thunk to HTTP endpoint | | Remote peer client | ✅ Done | `peer.ss` — see Phase 6.4 | -### Distribution (Phase 6) — ✅ IN-PROCESS COMPLETE +### Distribution (Phase 6) — ✅ COMPLETE (in-process + TCP) -Files: `lib/jerboa-db/replication.ss`, `lib/jerboa-db/cluster.ss`, `tests/test-cluster.ss` +Files: `lib/jerboa-db/replication.ss`, `lib/jerboa-db/cluster.ss`, +`lib/jerboa-db/transport.ss`, `tests/test-cluster.ss`, `tests/test-transport.ss` | Feature | Status | Notes | |---|---|---| @@ -2682,8 +2675,10 @@ Files: `lib/jerboa-db/replication.ss`, `lib/jerboa-db/cluster.ss`, `tests/test-c | Follower convergence | ✅ Done | Apply fiber replicates committed entries within 50 ms | | Read consistency levels | ✅ Done | `read-committed`, `read-latest`, `as-of-tx` | | Cluster status / introspection | ✅ Done | `cluster-status / cluster-leader?` | -| Integration test suite | ✅ Done | 6/6 tests passing (`tests/test-cluster.ss`) | -| TCP/TLS transport | 🚧 TODO | Phase 6.3 — needed for multi-process/multi-host; `(std net tcp)` + `(std net tls)` available | +| In-process cluster test suite | ✅ Done | 6/6 tests passing (`tests/test-cluster.ss`) | +| TCP transport test suite | ✅ Done | 12/12 tests passing (`tests/test-transport.ss`) — startup, election, transact, replicate, status, stop | +| TCP transport (plain) | ✅ Done | `transport.ss` — 12/12 tests pass; length-framed FASL, reconnect backoff 100ms→5s, `raft-node-add-peer!` thread-safe | +| TLS wrapping | 🚧 TODO | Replace `tcp-connect-binary`/`tcp-accept-binary` with `(std net tls)` equivalents | | Remote peer client (HTTP) | ✅ Done | `peer.ss` — connect-remote, remote-transact!, remote-q, remote-pull, multi-URL failover | | Remote peer — db-value cache | 🚧 TODO | Every op is a network round-trip; Datomic-style local snapshot not yet implemented | | Remote peer — WebSocket stream | 🚧 TODO | `remote-tx-stream` stub; needs fiber context wiring | @@ -2833,6 +2828,143 @@ generation, not the aggregation step itself. | Bulk-index load path | Load time | Defer RB-tree sort until end of load phase; currently ~120s at 1% scale | | Real MBrainz EDN loader | Full scale | Parse actual Datahike EDN files instead of synthetic data | +### Comparison vs Datahike (published numbers, full scale) + +Datahike publishes MBrainz results at **full scale** (262K artists, 1.31M +releases, 13.1M tracks). Our numbers are at **1% scale** (2,620 artists, +13,100 releases, 131,000 tracks) — a 100× smaller dataset. + +| Query | Datahike full scale | Jerboa-DB 1% scale | Scaled projection (×100) | Gap | +|---|---|---|---|---| +| Q1: exact name lookup | ~1ms | 0ms | <1ms | **None** — AVET lookup | +| Q2: releases by artist | ~1ms | 0ms | <1ms | **None** — two-hop AVET/AEVT | +| Q3: range predicate (startYear) | ~5ms | 2ms | ~200ms | **Moderate** — AVET range, degrades with scale | +| Q4: tracks × shared-artist releases | ~2–8s | 2468ms | **~250s** | **Critical** — output is 428K rows at 1%, ~43M at full | +| Q5: count releases per country | ~10ms | 4ms | ~400ms | **Moderate** — full AEVT scan | +| Q6: reverse ref lookup | ~1ms | 0ms | <1ms | **None** | +| Q7: pull | ~1ms | 0ms | <1ms | **None** | +| Q8: avg duration by status | ~500ms–2s | 2056ms | **~205s** | **Critical** — 655K intermediate bindings at 1% | + +> Datahike reference: https://github.com/replikativ/datahike/tree/main/bench +> Note: Datahike uses Hitchhiker tree (persistent B+ tree) + hash-join; numbers from +> their CI benchmark reports circa 2023. XTDB v2 (Apache Arrow + Parquet columnar +> execution) performs similarly to Datahike on Q1–Q3 and 2–5× faster on Q4/Q8. + +**The fast queries (Q1, Q2, Q6, Q7) are already competitive.** The critical gaps +are Q4 and Q8 — multi-hop joins that produce large intermediate result sets. + +### Write Throughput Degradation + +The load test measures write throughput at small scale (~5K entities) where the +in-memory RB-tree is shallow. MBrainz data loading reveals the degradation curve: + +| Phase | Entities | Observed throughput | Expected (flat) | +|---|---|---|---| +| Load test quick (batch=1) | ~5K total | ~154K ops/sec | baseline | +| Load test quick (batch=100) | ~5K total | ~184K ops/sec | baseline | +| MBrainz 1% data load | ~147K total | ~1,300 entities/sec | ~154K | + +**The effective throughput drops ~120× from 5K to 147K entities.** + +Root cause: the in-memory index is a persistent RB-tree (functional, immutable). +Each `transact!` creates a new tree root that shares structure with the previous +version. The tree depth grows as O(log N), meaning each insert performs O(log N) +allocations. At 147K entities × ~10 datoms × 4 indices = ~5.9M datom insertions, +the tree is ~23 levels deep. Combined with GC pressure from structural sharing, +throughput degrades geometrically. + +Datahike uses the **Hitchhiker tree** — a write-optimised persistent B+ tree that +batches writes in tree-local buffers, amortising structural allocation. Datomic +uses a similar segment-based approach with a dedicated background indexer process. + +### Performance Gap Analysis and Closing the Gaps + +#### Gap 1: Q4/Q8 Join Explosion (Critical) + +**Root cause:** The query engine executes clauses left-to-right (after a greedy +selectivity reorder). For Q4, the first anchoring clause binds all 131K tracks +before any filtering. There is no hash-join: every `(track, artist, release)` +triple is materialised as a binding-environment copy. + +**What Datahike does:** Hitchhiker tree stores cardinality statistics per attribute. +The planner uses these to choose the smallest-cardinality anchor. Binary joins are +executed as hash-joins (O(M+N) instead of O(M×N)). + +**Implementation plan to close the gap:** + +1. **Hash-join for shared-variable binary joins** (highest impact, ~2–10× speedup + on Q4/Q8, required for full-scale viability): + ``` + ;; When two clauses share a variable ?x, instead of nested-loop: + ;; (for each binding-A (for each binding-B (if (= ?x-A ?x-B) emit))) + ;; use hash-join: + ;; (build hash-table: ?x → [bindings-A]) + ;; (for each binding-B: lookup ?x-B in hash-table, emit cross-product) + ``` + Required changes: `query/planner.ss` (detect shared variables), `query/engine.ss` + (new `execute-hash-join` path), `query/engine.ss` (replace inner flatmap loop). + +2. **Cardinality statistics** (planner quality; enables better anchor selection): + Maintain per-attribute datom count in a dedicated `:db/stats` attribute updated + at `transact!` time. The planner scores clauses by estimated output size instead + of structural selectivity. Required changes: `schema.ss` (stats counters), + `tx.ss` (increment on add, decrement on retract), `query/planner.ss` (cost model). + +3. **Semi-join pushdown** (reduces output size for Q4 without needing full result): + If the query only needs `(count ?x)` or a projection, a semi-join (keep-any-match) + avoids materialising the full Cartesian product. This is most impactful for Q4's + 428K-row output at 1% scale. + +4. **Selectivity-based clause reordering with bound-variable propagation:** + Currently the planner reorders once at parse time. The planner should re-evaluate + clause cost after each clause executes, using actual intermediate result size. + This is a partial re-evaluation strategy (like Volcano/IteratorModel lazy pull). + +#### Gap 2: Write Throughput Degradation (High Impact at Scale) + +**Root cause:** Persistent functional RB-tree deepens O(log N) per insert; no +batching or background indexing. + +**Implementation options (easiest first):** + +1. **Batch-index path during bulk load** (quick win, ~5–10× improvement): + Collect all datoms in a transaction as a vector, sort once, and bulk-insert into + the RB-tree using a merge-sort approach rather than one-by-one splay insertions. + Required changes: `index/memory.ss` (new `index-bulk-add!`), `tx.ss` (use bulk + path when batch size > threshold). + +2. **Write-ahead log + lazy index merge** (structural fix, complex): + Keep a flat transaction log as the source of truth. Build the in-memory index + lazily (or in a background thread) by merging sorted log segments. This is + similar to LevelDB's own LSM approach and would give near-constant write throughput + at any scale. Required: new `index/wal.ss`, significant refactor of `core.ss`. + +3. **Persistent balanced BST with batch operations** (medium effort): + Replace the purely functional RB-tree with a persistent B-tree that supports + bulk-load (sorted input → balanced tree in O(N) rather than O(N log N)). The + `(std data sorted-map)` module may support this; verify with `jerboa_module_exports`. + +**Target after fix 1 alone:** ~20K entities/sec at 147K scale (15× improvement), +which would reduce MBrainz 1% load time from 112s to ~7s. + +#### Gap 3: Full-Scale MBrainz Feasibility + +At full scale (262K artists, 13.1M tracks) with the current engine: +- Write throughput: ~1,300 ents/sec → estimated **3+ hours** to load +- Q4: extrapolates to **~250 seconds** per query run +- Q8: extrapolates to **~200 seconds** per query run + +**Minimum required for full-scale viability:** +- Hash-join (Gap 1, item 1) — reduces Q4 from O(N²) to O(N), enabling <10s +- Batch-index bulk load (Gap 2, item 1) — reduces load time from hours to ~30min +- Cardinality stats (Gap 1, item 2) — prevents worst-case anchor selection + +**DuckDB path (alternative for analytics):** For Q4/Q8-class aggregation queries, +routing them through the DuckDB analytics layer (Phase 4) would bypass the Datalog +engine entirely. At that point the bottleneck becomes DuckDB sync latency, not +join ordering. This is a valid production strategy ("Datalog for navigation, SQL +for analytics"). + ### Data Loading Strategy (Full Scale) MBrainz data is distributed as EDN transaction files (Datomic format).