λgitsite

~ober/jerboa-db

Imported from ~/mine/jerboa-db

download snapshot

about

# jerboa-db

A Datomic-shaped Datalog database, written in Chez Scheme on top of
[Jerboa](../jerboa). EAVT/AEVT/AVET/VAET indices, time travel, ACID
transactions, pull syntax, Raft cluster — and a transparent DuckDB
columnar fallback for analytical queries.

About 10K lines of Scheme.

## Quickstart

```bash
make setup        # one-time toolchain prep on macOS
make test         # core/storage + plain/TLS transport security tests
make verify       # security, import, test, and native-audit gate
make release-evidence  # writes dist/release-evidence/
make mbrainz-quick  # 1% MBrainz benchmark, < 5s
make mbrainz      # full bench (slow on a laptop)
```

## Security status

`jerboa-db` is not public-production-ready yet. The deterministic local gate is
`make verify`; it covers security checks, import checks, storage/corruption
regressions, migrations, authenticated plain transport, generated-fixture TLS
replication/admission, and a sibling Jerboa native RustSec audit when available.
It first probes for the exact Raft wire and bounded TLS APIs this source uses.
The published Jerboa v0.2.3 toolchain does not contain them, so hosted CI is
intentionally fail-closed pending a reviewed containing Jerboa release.

The storage layer now validates segment-store paths, fixed segment filenames,
content-addressed segment reads, persisted roots, bounded/private backups,
transaction-log segment names, and transport frame sizes. The HTTP API binds
loopback only and requires a 32-byte-or-longer bearer token plus optional
per-database authorization; Raft defaults to loopback and rejects unauthenticated
public binding. See [`SECURITY.md`](SECURITY.md),
[`docs/threat-model.md`](docs/threat-model.md), and
[`docs/storage-hardening.md`](docs/storage-hardening.md).

The remaining release blockers are Linux peer tests with epoll-capable native
support, DuckDB analytics tests with a DuckDB-enabled native crate, stable
multi-node cluster/transport soak, LevelDB backend evidence, SBOM/reproducible
build notes, history secret scan, and external review.

## What it does

- **Datalog** — `(q '[:find ?e :where [?e :artist/name ?n]] db)`
- **Pull** — `(pull db [:artist/name :artist/country] eid)`
- **Time travel** — `(as-of db tx)` / `(since db tx)`
- **Transactions** — `(transact! conn ops)` with tempids and upserts
- **Cluster** — Raft replication over TCP/TLS (Phase 6)
- **OLAP** — `(parameterize ([current-analytics-engine ae]) (q …))` routes
  wide aggregations through DuckDB; same Datalog API, columnar speeds.

## How fast is it

MBrainz benchmark (Datahike's standard 8-query suite), 1% scale on a
laptop. Lower is better.

| Query | Datalog | DuckDB fallback | Datahike full |
|---|---:|---:|---:|
| Q1 — exact name lookup | 0 ms | — | ~1 ms |
| Q2 — 2-hop | 0 ms | — | ~1 ms |
| Q3 — range filter | 1 ms | — | ~5 ms |
| Q4 — wide join | 1416 ms | **281 ms** | 2–8 s |
| Q5 — group-by | 1 ms | — | ~10 ms |
| Q6 — reverse-ref | 0 ms | — | ~1 ms |
| Q7 — pull | 0 ms | — | ~1 ms |
| Q8 — avg-by-status | 958 ms | **9 ms** | 0.5–2 s |
| Bulk load 144K datoms | 1.0 s | | |

At 5% scale (655K tracks, ~3.3M datoms), the analytics gap widens — Q4
drops from 10s Datalog to 1.5s DuckDB; Q8 from 6.4s to 31ms. Run
`scheme --script benchmarks/mbrainz-bench.ss --scale 0.05` to verify.

## Where to read more

- [`status.md`](status.md) — one-page snapshot of where the project stands
- [`plan.md`](plan.md) — performance plan and shipped phases (§0 = results)
- [`jerboa-db.md`](jerboa-db.md) — long-form architecture & history
- [`CLAUDE.md`](CLAUDE.md) — working rules for AI agents on this repo

recent commits