test: regression test for analytics-sync retraction filtering
ober
73955a7200517b98686ca04fa8511f2d98733ae2
--- a/plan.md +++ b/plan.md @@ -90,11 +90,14 @@ sorted (e,a,v,tx) order and keeping only the highest-tx datom per routes eligible queries to DuckDB without changing the call site. Same `q` API, same result shape, same dedup semantics — just orders of magnitude faster on wide aggregations. -- **Open issue:** at 10% scale, Q4 row counts diverge between Datalog and - DuckDB (4.27M vs 8.27M). 1% and 5% are clean. Suspected duplicate-collapse - difference in DISTINCT semantics under specific result shapes; not yet - root-caused. Not blocking — auto-routing is opt-in per query, and 1%/5% - are the sizes the bench advertises. +- **Hidden retraction bug, surfaced at scale.** The 10%-scale Q4 row count + initially diverged between Datalog (4.27M) and DuckDB (8.27M). 1% and + 5% looked clean because the bench's `make-uuid!` generator only cycles + through 8192 UUIDs (lower-16-bit period of its LCG); at 10% the upsert + collisions finally became dense enough that `analytics-sync!` was + copying retracted-but-still-`added=true` rows into DuckDB. Fixed by + filtering sync to currently-live datoms (commit `fbc3b6c`). Q4 now + bit-identical at 1%/5%/10%. --- --- a/tests/test-sql-translate.ss +++ b/tests/test-sql-translate.ss @@ -59,3 +59,28 @@ (write (analytical-query p ae (db-value-schema (db conn)))) (newline) (analytics-close ae) + +;; Regression: analytics-sync must drop retracted/superseded values from +;; DuckDB. EAVT keeps every datom (assertions and retractions); a naive +;; sync that simply copies all rows leaves stale values participating in +;; joins. Earlier-asserted rows still have added=true even after the +;; value is later retracted, so the SQL filter `WHERE added` is not +;; sufficient on its own — sync has to filter to live datoms. +(define conn2 (connect ":memory:")) +(transact! conn2 (list + `((db/ident . user/name) (db/valueType . db.type/string) (db/cardinality . db.cardinality/one)) + `((db/ident . user/email) (db/valueType . db.type/string) (db/cardinality . db.cardinality/one) (db/unique . db.unique/identity)))) +;; Two transacts where the second upserts via unique email and changes the name. +(transact! conn2 '(((db/id . -1) (user/email . "a@x") (user/name . "Old")))) +(transact! conn2 '(((user/email . "a@x") (user/name . "New")))) +(define ae2 (new-analytics-engine (db-value-schema (db conn2)))) +(analytics-sync! ae2 (db conn2)) +(define live-names + (analytics-query ae2 + "SELECT v_string FROM datoms WHERE a_name = 'user/name' AND added")) +(display "Live names after upsert: ") (write live-names) (newline) +(unless (and (= 1 (length live-names)) + (string=? (cdar (car live-names)) "New")) + (error 'sync-test "expected only 'New' to be live, got" live-names)) +(display "PASS: analytics-sync filters retracted values\n") +(analytics-close ae2)