data: add final-wave lessons (version-keyed cache, hash-chain incremental seal, semantics-changing-optimization anti-pattern)
ober
50ca3878b8a389170012d7a3f828d9726634df2b
--- a/data/anti-patterns.sexp +++ b/data/anti-patterns.sexp @@ -5073,4 +5073,19 @@ ("title" . "Caching/indexing a crypto or on-disk-format path without invalidation + aliasing tests") - ("tools" "jerboa_security_scan"))) + ("tools" "jerboa_security_scan")) + (("advice" + . + "Gate the optimization on a conservative safety check: apply the fast path ONLY when you can prove it preserves semantics, and fall back to the safe path otherwise. Over-detection (taking the slow path when the fast path would have been safe) is fine — it only forgoes a perf win. Under-detection (taking the fast path when it changes semantics) is a correctness bug. Add a diagnostic counter to confirm which path is taken in tests. If no safe gating exists, document the deferral rather than ship a semantics-changing optimization.") + ("avoid" + . + "Applying a performance optimization that changes OBSERVABLE SEMANTICS without gating it on a safety check. Example: jerboa-pcre2 passing start/end byte offsets to avoid a substring copy — but PCRE2 does not treat a non-zero startoffset as a subject/line start, so start-anchored patterns (^, \\A, \\b) silently stop matching at a non-zero start. The naive optimization broke (^def \"abc def\" 4). Other examples: skipping pre-offset blocks without authenticated block sizes (returns corrupted plaintext on a wrong size guess); caching visibility without invalidation (serves a just-hidden item).") + ("id" . "optimization-that-changes-observable-semantics") + ("kinds" "performance" "correctness") ("pattern" . "") + ("severity" . "high") + ("tags" "optimization" "semantics" "correctness" "anchor" + "gating" "perf-bug") + ("title" + . + "Applying a perf optimization that changes observable semantics without a safety gate") + ("tools" "jerboa_howto"))) --- a/data/cookbooks.sexp +++ b/data/cookbooks.sexp @@ -7066,4 +7066,28 @@ "std net request") ("title" . - "Portable getaddrinfo IPv4+IPv6 resolution and DNS-rebinding-safe SSRF pinning"))) + "Portable getaddrinfo IPv4+IPv6 resolution and DNS-rebinding-safe SSRF pinning")) + (("code" + . + "(import (jerboa prelude))\n\n;; Version-keyed cache for data derived from a mutable source (e.g. a catalog\n;; whose visibility can change). A version counter is bumped at the SINGLE\n;; mutation point; caches are keyed by version and recomputed on a miss.\n;; Invalidation is synchronous (the version bump happens in the same lock as the\n;; mutation), so a stale entry is never served after a change.\n\n(def *catalog-version* 0)\n(def *visible-cache* #f) ; (version . value) or #f\n\n;; the SINGLE mutation point: bump the version under the state lock\n(def (swap-state! new-state)\n (with-lock state-lock\n (set! state new-state)\n (set! *catalog-version* (+ *catalog-version* 1))))\n\n(def (visible-set)\n (if (and *visible-cache* (= (car *visible-cache*) *catalog-version*))\n (cdr *visible-cache*) ; cache hit, O(1)\n (let ([v (compute-visible-set state)]) ; miss: compute once\n (set! *visible-cache* (cons *catalog-version* v))\n v)))\n\n;; Every code path that changes visibility (rebuild, hide/show, moderation)\n;; MUST go through swap-state! so the version bump invalidates all caches.") ("id" . "version-keyed-cache-invalidation") + ("imports" "(jerboa prelude)") + ("notes" + . + "Use when cached data is derived from a mutable source and staleness is a correctness/privacy bug (jerboa-imagesite: a stale 'visible' cache could serve a just-hidden crop). The version counter is bumped at the single mutation point under the same lock as the mutation, so invalidation is synchronous and a stale entry is never served. Every mutation path MUST route through the version-bumping setter. Cheaper than per-request recomputation (jerboa-imagesite face-crop went from ~60 full rebuilds per page to 1 per catalog version). For per-REQUEST memoization (data that doesn't outlive a request), a request-scoped parameter/memo is simpler; use the version-keyed cache when the data is shared across requests.") + ("tags" "cache" "invalidation" "version" "memoization" + "performance" "visibility") + ("title" + . + "Version-keyed cache with synchronous invalidation for dynamic data")) + (("code" + . + "(import (jerboa prelude))\n\n;; Append-only sealed delta frames chained by hash, so an append-only log can\n;; be sealed incrementally (O(delta) per append) instead of re-sealing the whole\n;; DB (O(db-size)) on every persist. Each frame is AEAD-sealed with a\n;; domain-tagged nonce (so no nonce reuse across frame kinds) and an AAD binding\n;; the previous chain hash, so tampering/removal/reordering/rollback of any\n;; append fails closed on open.\n;; chain-hash_0 = initial\n;; chain-hash_i = SHA256(chain-hash_{i-1} || sealed-frame_i)\n;; The expected FINAL chain hash is authenticated inside a small resealed head,\n;; so the whole chain is anchored to the head's AEAD seal.\n\n(def (seal-delta-frame prev-chain-hash generation salt nonce key plaintext)\n (let* ([aad (bytevector-append salt\n (u64->le-bytes generation)\n prev-chain-hash)]\n [sealed (aead-seal key nonce plaintext aad)])\n (values sealed\n (sha256 (bytevector-append prev-chain-hash sealed)))))\n\n;; open: replay the chain, recomputing chain hashes, and check the final hash\n;; equals the head-authenticated expected hash -> any tamper fails closed.\n(def (verify-delta-chain head-expected-hash frames salt generation key)\n (let loop ([frames frames] [chain initial-chain-hash])\n (cond [(null? frames) (equal? chain head-expected-hash)]\n [else (let* ([sealed (car frames)]\n [aad (bytevector-append salt (u64->le-bytes generation) chain)]\n [plain (aead-open key (frame-nonce sealed) sealed aad)]) ; raises on tamper\n (and plain\n (loop (cdr frames)\n (sha256 (bytevector-append chain sealed))))])))") ("id" . "append-only-hash-chain-incremental-seal") + ("imports" "(jerboa prelude)") + ("notes" + . + "Use when an append-only log must be sealed incrementally (O(delta) per append) but the underlying store image is NOT append-only (a SQLite image rewrites arbitrary pages on one insert, so it can't be sealed as a byte delta). Solution (jerboa-signal logdb): persist a small head container resealed per put in O(1) + an append-only file of sealed delta frames + a periodic whole-image checkpoint. Each delta frame's AAD binds the previous chain hash, so the chain is hash-chained; the expected final chain hash is authenticated inside the head's AEAD seal, anchoring the whole chain. Use a domain-tagged nonce per frame kind (head/checkpoint/delta) so there is no nonce reuse under the static key. Tampering/removal/reordering/rollback of any append fails closed on open. Preserve any external anti-rollback marker and wrong-passphrase fail-closed behavior. jerboa-signal immediate-mode persist went from O(db-size) full reseal per message to O(delta).") + ("tags" "hash-chain" "incremental" "aead" "append-only" + "integrity" "seal") + ("title" + . + "Append-only hash-chained incremental sealing (O(delta) integrity)")))