Initial jerboa-search import
ober
f97ac375f9f2627a437ae3112d1aa5a736302450
new file mode 100644 --- /dev/null +++ b/.gitignore @@ -0,0 +1,10 @@ +data/*.db +data/*.db-* +data/*.jerboa-index* +data/*.content/ +dist/ +build/ +lib/ +**/.jerbuild-hashes +*.so +*.wpo new file mode 100644 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,504 @@ +## The Jerboa Language — Quick Reference + +Jerboa is a Scheme dialect built on Chez Scheme. It is Gerbil-inspired but its own language. **All user-facing code is `.ss` files. Never write `.sls` files for the user** — those are internal implementation files. + +### File Structure + +Every Jerboa file looks like this: + +```scheme +(import (jerboa prelude)) ;; ONE import gives you the ENTIRE language +;; Optional extra imports for modules NOT in the prelude: +;; (import (std net request)) + +(def (my-function x y) + (+ x y)) + +(displayln (my-function 1 2)) +``` + +Run with: `jerboa run file.ss` + +**NEVER** write `(library ...)` forms — that's `.sls` internal syntax. + +### Reader Syntax Extensions + +``` +[...] → plain parentheses — same as Gerbil and Chez Scheme +{method obj args} → (~ obj 'method args) — method dispatch +name: → keyword #:name +:std/sort → (std sort) — Gerbil-style module path +#<<END ... END → heredoc string +``` + +Square brackets `[...]` are interchangeable with `(...)`, exactly like Gerbil and stock Chez Scheme. You can freely use them in bindings, match clauses, and anywhere you'd use parentheses: +```scheme +;; All of these are correct: +(let ([x 1] [y 2]) (+ x y)) +(for/collect ([x (in-range 5)]) (* x x)) +(match val ([list a b] (+ a b))) +(cond [(> x 0) "positive"] [else "non-positive"]) +``` + +### CRITICAL: Things That DO NOT EXIST in Jerboa/Chez + +Claude frequently hallucinates these from Gerbil, Gambit, Racket, or R7RS training data. +**NONE of them are real in Jerboa/Chez. STOP and use the correct form.** + +#### AI Compatibility Aliases (these now work in the prelude) +The following names from other Scheme dialects are aliased in `(jerboa prelude)`: +- `hash-has-key?` → `hash-key?` (Racket) +- `hash-table-set!` → `hash-put!` (Racket) +- `directory-exists?` → `file-directory?` (Gambit) +- `eql?` → `eqv?` (Common Lisp) +- `random-integer` → `random` (Gambit) +- `read-line` → `get-line` wrapper (Gambit) — works with or without port arg +- `force-output` → `flush-output-port` wrapper (Gambit) — works with or without port arg +- `string-map` → char-level map (Racket/R7RS) — `(string-map f str)` + +#### Hallucinated Functions (still do NOT exist) +- `symbol<?` — use `(lambda (a b) (string<? (symbol->string a) (symbol->string b)))` +- `string-contains?` — use `(string-contains str sub)` (returns index or #f, NOT boolean) +- `define-struct` — use `(defstruct name (fields ...))` +- `raise` with a string — use `(error 'who "message" irritants ...)` +- `environment-bound?` — Gerbil-only. No direct Chez equivalent + +#### Gerbil/Gambit-isms (from training data — wrong in Jerboa) +- `time->seconds` — use `(time-second (current-time))` for epoch seconds +- `thread-sleep!` — Gambit. Use `(sleep (make-time 'time-duration 0 seconds))` +- `thread-yield` — no Chez equivalent. Use `(sleep (make-time 'time-duration 0 0))` as workaround +- `path-expand` with 2 args — Gerbil takes `(path-expand rel base)`. Jerboa takes 1 arg. Use `(path-join base rel)` for 2-arg version +- `process-status` — Gerbil. Use `(std misc process)` API in Jerboa +- `user-info-home` — Gerbil. Use `(getenv "HOME")` +- `the-environment` — Gerbil. Use `(interaction-environment)` in Chez +- `condition/report-string` — Gerbil. Use `(with-output-to-string (lambda () (display-condition c)))` +- `make-class-type` — Gerbil. Use `(defstruct ...)` or `(defclass ...)` in Jerboa +- `string-subst` — Gerbil. Not in prelude. Use `(string-replace str old new)` or implement manually +- `open-fd-pair` — Gambit. Does not exist in Chez; requires different API + +#### R6RS/Racket-isms (wrong variant) +- `make-equal-hashtable` — R6RS. Use `(make-hash-table)` from Jerboa prelude +- `arithmetic-shift` — Racket. Use `(bitwise-arithmetic-shift n k)` or `(ash n k)` in Chez +- `pregexp-match` — Racket. Use `(std text regex)` or `(std pregexp)` API in Jerboa + +### CRITICAL: Common Arity Mistakes + +- `(list-of? pred)` → returns a PREDICATE. It takes 1 arg. Use: `((list-of? number?) lst)` +- `(maybe pred)` → returns a PREDICATE. It takes 1 arg. Use: `((maybe string?) val)` +- `(in-range end)` or `(in-range start end)` or `(in-range start end step)` — NOT `(in-range start step end)` +- `(hash-ref ht key)` or `(hash-ref ht key default)` — NOT `(hash-ref key ht)` +- `(string-split str delimiter)` where delimiter is a CHAR: `(string-split "a,b" #\,)` +- `(make-rwlock)` — takes **0 args**, NOT `(make-rwlock 'name)` (Gerbil takes a name; Jerboa does not) +- `(path-expand path)` — takes **1 arg**, NOT `(path-expand rel base)` (Gerbil takes 2; use `path-join` for 2-arg) +- `(sort list predicate)` — Jerboa `(std sort)`/prelude order. Raw Chez `sort` is predicate-first, but Jerboa-facing code should use list first. + +### Core Forms (all from `(import (jerboa prelude))`) + +#### Definitions +```scheme +(def x 42) ;; variable +(def (f x y) (+ x y)) ;; function +(def (f x (y 10)) body) ;; optional param with default +(def (f x . rest) body) ;; rest args +(def* f ((x) ...) ((x y) ...)) ;; multi-arity +(defrule (name pat) template) ;; macro +``` + +#### Data Structures +```scheme +(defstruct point (x y)) ;; → make-point, point?, point-x, point-y, point-x-set! +(defstruct (circle shape) (radius)) ;; inheritance (single only) +(defmethod (area (self circle)) body) ;; method on type +(~ obj 'method arg ...) ;; dispatch (or {method obj arg ...}) +(defrecord person (name age)) ;; struct + pretty-print + ->alist +(define-enum color (red green blue)) ;; → color-red, color?, color->name +``` + +#### Pattern Matching +```scheme +(match value + (42 "exact") ;; literal + ((list a b c) (+ a b c)) ;; list destructure + ((cons h t) h) ;; pair + ((? number?) "num") ;; predicate + ((? string? s) (string-upcase s)) ;; predicate + bind + ((and (? number?) (? positive?)) "positive number") + ((or "yes" "y") #t) + ((=> string->number n) n) ;; view pattern + (n (where (> n 0)) "positive") ;; guard + (_ "default")) ;; wildcard +``` + +#### Error Handling +```scheme +(try expr (catch (e) handler) (finally cleanup)) +(try expr (catch (error? e) handler)) +(unwind-protect body cleanup) +(with-resource (var init cleanup) body) +``` + +#### Result Type (Rust-inspired ok/err) +```scheme +(ok 42) (err "bad") (ok? r) (err? r) +(unwrap (ok 42)) ;; → 42 (raises on err) +(unwrap-or (err "x") 0) ;; → 0 +(map-ok f result) (map-err f result) +(and-then result f) ;; monadic bind +(try-result expr) ;; exceptions → (err condition) +(try-result* expr) ;; exceptions → (err "message string") +(sequence-results list-of-results) ;; → (ok list) or first (err) +(->? (ok 10) (+ 5) (* 2)) ;; → (ok 30), short-circuits on err +``` + +#### Iterators +```scheme +(for ((x (in-range 5))) (displayln x)) +(for/collect ((x (in-range 5))) (* x x)) ;; → (0 1 4 9 16) +(for/fold ((sum 0)) ((x (in-range 10))) (+ sum x)) ;; → 45 +(for/or ((x lst)) (and (pred? x) x)) ;; first truthy +(for/and ((x lst)) (pred? x)) ;; all truthy + +;; Iterators: in-list, in-vector, in-string, in-range, in-hash-keys, +;; in-hash-values, in-hash-pairs, in-naturals, in-indexed, +;; in-port, in-lines, in-chars, in-bytes, in-producer +``` + +#### Threading Macros +```scheme +(-> x (f a) (g b)) ;; thread first: (g (f x a) b) +(->> x (f a) (g b)) ;; thread last: (g b (f a x)) +(as-> x v (f v) (g v)) ;; named +(some-> x (f) (g)) ;; short-circuit on #f +(cond-> x test (f) t2 (g)) ;; conditional steps +(->? (ok x) (f) (g)) ;; result-aware thread first +``` + +#### Ergo Typing +```scheme +(: expr pred?) ;; checked cast +(using (p (make-point 1 2) : point?) + (+ p.x p.y)) ;; dot-access → (point-x p) etc. +((list-of? number?) '(1 2 3)) ;; predicate factory → #t +((maybe string?) #f) ;; accepts #f or string → #t +``` + +#### Hash Tables +```scheme +(def ht (make-hash-table)) +(hash-put! ht "key" "val") +(hash-ref ht "key") ;; error if missing +(hash-ref ht "key" "default") ;; with default +(hash-get ht "key") ;; → val or #f +(hash-key? ht "key") ;; → #t/#f +(hash-remove! ht "key") +(hash->list ht) (hash-keys ht) (hash-values ht) +(hash-for-each (lambda (k v) ...) ht) +(list->hash-table '(("a" . 1) ("b" . 2))) +``` + +#### Strings +```scheme +(string-split "a,b,c" #\,) ;; → ("a" "b" "c") NOTE: char delimiter +(string-join '("a" "b") ",") ;; → "a,b" +(string-trim " hi ") ;; → "hi" +(string-prefix? "he" "hello") ;; → #t +(string-suffix? "lo" "hello") ;; → #t +(string-contains "hello" "ell") ;; → 1 (index, not boolean!) +(string-empty? "") ;; → #t +(str "age: " 42 "!") ;; → "age: 42!" (auto-coerce) +``` + +#### Lists +```scheme +(flatten '(1 (2 (3)))) ;; → (1 2 3) +(unique '(1 2 2 3)) ;; → (1 2 3) +(take lst n) (drop lst n) (take-last lst n) (drop-last lst n) +(every pred lst) (any pred lst) (filter-map f lst) +(group-by f lst) (zip lst1 lst2) (frequencies lst) +(partition pred lst) (interleave l1 l2) (mapcat f lst) +(distinct lst) (keep f lst) (split-at lst n) +(append-map f lst) (snoc lst elem) +``` + +#### Functional Combinators +```scheme +(compose f g) (comp f g) ;; (f (g x)) +(partial f arg ...) ;; partial application +(complement pred) (negate pred) ;; logical not +(identity x) (constantly v) ;; basic combinators +(curry f arg) (flip f) ;; currying, arg swap +(conjoin p1 p2) (disjoin p1 p2) ;; predicate AND/OR +(juxt f g) ;; → (lambda (x) (list (f x) (g x))) +(cut f <> y) ;; SRFI-26 partial: (lambda (x) (f x y)) +``` + +#### JSON +```scheme +(string->json-object "{\"key\":\"val\"}") ;; → hash table +(json-object->string ht) ;; → JSON string +(read-json port) (write-json obj port) +``` + +#### CSV +```scheme +(csv->alists "name,age\nAlice,30") ;; → (((name . "Alice") (age . "30"))) +(read-csv-file "data.csv") ;; → list of row lists +(write-csv-file "out.csv" rows) +``` + +#### DateTime +```scheme +(datetime-now) (datetime-utc-now) +(make-datetime 2026 3 27 12 0 0) +(parse-datetime "2026-03-27T12:00:00Z") +(datetime->iso8601 dt) (datetime->epoch dt) +(datetime-add dt duration) (datetime-diff dt1 dt2) +(datetime<? dt1 dt2) (day-of-week dt) (leap-year? 2024) +``` + +#### Paths +```scheme +(path-join "/home" "user" "f.txt") ;; → "/home/user/f.txt" +(path-directory "/a/b/f.txt") ;; → "/a/b" +(path-extension "file.txt") ;; → "txt" +(path-absolute? "/home") ;; → #t +``` + +#### File I/O +```scheme +(read-file-string "f.txt") ;; → entire file +(read-file-lines "f.txt") ;; → list of lines +(write-file-string "f.txt" "data") +``` + +#### Pretty Printing +```scheme +(pp expr) (pp-to-string expr) (pprint expr) +``` + +#### Formatting +```scheme +(format "~a is ~a" "Alice" 30) ;; → "Alice is 30" +(printf "x = ~a\n" 42) +(displayln "hello" " " "world") +``` + +#### Anaphoric / Conditional Binding +```scheme +(awhen (find x) (use it)) ;; binds result to `it` +(aif (find x) (use it) (default)) +(when-let (x (find y)) (use x)) +(if-let (x (find y)) (use x) (default)) +``` + +#### Loops +```scheme +(while test body ...) +(until test body ...) +(dotimes (i 10) body ...) ;; 0..9 +``` + +#### Misc Sugar +```scheme +(assert! (> x 0)) (assert! (> x 0) "message") +(alist (name "Alice") (age 30)) ;; → ((name . "Alice") (age . 30)) +(let-alist data (name age) body) +``` + +### What's NOT in the Prelude (requires separate import) + +```scheme +(import (std net request)) ;; HTTP client +(import (std net httpd)) ;; HTTP server +(import (std db sqlite)) ;; SQLite +(import (std actor)) ;; Actor system +(import (std async)) ;; Async/await +(import (std crypto digest)) ;; SHA, MD5 +(import (std text regex)) ;; Regex +(import (std text xml)) ;; XML +(import (std text yaml)) ;; YAML +(import (std os env)) ;; Environment variables +(import (std os signal)) ;; Signal handling +(import (std security sandbox)) ;; Sandboxing +``` + +### Chez Scheme Conflicts (handled by prelude) + +The prelude shadows these Chez builtins with Jerboa versions: +`make-hash-table`, `hash-table?`, `sort`, `sort!`, `printf`, `fprintf`, +`path-extension`, `path-absolute?`, `with-input-from-string`, +`with-output-to-string`, `iota`, `1+`, `1-`, `partition`, +`make-date`, `make-time` + +All standard Chez Scheme is still available — the prelude just re-exports +improved versions of the above. + +--- + +## Repository Boundaries + +When working in a Jerboa project, **ONLY modify files in the current repo** unless the user explicitly names another path. + +Common sibling repos that exist but must NOT be touched without explicit instruction: +- `~/mine/jerboa-emacs` — **NEVER touch**. Another model owns it. +- `~/mine/jerboa-mcp` — Legacy node MCP, superseded. The active MCP server now lives in THIS repo at `mcp/` + `data/`. Don't modify the legacy repo unless told. +- `~/mine/jerboa-shell` — Only modify when user explicitly says to work there. +- `~/mine/gerbil-mcp` — **NEVER touch**. Deprecated. +- `~/mine/gerbil-orig` — Read-only reference for upstream Gerbil. Never modify. + +If a user instruction mentions a file path, use EXACTLY that path. Do not substitute a similar-looking path from another repo. + +--- + +## Build & Verification + +After modifying any `.ss` or `.sls` (Jerboa Scheme) source files, always run the build command and fix any errors before moving on. Common issues include: missing imports, wrong function names, and duplicate definitions. + +### Stale Artifacts + +If edits seem to have no effect after `make build`, delete stale compiled files: +```bash +find lib -name "*.so" -delete && find lib -name "*.wpo" -delete && make build +``` +Run `jerboa_stale_static` to detect stale `.so` files before debugging "why doesn't my edit work?". + +## Pre-commit Requirements + +**ALWAYS** run a clean build **before** committing any code to this repository. Pick the right target for the *current* platform: + +- **Linux**: run `make docker-build` — the Docker image must build cleanly against the full musl-static release pipeline. +- **macOS / FreeBSD / other**: run `make binary` — the native local build must succeed. Do **not** run `make docker-build` here; Docker on non-Linux hosts is slow and not the canonical pipeline for those platforms. + +Do not commit if the build fails. + +## Act First, Read Less + +When making changes, read only what you need to make the edit, then make it. +Do not read more than 3 files before acting. Do not re-read files you already +read. Do not verify things you already know. If you have enough context to make +a change, make it. The user will interrupt you if you are wrong. + +## Jerboa MCP Tools — MANDATORY Usage + +Jerboa is a niche Scheme dialect with limited training data. **Never guess — always verify** with MCP tools. Tool descriptions are available at runtime via the MCP server; this section covers **when** and **why** to use each tool. + +### MANDATORY Workflow Order (for writing new code) + +1. **`jerboa_howto`** — BEFORE writing code, search cookbook for verified patterns +2. **`jerboa_module_exports`** / **`jerboa_function_signature`** — confirm APIs exist and check arities +3. Write the code +4. **`jerboa_verify`** — combined syntax + compile + lint + arity + duplicate check (use instead of individual tools) +5. **`jerboa_security_scan`** — for code involving FFI, shell commands, or file I/O + +### Essential Tools (use proactively) + +| When | Tool | +|---|---| +| Before writing ANY Jerboa code | `jerboa_howto` — cookbook has verified patterns with correct imports | +| Check what a module exports | `jerboa_module_exports` — never guess function names | +| Check function arity/args | `jerboa_function_signature` — prevents wrong-arg-count errors | +| Validate your code | `jerboa_verify` — one-stop syntax+compile+lint+arity check | +| Test expressions interactively | `jerboa_eval` — use `imports` param for modules, `env` for FFI paths | +| Persistent interactive testing | `jerboa_repl_session` — maintains state across evaluations | +| Debug an error message | `jerboa_explain_error` + `jerboa_error_fix_lookup` | +| Understand unfamiliar code | `jerboa_file_summary` + `jerboa_document_symbols` | +| Find where something is defined | `jerboa_find_definition` — source file, module, kind, arity | +| Search for symbol by substring | `jerboa_apropos` or `jerboa_smart_complete` | +| Build the project | `jerboa_build_and_report` or `jerboa_make` — prefer over bash `make` | +| Run tests | `jerboa_run_tests` — prefer over ad hoc shell test invocations | +| Check for stale .so artifacts | `jerboa_stale_static` — common cause of "edit has no effect" | +| Macro expansion | `jerboa_expand_macro` / `jerboa_trace_macro` | +| Inspect struct/class types | `jerboa_class_info` — fields, inheritance, constructor signature | +| FFI work | `jerboa_ffi_scaffold` / `jerboa_ffi_type_check` / `jerboa_ffi_null_safety` | +| Port Gerbil code | `jerboa_migration_check` + `jerboa_translate_scheme` | +| Detect paren imbalance | `jerboa_check_balance` — use BEFORE `make build` after deep edits | +| Full project audit | `jerboa_project_health_check` — balance, exports, cycles, duplicates | +| Security audit | `jerboa_security_audit` + `jerboa_import_policy_check` | +| Static build audit | `jerboa_static_symbol_audit` + `jerboa_boot_library_audit` | +| Explore a module | `jerboa_module_catalog` (replaces multiple `jerboa_doc` calls) | +| Look up any symbol | `jerboa_doc` — type, arity, qualified name, related symbols | +| Read stdlib source | `jerboa_stdlib_source` — see internal implementations | + +### Cookbook & Knowledge Management + +- **`jerboa_howto`** / **`jerboa_howto_get`**: Search and retrieve verified recipes +- **`jerboa_howto_add`**: Save new patterns to cookbook (MANDATORY when you discover something non-trivial) +- **`jerboa_howto_run`** / **`jerboa_howto_verify`**: Validate recipes still work +- **`jerboa_error_fix_add`**: Save error→fix mappings for common mistakes +- **`jerboa_anti_pattern_lookup`**: Search reusable local-model mistakes and failed strategies + +**The knowledge base is `data/*.sexp` in THIS repo**, embedded into `jmcp` at build time. The write tools above edit it live — the server reads `data/` from disk first, with the embedded copy as fallback (`JERBOA_MCP_REPO` points every client at this repo). When you add a stdlib/language feature, also update `data/` (a cookbook recipe + `api-signatures.sexp` + `changelog.sexp`) and **commit it**. Run `make jmcp` (or `make jmcp-portable`) only to refresh the embedded copy shipped in portable binaries. + +### Code Generation & Refactoring + +`jerboa_rename_symbol`, `jerboa_balanced_replace`, `jerboa_wrap_form`, `jerboa_splice_form`, `jerboa_scaffold_test`, `jerboa_generate_module`, `jerboa_translate_scheme`, `jerboa_project_template`, `jerboa_httpd_handler_scaffold`, `jerboa_db_pattern_scaffold`, `jerboa_actor_ensemble_scaffold` + +### Feature Suggestions + +- **`jerboa_list_features`** / **`jerboa_suggest_feature`** / **`jerboa_vote_feature`**: Track and submit tooling improvement ideas + +--- + +## MANDATORY: Save What You Learn + +Jerboa is niche — every non-trivial pattern you discover prevents future sessions from re-discovering it. + +### Save to cookbook (`jerboa_howto_add`) whenever you: +- Discover a working pattern through `jerboa_eval` or trial-and-error +- Figure out correct imports, arities, or calling conventions that weren't obvious +- Find a workaround for a Jerboa quirk or undocumented behavior + +**Before saving**: check `jerboa_howto` to avoid duplicates. **Do NOT save**: trivial one-liners, project-specific logic, or existing recipes. + +**Recipe format**: `id` (kebab-case), `tags` (4-6 search keywords incl. module name), `imports` (all required), `code` (complete working example), `notes` (gotchas/alternatives). + +### Save anti-patterns (`data/anti-patterns.sexp`) whenever you: +- See a plausible local-model strategy that failed verification +- Find a weak verifier pattern that allowed false success +- See a repeated repair loop, such as broad-reading after a concrete error +- Find a generic runtime mistake, such as missing lower-bound checks before vector access + +**Before saving**: check `jerboa_anti_pattern_lookup` to avoid duplicates. If none exists, call `jerboa_anti_pattern_add`; only edit `data/anti-patterns.sexp` directly if the writer tool is unavailable. Save the normalized reusable mistake, not the whole trace or benchmark name. + +**Anti-pattern format**: `id`, `title`, `kinds`, `severity`, `tags`, `pattern`, `avoid`, `advice`, `tools`. + +### Save error fixes (`jerboa_error_fix_add`) whenever you: +- See exact compiler/runtime/verifier text with a repeatable repair +- Hit an error that `jerboa_failure_advisor` should classify better next time +- Debug a local-model generated-code failure where a short diagnosis prevents another failed iteration + +**Before saving**: check `jerboa_error_fix_lookup` with the exact error text. **Do NOT save**: one-off project business-logic mistakes. + +**Error-fix format**: `id`, `pattern`, `fix`; optional `type`, `explanation`, `code_example`. + +### Suggest tooling improvements (`jerboa_suggest_feature`) whenever you: +- Make multiple sequential tool calls that could be one tool +- Fall back to bash because an MCP tool is missing or insufficient + +**Before suggesting**: check `jerboa_list_features`; vote with `jerboa_vote_feature` if it already exists. + +### Save Discoveries Mechanisms +- **`/save-discoveries` skill**: invoke anytime to review session and save recipes, anti-patterns, error fixes, feature suggestions, and security patterns +- **PreCompact hook**: add `PreCompact` hook with `type: "prompt"` in `.claude/settings.json` to auto-save before context compaction + +--- + +## Common Workflows + +- **Write new code**: `jerboa_howto` -> `jerboa_module_exports` -> write code -> `jerboa_verify` -> `jerboa_security_scan` +- **Debug an error**: `jerboa_explain_error` -> follow suggested tools -> `jerboa_howto` for fix patterns +- **Understand unfamiliar code**: `jerboa_file_summary` -> `jerboa_document_symbols` -> `jerboa_module_deps` +- **Refactor a module**: `jerboa_check_exports` -> `jerboa_find_callers` -> `jerboa_rename_symbol` -> `jerboa_check_import_conflicts` +- **Build project**: `jerboa_build_conflict_check` -> `jerboa_make` -> `jerboa_build_and_report` +- **Port from Gerbil**: `jerboa_migration_check` -> `jerboa_translate_scheme` -> `jerboa_verify` -> `jerboa_check_syntax` +- **Audit project quality**: `jerboa_verify` -> `jerboa_lint` -> `jerboa_dead_code` -> `jerboa_dependency_cycles` +- **Debug a crash**: `jerboa_stale_static` -> `jerboa_bisect_crash` -> `jerboa_ffi_type_check` +- **Learn a module**: `jerboa_stdlib_source` -> `jerboa_module_catalog` -> `jerboa_module_quickstart` +- **Security audit**: `jerboa_security_audit` -> `jerboa_import_policy_check` -> `jerboa_unsafe_import_lint` +- **Static build audit**: `jerboa_static_symbol_audit` -> `jerboa_boot_library_audit` -> `jerboa_rust_musl_build` +- **Safe-by-default check**: `jerboa_safe_prelude_check` -> `jerboa_resource_leak_check` -> `jerboa_safe_prelude_generate` +- **Debug editor command**: `jerboa_command_trace` with `project_path` and `buffer_type` + +## Workflow Conventions + +When implementing new features, always complete the documentation update in the same session. Document non-trivial solutions as howto recipes in the cookbook system. new file mode 100644 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,586 @@ +# Architecture + +## Data path + +```text +seed -> durable frontier -> guarded fetch -> sandboxed HTML parser + |-> normalized links -> frontier + `-> body blob + metadata -> BM25 postings + `-> query API/UI +``` + +`jsqlite` is the source of truth for page metadata, body digests, frontier +state, links, and a monotonically increasing index generation. Page text is an +immutable SHA-256-addressed blob below `<database>.content/`; identical text is +stored once. Existing blobs are checksum-verified on write reuse and repaired +through the same temporary-file replacement path if corrupted. One database +connection and blob-reference transition are +serialized behind a store mutex because crawl workers and HTTP request threads +share them. Updating a page and advancing its generation is one database +transaction. + +After parsing a page, its complete outgoing-link replacement and all eligible +frontier additions are committed in one batch transaction. This avoids one +mutex acquisition and one database operation cycle per discovered URL while +retaining the frontier's unique-URL guarantee. Frontier admission batches the +URL existence lookup, then applies ordered per-URL admission so capacity, +cross-run transfer, and duplicate accounting remain deterministic. +Outgoing link rows are inserted with bounded multi-row statements after a single +delete of the prior graph, and crawlable membership is computed from a hash set +rather than by rescanning the crawl-link list for every URL. + +`(jerboa-index core)` owns the derived inverted index in Jerboa memory: a term +dictionary, per-term postings, document lengths, title frequencies, and BM25 +ranking. `(jerboa-index segments)` checkpoints only changed documents and +tombstones into immutable files below `<database>.jerboa-index.segments/`, then +atomically publishes `<database>.jerboa-index` as the new manifest. Both file +types have versioned, SHA-256-checked, bounded safe-FASL envelopes. + +The core stores total document length alongside its document table and updates +the aggregate under the same mutex as insert, replacement, removal, and +snapshot application. Snapshot construction recomputes it while validating +document rows. Both core BM25 queries and hybrid overlay statistics can thus +derive average length without walking every live document. The aggregate is +derived and is never serialized as an independently trusted value. +For a single-term core query, each posting's BM25 score is complete immediately +and streams into the shared bounded top-k collector. Multi-term queries retain +the document score table required to combine contributions, but the common +fallback single-term path uses `O(limit)` ranking state instead of +`O(matches)`. + +Startup validates the manifest generation, descriptor count, contiguous +generation ranges, safe filenames, checksums, closed payload schemas, and +tombstone uniqueness before accepting the chain. It applies deltas to an empty +core index in order. Any failure rebuilds from SQLite and body blobs and +publishes a new base segment. The segment chain is therefore a disposable +cache, never a second source of truth. + +`(jerboa-index mmap)` implements the next query representation as a versioned +binary file: fixed-width sorted document and term tables, UTF-8 term slices, +fixed-width posting lists whose document references are table ordinals, and +compact block-max summaries for terms spanning more than one 128-posting block. +Each summary stores maximum body and title +frequency plus minimum document length. It validates magic/version/generation, +declared file size, a length-bound hierarchical SHA-256 payload checksum, +monotonic IDs and terms, every offset and count, posting ordinal and frequency, +and every recomputed block summary before exposing the mapping. Queries +binary-search terms and read posting IDs and lengths directly from `(std mmap)`. +Constant-sized WAND cursor caches avoid repeated reads of the current posting; +no base-sized document metadata hash remains in the Scheme heap. Ranking is +regression-tested against `(jerboa-index core)` to identical BM25 scores. +Version 4 hashes fixed 1 MiB payload chunks and then hashes their length-prefixed +digest manifest, bounding checksum working memory independently of sidecar +size. Version-1 through version-3 sidecars are disposable and regenerate +through the normal invalid-sidecar fallback. + +`native/qix_validate.c` is a dependency-free, ABI-canary-checked acceleration +boundary for structural validation. It validates the document table, +dictionary, postings, and block summaries in one native pass. Existing +`(std crypto native)` SHA-256 hashes bounded 1 MiB bytevectors copied from the +mapping, avoiding both per-byte Scheme hashing and a sidecar-sized allocation. +Jerboa first validates the header and all top-level offsets, owns mmap lifetime, +hashes the small length-prefixed digest manifest with its pure implementation, +and turns native diagnostics into ordinary conditions. Tests recompute complete +QIX checksums with the pure implementation and present checksummed structural +corruption, so acceleration cannot silently weaken format compatibility or +validation. The development loader accepts only its canonical, validated build +directory; a static host may pre-register the same ABI symbols. + +The qix publisher computes all fixed offsets from its locked core read view, then +emits the header, document table, dictionary, term data, postings, and block +summaries sequentially through the same 1 MiB checksum buffer. Native SHA-256 +removes per-byte Scheme work without an external crypto dependency. After the +final chunk it seeks back to write the digest and atomically renames the completed +temporary file. Peak publication memory is therefore independent of sidecar +bytes. A mutex-scoped core read view retains sorted document IDs and term names, +then handles and releases one term's posting IDs at a time. Terms covering at +least one eighth of the corpus scan the already sorted document table and test +posting membership, avoiding a second nearly corpus-sized sort. Sparser terms +sort only their own IDs; their monotonic streams use forward galloping search +to derive document-table ordinals in constant work for adjacent postings and +logarithmic work across gaps. Posting records are written directly into the +existing checksum chunk with endian-aware Jerboa bytevector stores, with the +generic streaming writer handling only records that cross a chunk boundary. +The publisher therefore needs neither a full immutable snapshot, a +corpus-sized document-ordinal hash, nor one tiny bytevector copy per posting. + +The service uses this mapping as its query base whenever the sidecar generation +matches the chain's base generation. A cumulative overlay contains documents +changed after that base, while an override set suppresses replaced and +tombstoned base postings. `(jerboa-index hybrid)` computes document count, +average length, per-term document frequency, and BM25 scores across the unified +view; it does not merge independently ranked result lists. A missing, stale, or +corrupt sidecar falls back to the full core and can be regenerated by +compaction. + +Mapped term document frequency comes directly from the validated dictionary. +For live replacements and tombstones, the adapter subtracts only overridden +document IDs found by binary search in the sorted posting list. The override +set is bounded by compaction policy, so hybrid queries no longer traverse each +base posting list once to count and again to score. + +Core, mapped, and hybrid queries share `(jerboa-index top-k)`. Once multi-term +BM25 scores are accumulated, a bounded worst-first heap selects results in +`O(matches log limit)` time and `O(limit)` selection space instead of sorting +and materializing every scored entry. Equal scores consistently prefer the +smaller document ID. Every single-term engine streams each complete score directly +into the bounded heap, retaining `O(limit)` ranking state instead of an +`O(matches)` score hash. Mapped traversal also derives a safe BM25 upper bound +for each posting block from its summary and skips blocks that cannot beat the heap root; +ascending document IDs make equal-score skips safe under deterministic tie +ordering. + +`(jerboa-index wand)` implements exact document-at-a-time weak-AND over +monotonic posting cursors. Each mmap cursor has a validated term-wide BM25 upper +bound and binary `advance-to` seek. Cursors are ordered by current document ID; +their bounds identify a pivot, only matching pivot documents are fully scored, +and the search terminates when all remaining bounds can at best tie the heap +root. Hybrid base cursors use global live document count, average length, and +document frequencies while rejecting override IDs. Once a pivot reaches the +front of the cursor set, participating base cursors sum validated block-local +bounds. When that sum cannot enter top-k, all cursors seek to the minimum of +the participating block-exclusive ends and every nonparticipating cursor's +next document. No new term can enter before that boundary, so the entire +half-open document range is safely skipped. The bounded overlay is scored with +the same global statistics, then merged with base top-k; scores are therefore +comparable and the union remains exact. Using block maxima during the earlier +pivot-selection phase would tighten sparse mixed-frequency queries further. + +Constructing a WAND source needs each term's maximum body frequency, maximum +title frequency, and minimum document length. Those raw values depend only on +the immutable mapping, but deriving them otherwise scans every block summary on +every query. Each mapping therefore owns a mutex-protected, direct-mapped cache +of 4,096 term-statistic entries. It is strictly bounded, supports concurrent +readers, and caches no overlay-dependent BM25 value. A collision only replaces +one entry and cannot affect correctness. + +Each managed index has a writer-preferring Jerboa read/write lock. Search takes +the store mutex only long enough to acquire an index read lease, then releases +the store while traversing immutable mapped postings; multiple mapped searches +can run concurrently. Overlay mutation, checkpoint publication, compaction +swap, and mapping close take the write side. The global order is always store +mutex then index lock. Search releases its read lease before reacquiring the +store to hydrate result metadata and snippets, preventing lock inversion. +Hydration uses prepared `id = ?` INTEGER PRIMARY KEY point lookups; current +jsqlite measurements make these faster than equivalent `UNION ALL` compound +statements, while `IN (...)` currently falls back to a table scan. Rows are +mapped by ID and replayed in top-k order so missing rows cannot perturb the +remaining ranks. + +Search pagination retains deterministic score/document-ID ordering by asking +the same ranking engine for exactly `offset + limit + 1` candidates, slicing +that ranked hit list before metadata hydration, and using the extra hit only to +derive `has_more`. The HTTP boundary caps `limit` at 100 and `offset` at 1,000, +so top-k state, hydration work, and response size remain bounded. No full +corpus count or second query is required. HTML navigation percent-encodes the +original query; the JSON API reports effective bounds and previous/next offsets +explicitly. +The boundary parses the query text once into the shared bounded term list; core, +mmap, and hybrid ranking all accept that list directly, and snippets reuse it. +Snippet generation derives the first eight display terms once per response and +uses the original bounded body window directly when it contains no uppercase +characters, falling back to a lowercased window only when case folding is +actually needed. +The older string-query entry points remain as compatibility wrappers. + +Store ownership also defines derived-resource lifetime. `(jerboa-search index)` +registers a close hook with `(jerboa-search store)`; `close-search-store` +acquires the store mutex, takes the index write lock, closes any mmap, removes +the process registry entry, and only then closes jsqlite. Close is idempotent, +and later store operations fail as closed instead of using a stale handle. + +After a valid mapping is active, the adapter releases references to both the +managed full core and the segment chain's reconstructed core. New documents and +replacements update only the cumulative overlay and the pending checkpoint +delta. Status derives exact document counts from mapped metadata plus overrides. +Term counts use the mapped header plus binary membership checks for bounded +overlay terms, avoiding a base-wide scan. Add-only overlays remain exact; +replacements or tombstones may leave dead base terms in the reported upper +bound, so `/api/status` exposes `terms_exact=false` until compaction. +Fallback or explicit compaction may reconstruct a full core, but successful +sidecar installation releases it again; `/api/status` exposes `core_resident` +so this invariant is observable. + +On startup the adapter first validates the manifest generation and base +generation, then opens and validates the binary sidecar. When successful it +skips safe-data decoding of the base segment entirely and replays only later +delta segments into the overlay (`source=mmap-direct`). Missing, stale, or +corrupt sidecars take the full replay path. A compacted 1,000-document base +starts in 56 ms in the current development benchmark. + +Initial rebuild, explicit compaction, and background compaction publish binary +sidecars. Background workers prepare them off-lock. After generation +revalidation, the authoritative safe-data manifest is committed first and the +optional sidecar is swapped second; sidecar failure never rolls back the +manifest or invalidates the previous mapped query view. + +Checkpointing schedules compaction on a background Jerboa thread at eight +segments or 1,000 changed document IDs since the mapped base, whichever comes +first. This bounds both chain traversal and the amount of overlay state replayed +on direct startup; `JERBOA_SEARCH_MAX_OVERLAY_DOCUMENTS` tunes the document +threshold. The worker reconstructs only the immutable generation it was given +and prepares a new base without holding the store lock. Publication briefly +takes the lock and proceeds only if that generation is still current; otherwise +the prepared file is discarded. This prevents a slow compactor from rolling a +newer manifest backward. + +Compaction publishes the new base before deleting files explicitly named by the +prior validated manifest. It does not recursively scan the segment directory, +so cleanup cannot follow an injected symlink or delete an unrelated path. +Manual synchronous compaction remains available at `POST /api/index/compact`. +The crawler checkpoints every 1,000 processed pages by default, bounding the +duplicate in-memory delta; `JERBOA_SEARCH_CHECKPOINT_EVERY` tunes that interval. + +Fetched page metadata, generation advance, outgoing-link replacement, and new +frontier URLs commit in one jsqlite transaction. Only after that authoritative +commit does the adapter mutate its derived text index; an index-side failure +closes and drops the cache so the next access rebuilds from committed truth. +This removes the former per-page split transaction and prevents a crash from +publishing a page without its discovered crawl graph. + +Bulk indexing uses the same authoritative store/index boundary but commits a +caller-provided document batch in one jsqlite transaction and advances the +generation once for the batch. The live derived index is then updated under the +managed index write lock for each committed page ID. Single-page crawler writes +keep their per-page durability; the batch API exists for controlled import and +reindexing paths where the caller can choose the batch size. + +Content collection is explicit and quiescent: `/api/content/gc` obtains the +store lock, derives the live digest set from SQLite, and deletes only +unreferenced blobs and abandoned temporary files at managed content-store paths +of the form `root/xx/<digest>` or `root/xx/<digest>.tmp-*`. The HTTP endpoint +rejects GC while the crawler is running. `/api/crawl/stop` sets the stop flag, +joins every Jerboa worker, and checkpoints only after the final in-flight +response exits; the worker reservation counter also makes `max-pages` a hard +concurrent cap. + +## Administrative boundary + +Search, status, health, and static presentation are read-only public routes. +Every state-changing route passes through one before-filter. When +`JERBOA_SEARCH_ADMIN_TOKEN` is configured, the filter requires an RFC 6750-style +bearer header and compares its credential with Jerboa's timing-safe string +comparison. The accepted token is bounded to 32–1024 characters and remains in +the app-construction closure; application settings retain only an enabled flag. +Authentication failures stop routing before a request body is parsed or a +crawler/index/content-store operation runs, return `401` JSON, advertise the +bearer challenge, and prohibit response caching. + +Token-free operation is intentionally a local development mode. The listener +guard accepts only explicit loopback bind names in that mode and fails before +opening a public socket for any other bind. A loopback listener forwarded by a +reverse proxy is no longer a local trust boundary, so deployments using a proxy +must configure the bearer token as well. This boundary controls service +mutation; outbound crawler SSRF controls remain independent and apply to every +fetch regardless of authentication. + +The built-in listener is plain HTTP. Bearer authentication does not provide +transport secrecy, so deployments must expose it through a TLS-terminating +reverse proxy and keep the proxy-to-service hop on loopback or a trusted private +network. Direct Internet exposure of the listener is outside the supported +security model. + +## Process lifecycle + +The launcher converts `SIGINT` and `SIGTERM` into a flag only; no allocation, +locking, networking, or database work runs in OS signal context. The main +Jerboa thread observes that flag within 100 ms and executes one idempotent +shutdown closure. It first closes HTTP admission and waits up to 30 seconds for +active request threads, then joins crawler workers. Crawler stop publishes the +last incremental checkpoint. Shutdown waits for any resulting background +compaction before closing the managed mmap view and SQLite connection through +the store's registered owner hooks. + +Each cleanup stage is guarded so one failure does not suppress later cleanup; +the first failure is re-raised after all stages have been attempted. The +shutdown closure marks itself before releasing native handles, so repeated or +concurrent calls cannot double-close them. Integration coverage terminates the +real launcher with `SIGTERM`, requires its completion marker, and restarts on +the same port and database to prove listener and storage ownership were +released. + +## Admission and input bounds + +Jerboa's HTTP server enforces global connections, per-IP connections, +per-IP accept rate, and idle/header/body deadlines before Sinatra dispatch. +The launcher exposes bounded environment overrides for those controls, and +`/api/status` reports the effective values so deployment automation can verify +configuration. Per-IP concurrency is additionally clamped to the configured +global concurrency. + +The transport parser has independent 16 KiB header and 4 MiB body ceilings. +The application before-filter imposes a stricter 64 KiB body policy before +route-level form decoding. Search routes reject queries beyond 1,024 characters +before tokenization or index access, and crawl routes reject seeds beyond 4,096 +characters before URL parsing or SSRF resolution. These layered limits keep +memory and CPU cost bounded even when a client sends syntactically valid but +pathological input. + +## Health and telemetry + +`/healthz` is deliberately a shallow liveness probe, allowing an orchestrator +to distinguish a wedged process from a temporarily unready dependency. +`/readyz` exercises authoritative store/index status and fails with `503` when +that access raises or the compactor records an error. It emits only bounded +state—generation and query-engine name—and suppresses internal condition text +and paths on failure. + +`/metrics` refreshes a private `(std metrics)` registry during each scrape. +All series are fixed, label-free gauges, preventing attacker-controlled label +cardinality from becoming a memory sink. Registry refresh and serialization +share one mutex so concurrent scrapes cannot observe half-updated values. +Metrics contain aggregate crawler/index state only; URLs, search terms, bearer +credentials, content paths, and exception messages are excluded. A failed +refresh preserves exposition format, sets `jerboa_search_up` to zero, and +returns `503`. +`/api/status` follows the same public boundary for sensitive internals: it +reports aggregate crawl/index state and effective limits, while omitting seed +URLs, local manifest/content paths, and raw compaction exception text. +Bounded-state gauges cover compiled robots-cache occupancy and capacity, live +per-origin politeness deadlines, and cumulative deadline expiry events so +operators can verify that hostile cross-origin discovery does not retain +origins forever. + +## Crawl policy + +Depth, page budget, worker count, per-host delay, origin scope, checkpoint +interval, and retry bounds are immutable for a crawler instance and selected at +process startup. +The launcher applies conservative defaults and hard upper bounds before calling +the crawler constructor; the constructor independently validates types and +lower bounds so in-process library users cannot create nonsensical negative or +zero-worker configurations. A zero page budget is an intentional disabled-crawl +mode used by lifecycle checks. Effective policy is visible in `/api/status` and +fixed-name metrics. + +The page-attempt budget also derives a separate durable frontier ceiling: ten +times `max-pages`, floored at 1,000 for enabled crawls and capped at 1,000,000. +`crawl_runs.frontier_limit` survives pause/restart and is refreshed when a run +resumes. Batch enqueue reads remaining capacity once inside its existing SQLite +transaction, then admits new or transferred URLs until full; same-run +duplicates are free. HTML commits, 304 edge replay, and sitemap expansion all +use this path. Status and `jerboa_search_crawl_max_frontier` expose the limit. +Rejected discovery attempts are accumulated transactionally in +`crawl_runs.discovery_dropped` and exposed as `discovery-dropped` plus the +fixed-name `jerboa_search_crawl_discovery_dropped` metric. Repeated same-run +URLs do not increment it; rejected absent or cross-run URLs do. + +Before a worker processes a claimed row it must acquire a crawler-local host +lease. A collision returns the row to `queued` with a short durable deadline +and reverses the claim's attempt increment. The worker then seeks other due +work, so one origin cannot monopolize the pool. The lease is released through +`dynamic-wind`; the existing host clock still spaces each robots, sitemap, and +page request and host work remains serialized. + +Same-origin mode is a discovery boundary, not the SSRF boundary. Disabling it +allows links to other origins into the frontier, but each fetch and redirect +still traverses the existing public-HTTPS network policy, DNS rebinding checks, +unsafe-port rejection, robots rules, and host-specific scheduling. The global +reservation counter continues to enforce the page cap across every origin and +worker concurrently. When same-origin mode is enabled, a redirect's final URL +is checked against the seed origin before any status transition or indexing. + +Crawler-local waits use 25 ms stop-aware slices. If shutdown interrupts a host +slot wait, a private sentinel unwinds to the row processor, which durably +returns the row to `queued` and reverses its claim attempt. This bounds local +wait latency without shortening normal politeness; active synchronous network +calls still observe their existing request deadlines before workers join. + +Successful response admission is explicit: exact `text/html` and +`application/xhtml+xml` media types enter the sandboxed document path, exact +`text/plain` enters the text-only path, and every other media type is marked +skipped without creating a page row or incrementing `indexed`. Ordinary page +fetch bodies are capped by the shared network policy and exposed as +`page-body-bytes` in crawl status. +Policy and media skips share the status counter and fixed +`jerboa_search_crawl_skipped` metric. + +Tokenization is an allocation boundary as well as text normalization. It scans +by string index, ignores alphanumeric runs beyond 128 characters, and emits at +most 50,000 tokens per document with title priority. The shared +`document-token-lists` helper drives live core/overlay/delta mutation and the +authoritative `token_count`; rebuilds enter through the same core upsert path. +Thus an 8 MiB dense response cannot create millions of posting-list cells or +an unbounded hash key, and every derived representation sees identical lengths. + +Query parsing scans at most 256 tokens and retains the first 32 distinct terms. +The same `query-terms` function feeds core, mmap, hybrid overlay scoring, +hydration, and snippets. This makes per-request dictionary lookups and WAND +cursor fanout constant-bounded while preserving deterministic prefix semantics +across every query representation. +Paged search windows are capped before ranking so direct library callers cannot +materialize unbounded top-k collectors. `/api/status` exposes the +`search_page_window` ceiling. + +Robots rules are an independent admission gate before the page request. The +parser combines every group whose product token exactly matches +`JerboaSearch`, falls back to `*` only when there is no exact group, preserves +groups across blank lines, and applies the longest matching pattern with allow +winning an equal-length tie. Its `*`/terminal-`$` matcher uses a remembered-star +linear scan rather than recursive backtracking. Before comparison, rules and +URI paths share an ASCII octet form: encoded unreserved bytes decode, retained +percent triplets use uppercase hex, raw non-ASCII becomes percent-encoded UTF-8, +and URI `*`/`$` characters become data encodings. Rule specificity counts a +percent triplet as its represented octet, not three source characters. +Successful policy responses +are cached per origin for the run; permanent `4xx` absence caches allow-all. +Network/server failures and conservative throttle/redirect outcomes are not +cached and raise a structured network condition, feeding the existing durable +page retry path instead of silently bypassing robots policy. + +`Sitemap:` is parsed independently of user-agent groups, under the same +8,192-character per-line ceiling as robots directives. Same-origin canonical +locations become `kind=sitemap` frontier rows, so pause, crash recovery, retry +deadlines, deduplication, and crawl-run ownership require no parallel queue. +Successful URL sets add `kind=page` rows at seed depth; sitemap indexes add +same-origin sitemap rows up to three levels deep. The worker accepts at most +4 MiB and 50,000 unique locations per document; these limits and the nesting +limit are visible in crawl status. Its purpose-built XML subset does not load +external entities or DTDs and rejects nested markup in plain `loc` content. +Sitemap fetches are included in the global attempt reservation and reported +separately as `sitemaps-this-run` and +`jerboa_search_crawl_sitemaps`. + +After a successful response, document directives independently control index +and link admission. The HTML parser accumulates case-insensitive generic +`robots` and product-specific `JerboaSearch` meta directives. The crawler then +combines them with generic or `JerboaSearch`-scoped `X-Robots-Tag` directives; +restrictions are cumulative, `none` maps to `noindex,nofollow`, and unrelated +user-agent scopes do not apply. Colon-valued directives such as `max-snippet` +are recognized as directives rather than mistaken for user-agent prefixes. + +Indexability is an explicit persisted page property, independent of HTTP +status. A successful `noindex` replacement commits its status, body reference, +metadata, links, and generation, then tombstones the document in the derived +live index. SQLite rebuilds select only successful, indexable rows, so unload, +restart, missing cache, or compaction cannot resurrect excluded content. A +later indexable fetch of the same URL restores it normally; legacy page tables +migrate existing rows with `indexable=1`. + +Follow policy is deliberately separate from graph capture. Page-level +`nofollow` suppresses every outgoing frontier addition, while per-anchor +`rel=nofollow` suppresses only that target. Both kinds of links remain in the +authoritative outgoing graph, preserving provenance and future graph-analysis +options without causing crawler discovery. + +Link resolution first computes the HTML document base from the first +`base[href]`; an invalid or non-web value freezes the fallback response URL and +does not allow a later base to take over. Canonicalization applies RFC 3986 dot +segment removal while preserving repeated and trailing path slashes, handles a +query-only reference against the current document path, and rejects every +explicit scheme other than HTTP(S). This prevents semantically distinct paths +from collapsing to one frontier key. +Raw whitespace, including leading or trailing whitespace, control characters, +and backslashes are rejected at the shared URL boundary before trimming or +resolution instead of being normalized like browser input. +Equivalent percent encodings are collapsed conservatively: ASCII unreserved +bytes are decoded, retained triplets use uppercase hex, and malformed triplets +are rejected. Dot removal runs after safe decoding, while encoded reserved +delimiters such as `%2F` remain data and cannot change path structure.