λgitsite

~ober/jerboa-search

Imported from ~/mine/jerboa-search

download snapshot

about

# jerboa-search

`jerboa-search` is a Jerboa-native crawler and full-text search engine. It is
an equivalent product to the useful single-node core of YaCy, not a protocol or
data-format port.

It composes existing local projects:

- `jsqlite` stores authoritative document metadata, crawl state, links, content
  references, and the index generation.
- `jerboa-websearch` provides SSRF-safe HTTPS fetching and sandboxed HTML5
  parsing.
- `jerboa-sinatra` serves the search UI and JSON crawl/search APIs.
- Jerboa supplies native threads, HTTP, JSON, WASM isolation, and build tooling.

The new search-specific pieces are a BM25 inverted index and a
content-addressed body store. The body store deduplicates immutable page text
outside SQLite. Generation-bound, checksummed immutable index segments make
checkpoints proportional to changed documents and make ordinary restarts fast.
A missing, stale, discontinuous, or corrupt segment chain is discarded and
rebuilt from authoritative metadata and content. There is no YaCy/Solr
compatibility layer.

The mutable core maintains document count through its document table and total
document length as an incremental invariant. BM25 average length and hybrid
overlay statistics are therefore constant-time metadata reads rather than an
unconditional document-table traversal on every query. Single-term core queries
stream final BM25 scores directly into the bounded deterministic top-k heap;
only multi-term queries need a score accumulator.

The qix version-4 sidecar stores posting document references as validated
ordinals into its fixed-width mapped document table. IDs and lengths are read
directly from that table, so the immutable base needs no corpus-sized Scheme
metadata hash; hot WAND cursors cache only their current posting. Its
length-bound hierarchical SHA-256 checksum validates fixed 1 MiB chunks, so
opening a large mapping never copies the complete payload into the Scheme heap.
The build produces a small dependency-free native QIX shim that validates
fixed-width records, posting invariants, and block summaries in one bounded
pass. Jerboa's existing native digest hashes each chunk, while Jerboa retains
mapping ownership, layout checks, error handling, and the final length-prefixed
manifest hash. The checksum and QIX v4 bytes are unchanged and
regression-checked against the pure Jerboa hash.
Publication uses the same bounded chunks as its sequential output buffer and
seeks back only to install the final checksum before atomic rename; it does not
materialize a second sidecar-sized bytevector. A mutex-scoped sorted core view
streams documents and one term's postings at a time. Dense postings reuse the
already sorted document table; sparse postings sort only their own IDs and use
forward galloping lookup. Fixed-width posting fields are stored directly into
the checksum chunk with Jerboa's endian-aware bytevector operations, avoiding
millions of tiny intermediate copies without changing QIX v4 bytes.

All query engines use a bounded deterministic top-k heap rather than sorting
every match. Ranking selection scales with `log(limit)` and retains only the
requested candidates. Single-term core, mapped, and hybrid queries stream scores
directly into that heap. Mapped and hybrid traversal additionally uses validated
128-posting block maxima to skip blocks whose BM25 upper bound cannot enter
top-k. Multi-term mapped queries use
exact weak-AND (WAND) cursors with binary `advance-to` seeks and shared hybrid
statistics, so they score only pivot candidates whose safe term-bound sum can
beat top-k. Participating cursors then sum validated block-local bounds before
exact BM25 scoring; when that sum cannot win, binary seeks advance every cursor
past the complete half-open range covered by those bounds. Block-max-aware
pivot scheduling across virtual block boundaries remains a possible research
refinement; it requires separate evidence because substituting a current-block
bound directly into ordinary WAND pivot selection is not safe.
Immutable term-wide maxima are reconstructed once from block summaries and
retained in a mutex-protected 4,096-slot direct-mapped cache. Cache entries hold
only raw QIX statistics, so hybrid document counts and frequencies remain live;
collisions merely trigger another safe scan and memory stays corpus-independent.

Mapped queries share a per-index Jerboa read lock and do not hold the SQLite
store mutex while scoring postings. Writers and atomic index publication use
the exclusive side, while result metadata hydration happens after the read
lease is released. Retained result rows are fetched with prepared primary-key
point lookups and replayed in rank order.

Closing a search store automatically unloads its managed index and mmap.
Explicit `unload-search-index!` remains useful for restart/fallback tests but is
not required for ordinary ownership cleanup.

Crawler ingestion commits each page, generation, outgoing links, and newly
discovered frontier URLs atomically before updating the disposable text index.
Controlled bulk import paths can use `index-documents!` to commit a document
batch with one generation advance, while ordinary crawler writes retain
per-page durability.

## Run

Building requires a C compiler for the hardened QIX validation shim.
No external crypto library is required.
Every Scheme test converts a false `run-tests!` result into process status 1,
so assertion failures stop the aggregate target instead of merely printing a
failure summary. Migration fixtures also remove a same-name interrupted-run
database before setup and pass cleanup procedures—not invoked return values—to
`with-resource`.

```sh
make test
make integration
N=1000 make bench
N=100000 ABSENT_ROUNDS=1000 COMMON_ROUNDS=10 make core-query-bench
ROUNDS=100000 REPEATED=8 make query-terms-bench
LINKS=600 ROUNDS=100 make link-replacement-bench
URLS=200 ROUNDS=20 make frontier-enqueue-bench
N=1000 make restart-bench
N=1000 make segment-bench
N=1000 make mmap-bench
make mmap-capacity-bench CAPACITY_N=1000000 CAPACITY_ROUNDS=1000
N=100000 ROUNDS=10 LIMIT=20 make ranking-bench
N=1000 WORKERS=4 QUERIES=500 make concurrency-bench
N=1000 LIMIT=20 ROUNDS=1000 make hydration-bench
N=1000 LIMIT=20 OFFSET=100 ROUNDS=1000 make search-page-bench
N=1000 LIMIT=20 OFFSET=100 BODY_PREFIX=10000 ROUNDS=1000 make search-page-bench
N=10000 OVERRIDES=100 ROUNDS=1000 make document-frequency-bench
N=1000 make ingest-transaction-bench
MODE=file-store-batch N=2000 make memory-scaling-bench
make run
```

Open <http://127.0.0.1:8890>. The first crawl must use a public HTTPS URL.
Crawler fetches reject loopback, private networks, IP literals, unsafe ports,
and unsafe redirects.

`SIGINT` and `SIGTERM` request an ordered shutdown. The listener stops accepting
connections, active HTTP requests drain, crawler workers join, the current
index delta checkpoints, background compaction quiesces, and mapped-index plus
SQLite resources close. Repeated shutdown calls are harmless. A drain or
compaction failure is reported only after every remaining cleanup stage has
been attempted, causing a non-zero service exit instead of claiming a clean
shutdown.

Environment overrides:

- `JERBOA_RUNTIME_HOME` — one coherent Jerboa runtime/library home (defaults
  to `jerbuild --jerboa-home`; do not mix it with a source checkout containing
  separately compiled `.so` files)
- `JERBOA_SEARCH_DB` — database path (default `data/search.db`)
- `JERBOA_SEARCH_PORT` — listen port (default `8890`)
- `JERBOA_SEARCH_BIND` — listen address (default `127.0.0.1`)
- `JERBOA_SEARCH_ADMIN_TOKEN` — optional bearer token protecting every
  state-changing HTTP route; it must contain 32–1024 characters. A
  non-loopback bind is refused unless this is configured.
- `JERBOA_SEARCH_WORKERS` — crawler workers (default `4`)
- `JERBOA_SEARCH_MAX_DEPTH` — maximum discovered-link depth (default `3`,
  capped at `32`; `0` fetches only the seed)
- `JERBOA_SEARCH_MAX_PAGES` — hard page-attempt budget per crawl (default
  `1000`, capped at `10000000`; `0` disables fetching)
- `JERBOA_SEARCH_DELAY_MS` — minimum delay between requests to the same host
  (default `500`, capped at ten minutes; `0` is allowed). Crawler-local waits
  are interruptible in 25 ms slices, so stop does not wait out the interval.
- `JERBOA_SEARCH_SAME_ORIGIN_ONLY` — whether discovered links must share the
  seed's scheme/host/port (default `true`; accepts `true/false`, `yes/no`,
  `on/off`, or `1/0`)
- `JERBOA_SEARCH_CHECKPOINT_EVERY` — processed pages between incremental index
  checkpoints (default `1000`, capped at `1000000`)
- `JERBOA_SEARCH_MAX_OVERLAY_DOCUMENTS` — changed-document threshold for
  scheduling background segment compaction (default `1000`, capped at
  `1000000`)
- `JERBOA_SEARCH_PAGE_UPSERT_BATCH_SIZE` — maximum rows per bulk page-upsert
  statement for controlled import/reindex paths (default `500`, capped at
  `500`; ordinary crawler writes remain per page)
- `JERBOA_SEARCH_MAX_ATTEMPTS` — total attempts allowed per URL, including the
  initial request (default `3`, capped at `16`)
- `JERBOA_SEARCH_RETRY_BASE_MS` — initial delay before retrying a transient
  fetch failure (default `1000`, capped at ten minutes; `0` is allowed)
- `JERBOA_SEARCH_RETRY_MAX_MS` — exponential-backoff ceiling (default `60000`,
  capped at one hour and never lower than the configured base delay)
- `JERBOA_SEARCH_MAX_CONNECTIONS` — global concurrent HTTP connections
  (default `128`, capped at `4096`)
- `JERBOA_SEARCH_MAX_CONNECTIONS_PER_IP` — concurrent connections per client
  IP (default `16`, capped by the global limit)
- `JERBOA_SEARCH_ACCEPTS_PER_SECOND_PER_IP` — accepted connections per second
  per client IP (default `32`, capped at `10000`)
- `JERBOA_SEARCH_IDLE_TIMEOUT_MS` — idle connection timeout (default `5000`)
- `JERBOA_SEARCH_HEADER_TIMEOUT_MS` — request-header deadline (default `15000`)
- `JERBOA_SEARCH_BODY_TIMEOUT_MS` — request-body deadline (default `30000`);
  all three timeout settings are capped at five minutes
- `JERBSEARCH_HTML_PARSER_WASM` — shared sandbox parser artifact

## Scope

The service implements document crawling, durable frontier recovery and retry
deadlines, link graph capture, document replacement, BM25 search, HTML results,
JSON APIs, crawl controls, and health/status endpoints. It deliberately excludes YaCy peer
protocols, DHT/RWI exchange, Solr compatibility, legacy templates, and YaCy
data migration.

For a database at `data/search.db`, durable page bodies live below
`data/search.db.content/` and the derived index cache is
`data/search.db.jerboa-index` (manifest) plus
`data/search.db.jerboa-index.segments/` (immutable segments). A validated
`data/search.db.jerboa-index.qix` binary sidecar serves mmap-hybrid queries.
Back up the
database and content directory together while the service is stopped. The
manifest, segments, and binary sidecar are optional derived data: deleting them
only makes the next startup rebuild the index.

Operational endpoints are:

- `GET /api/search?q=...&limit=20&offset=0` — deterministic BM25 results.
  `limit` is capped at `100` and `offset` at `1000`. Responses include the
  effective `limit`, `offset`, returned `count`, `has_more`, and bounded
  previous/next offsets. The HTML `/search` route accepts the same parameters
  and renders encoded Previous/Next navigation. Ranked metadata hydration uses
  prepared primary-key point lookups, so every supported offset avoids
  compound-select and bind-variable ceilings.
- `GET /api/status` — crawler, generation, checkpoint generation, segment
  count, changed-document overlay size/threshold, query engine, full-core
  residency, compaction activity/failure, source, and dirty status. Documents
  are exact; `terms_exact` distinguishes an exact term count from the safe
  upper bound used while base documents are replaced or tombstoned. Its
  `http` object reports authentication state, effective connection/rate/timeout
  settings, and application input ceilings without exposing the bearer token,
  seed URL, local filesystem paths, or internal exception text.
  The crawl object includes the durable `crawl-id`, its lifecycle state, and
  frontier counts and `max-frontier` ceiling scoped to that run.
  `not-modified-this-run` counts accepted
  conditional `304` responses without exposing page validators, while
  `sitemaps-this-run` counts successfully processed sitemap documents.
- `GET /healthz` — process liveness only; it does not touch SQLite or the index.
- `GET /readyz` — readiness for search traffic. It opens/validates the current
  index view and returns `503` when the store is unavailable or background
  compaction has failed. Successful responses include generation and query
  engine; failures do not expose exception text or filesystem paths.
- `GET /metrics` — Prometheus text exposition with fixed, label-free gauges for
  readiness, index documents/terms/generation/segments/overlay/dirty/compaction,
  and crawler running/workers/retry-policy/frontier/conditional-fetch,
  robots-cache occupancy, and host-deadline cleanup state. It returns `503`
  with `jerboa_search_up 0` when metric refresh cannot read the store/index.

Cross-origin discovery does not relax fetch security. When
`JERBOA_SEARCH_SAME_ORIGIN_ONLY=false`, every discovered URL still passes the
public-HTTPS, DNS/IP, safe-port, redirect, robots, and per-host politeness
checks. The page budget remains global across all discovered origins.
With same-origin policy enabled, the final URL after redirects must also remain
on the seed origin; a cross-origin final response is rejected before storage.

Robots handling follows RFC 9309 group selection and longest-match behavior.
Exact, case-insensitive product-token groups are merged; wildcard rules are
used only when no exact group exists, and blank lines do not accidentally end a
group. A missing `robots.txt` reported by a permanent `4xx` permits crawling.
Server/network failures, throttling responses, and unresolved redirects do not
fall through to allow-all: the page remains unfetched and enters the bounded
durable retry policy. Robots wildcard matching is linear in the rule and URL
length, avoiding recursive backtracking on untrusted policy text. Matching
normalizes percent-encoded ASCII unreserved bytes, uppercases retained reserved
encodings, and percent-encodes raw UTF-8. Encoded literal `*` and `$` bytes
remain data and cannot become rule metacharacters.

Global `Sitemap:` directives in a successful `robots.txt` are discovered once
per origin and entered as durable frontier work. Oversized `Sitemap:` directive
lines share the same 8,192-character robots line ceiling and are ignored. Both
URL-set and sitemap-index documents are supported. Fetches are capped at 4 MiB,
each document yields at most 50,000 unique locations, indexes nest at most
three levels, and redirects and emitted locations must remain on the sitemap's
origin. DTDs and nested markup inside ordinary `<loc>` text are rejected; the
narrow parser resolves only the five predefined XML entities and valid numeric
character references.
Sitemap attempts share the crawl's hard request-attempt budget, durable retry
policy, politeness schedule, and run isolation. `/metrics` publishes
`jerboa_search_crawl_sitemaps`.

Robots responses are read under a 524,288-byte transport ceiling. Their
per-crawl in-memory cache uses a fixed 128-entry FIFO ring, bounding worst-case
retained policy state even when cross-origin crawling encounters an unbounded
stream of hosts. Policies are parsed and normalized once per origin and cached
as compiled matchers instead of reparsing raw text for every page. Parsing
retains at most 10,000 rules, 256 agents in one group, and 8,192 characters per
directive line. Cache occupancy, capacity, response bytes, and parser ceilings
are reported as `robots-cache-entries`, `robots-cache-capacity`,
`robots-body-bytes`, `robots-rule-limit`, `robots-agent-group-limit`, and
`robots-line-chars` in crawl status. Page and sitemap fetch ceilings are
likewise reported as `page-body-bytes`, `sitemap-body-bytes`,
`sitemap-location-limit`, and `sitemap-depth-limit`.

Discovery growth is independently bounded. Each crawl run persists a frontier
ceiling equal to 10× `max-pages`, with a 1,000-row floor for nonzero crawls and
a 1,000,000-row hard cap. Every transactional HTML-link, conditional-replay,
and sitemap enqueue shares the remaining capacity; duplicate URLs consume no
additional slot. This prevents a small request budget from generating an
unbounded SQLite queue. The effective ceiling is exposed as `max-frontier` and
`jerboa_search_crawl_max_frontier`.
Discovery attempts rejected after that ceiling is full are persisted per crawl
run, exposed as `discovery-dropped` in status, and published as
`jerboa_search_crawl_discovery_dropped`. The counter measures attempts rather
than unique URLs; rediscoveries already assigned to the current run are free.

Workers acquire a non-blocking per-host lease before processing a claimed row.
If another worker owns that host, the row is durably deferred without consuming
a fetch attempt and the worker can claim a ready URL from another origin. This
prevents a same-host queue prefix from occupying every worker while preserving
the configured request delay and one-at-a-time robots/page processing per host.
Per-origin delay deadlines are indexed by a min-heap and removed from the host
map as soon as they expire. Stale heap entries from a superseded deadline are
discarded without deleting its newer value, so cross-origin crawls retain only
the active politeness window instead of every host ever observed.
`host-deadlines` and `host-expiry-events` expose both live structures in crawl
status; `jerboa_search_crawl_host_deadlines` and
`jerboa_search_crawl_host_expiry_events` publish them as fixed-name gauges.
`jerboa_search_crawl_robots_cache_entries` and
`jerboa_search_crawl_robots_cache_capacity` likewise expose bounded compiled
robots-policy cache occupancy and its ceiling.

Successful page fetches have a second, document-level admission policy.
Case-insensitive HTML `robots` and `JerboaSearch` meta directives are combined
with generic and `JerboaSearch`-scoped `X-Robots-Tag` response directives; the
most restrictive `noindex`/`nofollow` result wins. `none` means both. A
`noindex` page retains its real HTTP status, metadata, body, and outgoing graph
in authoritative storage, but is removed from the live BM25 view and excluded
from every rebuild. `nofollow` retains outgoing links in the graph but adds none
of them to the crawl frontier. An individual `rel=nofollow` link is likewise
kept in the graph but excluded from discovery. This separates durable fetch
history and graph analysis from search-result and crawl admission.

Relative HTML links use the first `<base href>` in tree order, as defined by
the HTML document-base algorithm. An invalid or non-web first base falls back
to the response URL and later base elements remain ignored. URL
canonicalization removes dot segments without collapsing repeated or trailing
slashes, query-only references retain the current document path, and any
explicit non-HTTP(S) URI scheme is rejected before frontier admission.
Canonical identity also decodes percent-encoded ASCII unreserved bytes and
uppercases retained percent triplets. Encoded reserved delimiters stay encoded,
malformed triplets are rejected, and dot segments are removed after safe
unreserved decoding. Raw whitespace, including leading or trailing whitespace,
control characters, and backslashes are rejected before any trimming or
resolution. Canonical URL identity is capped at 4,096 characters at the shared
URL-module boundary, including relative-resolution output. The same
ceiling therefore applies to seeds, redirects, HTML links, canonical hints,
robots sitemap directives, and sitemap locations before hashing or durable
storage. Callers should provide percent-encoded URL data.

The first `link[rel~=canonical][href]` is resolved against the effective HTML
base. A valid non-self target is accepted only on the response origin, retained
in the outgoing graph, and admitted through the ordinary bounded frontier.
When follow policy and depth allow that admission, the variant page remains in
durable fetch history but is excluded from live and rebuilt search results.
Malformed, cross-origin, self, `nofollow`, and depth-boundary hints do not
suppress the fetched page.

Transport conditions and HTTP `408`, `425`, `429`, and `5xx` responses are
retried with bounded exponential backoff. Retry deadlines and attempt counts are
stored in SQLite, so a restart neither forgets nor resets them. Other HTTP
responses and failures in parsing, persistence, or indexing are terminal. Every
request attempt, including one that is rescheduled, consumes the crawl's hard
page-attempt budget. `/api/status` exposes the effective retry policy and the
number of queued URLs currently waiting for a deadline; `/metrics` publishes
the same values as fixed, label-free gauges.
Persisted retry/failure diagnostics are truncated at 4,096 characters at the
store boundary, preventing an unusually verbose network or parser condition
from amplifying durable frontier storage. Status exposes
`frontier_error_chars`.
Valid `Retry-After` delta-seconds and each of the three RFC 9110 HTTP-date wire
formats are honored for page and sitemap responses. The server value can extend
but never shorten exponential backoff, and the configured retry ceiling still
caps it. Malformed or over-128-character values are ignored.

Only exact `text/html`, `application/xhtml+xml`, and `text/plain` success
responses enter the text index; content-type parameters are allowed, but media
type substrings are not guessed. Unsupported successful media is completed as
skipped work rather than falsely incrementing the indexed count. Robots
exclusions use the same skipped counter; `jerboa_search_crawl_skipped` exposes
it to Prometheus.

Index amplification is separately bounded: each document contributes at most
50,000 tokens, consuming title tokens before body tokens, and alphanumeric runs
longer than 128 characters are ignored. The tokenizer scans string indices
without first allocating a character list. Live ingestion, persisted
`token_count`, rebuilds, segments, and mmap sidecars all use this same bounded
logical document. `/api/status` exposes `document_tokens` and `term_chars`.
Queries likewise use at most the first 32 distinct terms found within a bounded
256-token prefix. Core, mmap, hybrid overlay ranking, result hydration, and
snippet selection share this parser, preventing query text from multiplying
term-stat lookups or WAND cursors without bound. Status exposes `query_terms`
and `query_token_scan`. Direct paged search windows are bounded to the first
100,000 ranked candidates, while the HTTP API clamps offsets lower for
interactive use; status exposes `search_page_window`.
Preview localization scans at most the first 65,536 body characters and first
eight query terms. It reuses the parsed query term list and avoids lowercasing
the body window when that window is already lowercase, while retaining
case-insensitive matching for mixed-case content. Ranking still uses the full
bounded token view; when a match lies outside the preview window, the snippet
safely falls back to the document start. Status exposes `snippet_scan_chars`
and `snippet_terms`.

HTML extraction has independent post-sandbox ceilings. A representation may
contain at most 100,000 parsed nodes with nesting no deeper than 256, retain at
most 2,097,152 readable characters, and contribute at most 10,000 unique links
including an accepted canonical target. The canonical slot is reserved before
ordinary-link traversal, and traversal stops as soon as its graph budget is
full. Whitespace normalization scans by string index instead of materializing a
second character list. `/api/status` exposes `document_nodes`,
`document_depth`, `document_text_chars`, and `document_links`.
Titles are collected directly under a 1,024-character ceiling rather than
materializing an unbounded title subtree. HTML/header robots values and link
`rel` values use an index-based 8,192-character directive scanner; oversized
directives fail closed as `noindex,nofollow` and cannot produce a canonical
hint. Status exposes `document_title_chars` and `directive_chars`.

Each accepted seed belongs to a durable numeric crawl run. Starting the same
canonical seed resumes its latest paused run, retaining queued URLs, attempts,
and retry deadlines. Starting a different seed creates an isolated run, so it
cannot claim frontier rows left by another seed. An explicit stop, process
restart, or exhausted page-attempt budget leaves unfinished work paused; a
fully drained frontier becomes completed, and starting that seed again creates
a new run. Newly rediscovered URLs may move from an older run into the current
run, but duplicate links within one run remain no-ops.
Stopping interrupts local politeness and frontier-poll waits within one 25 ms
slice. A row interrupted before its request is durably requeued and its claim
attempt is reversed. Synchronous network activity is allowed to finish under
its existing 10/20-second request deadline.

Successful responses persist bounded `ETag` and `Last-Modified` validators.
Later crawl runs send them as `If-None-Match` and `If-Modified-Since`; validator
values containing HTTP control characters or more than 1,024 characters are
never reused as request headers. An exact-URL `304 Not Modified` refreshes
fetch time and any returned validators without replacing content, advancing the
search generation, or perturbing live rankings. The same transaction replays
only the page's
previously approved crawl edges into the new run, so unchanged seeds do not
truncate traversal and `nofollow` links remain graph-only. Legacy link rows
default to non-crawlable until a fresh representation classifies them.

An observed `404 Not Found` or `410 Gone` retires the final URL immediately.
The authoritative row keeps its prior title, body reference, and link history,
but records the new status, clears cache validators, becomes non-indexable, and
advances the search generation. Live postings are tombstoned and rebuilds also
exclude the row, so stale content cannot reappear after restart or compaction.
A later successful response replaces the preserved state and restores the URL
normally. Retryable responses such as `408`, `429`, and `5xx` never retire the
last good representation. `/api/status` exposes `retired-this-run`, and
`/metrics` publishes `jerboa_search_crawl_retired`.
- `POST /api/index/checkpoint` — append changes as an immutable segment and
  atomically publish its manifest; reaching eight segments or 1,000 changed
  document IDs since the mapped base schedules background compaction without
  delaying the checkpoint response. Set
  `JERBOA_SEARCH_MAX_OVERLAY_DOCUMENTS` to tune the latter threshold.
  Responses include `ok`, `generation`, and `checkpoint_generation`, not local
  manifest paths.
- `POST /api/index/compact` — replace the current chain with one full immutable
  segment and delete only obsolete files named by the old validated manifest.
  Responses include `ok`, `generation`, and `checkpoint_generation`, not local
  manifest paths.
- `POST /api/content/gc` — remove unreferenced managed content blobs and
  abandoned managed blob temporaries; the crawler must be stopped so newly
  fetched content cannot race collection. The stop endpoint joins all fetch
  workers before returning, so a successful stop is a quiescent GC boundary.

All state-changing routes are administrative: `POST /crawl`,
`POST /api/crawl`, `POST /api/crawl/stop`, `POST /api/index/checkpoint`,
`POST /api/index/compact`, and `POST /api/content/gc`. When
`JERBOA_SEARCH_ADMIN_TOKEN` is set, clients must send
`Authorization: Bearer <token>` on each of them. Missing or incorrect
credentials receive `401`, a `WWW-Authenticate` challenge, and a JSON error;
the HTML home page hides its crawl form because browser forms cannot attach the
header. Search, status, and health endpoints remain public. Token comparison
uses Jerboa's timing-safe crypto primitive, and the token is retained only in
the application closure—not exposed in settings or responses.

The application rejects bodies above 65,536 bytes with `413` before form
decoding, search queries above 1,024 characters with `414`, and crawl seeds
above 4,096 characters with `400`. Discovered URLs enforce that same ceiling
inside canonicalization and are simply ineligible for admission. The
underlying Jerboa transport independently bounds request headers at 16 KiB and
bodies at 4 MiB, so malformed traffic is bounded before application policy
runs.

Without a token, the administrative routes remain available for the default
local-only workflow, but the server accepts only `127.0.0.1`, `::1`, or
`localhost` as its bind address. Set the token even on a loopback bind when a
reverse proxy makes the service reachable by other machines. Supply the token
through the environment or a service manager's secret facility; do not place it
in URLs, request bodies, or command-line arguments. Bearer credentials are
replayable, so any non-local client connection must use TLS. The current
launcher serves plain HTTP and is intended to sit behind a TLS-terminating
reverse proxy on a private or loopback connection.

See [ARCHITECTURE.md](ARCHITECTURE.md) for scaling boundaries and package
extraction points.

recent commits