docs: document the forge guardrails feature
ober
a24ddc195ce844455563042ded20cc74003cb8f9
--- a/README.md +++ b/README.md @@ -1 +1,24 @@ # jerboa-code + +A portable AI coding agent written in Jerboa Scheme (compiled by jerbuild to +Chez). Source lives under `src/jcode/`; entry point is `main.ss`. + +```bash +make build # compile src/ → lib/ +make test # run test/run.ss +make binary # produce the standalone ./jcode +make run # interactive agent REPL +``` + +## Forge guardrails + +jcode embeds a native port of [forge](https://github.com/azambelli/forge) — a +reliability layer that makes small / self-hosted models call tools dependably. +The guardrails (rescue, validation + retry budget, respond-forcing, per-model +sampling, compaction) are **always on for every provider**. The `/forge` +command is the control / status / ablation surface, and `jcode proxy` exposes +the same guardrails over an OpenAI-compatible HTTP endpoint. + +See **[docs/FORGE.md](docs/FORGE.md)** for the command reference, the workflow +engine, the proxy, and the eval/ablation harness. Design notes and forge-source +citations are in [docs/FORGE_PORT_PLAN.md](docs/FORGE_PORT_PLAN.md). new file mode 100644 --- /dev/null +++ b/docs/FORGE.md @@ -0,0 +1,181 @@ +# Forge — reliability guardrails for local LLM tool-calling + +jcode ships a native port of [forge](https://github.com/azambelli/forge), a +reliability layer that makes small / self-hosted models call tools dependably. +It is implemented in 100% Jerboa Scheme (`src/jcode/{guardrails,core,proxy,eval}`) +and is **always on for every provider** — local *and* cloud. The `/forge` +command is therefore a **control / status / ablation** surface, not an on/off +gate, and `jcode proxy` exposes the same guardrails to any OpenAI client. + +For the design rationale and the forge-source citations behind each constant, +see [`FORGE_PORT_PLAN.md`](FORGE_PORT_PLAN.md). + +--- + +## Command reference + +### `/forge` (interactive REPL) + +| Command | Effect | +|---|---| +| `/forge` · `/forge status` | Show guardrail state: rescue, retry budget, respond-forcing, sampling policy, compaction strategy, VRAM tier. | +| `/forge on` · `/forge enforce [on]` | Respond-forcing **ON** — bare text is retried as a tool call (`respond` injected). | +| `/forge off` · `/forge enforce off` | Respond-forcing **OFF** — bare text is a normal final answer. | +| `/forge sampling off` | Apply no per-model sampling params. | +| `/forge sampling on` | Use the model card's sampling profile if known, else backend defaults. | +| `/forge sampling strict` | Use the card profile; error on an unknown model. | +| `/forge workflow` | Describe + self-test the workflow engine with a scripted responder. | +| `/forge proxy` | Describe + self-test the OpenAI-compatible proxy pipeline. | +| `/forge ablation` · `/forge eval` | Describe + self-test the deterministic eval/ablation harness. | + +The `workflow` / `proxy` / `ablation` self-tests run real code against scripted +inputs (no live model), so they double as a smoke test of the whole stack. + +### `jcode proxy` (subcommand) + +```bash +jcode proxy [--port N] [--bind ADDR] # default 127.0.0.1:8080 +``` + +Serves the configured provider behind the guardrails over an OpenAI-compatible +HTTP endpoint. See [The proxy](#the-openai-compatible-proxy) below. + +--- + +## The guardrails + +All of these wrap every provider call automatically. + +- **Rescue** (`guardrails/rescue.ss`) — recover tool calls the model emitted as + prose instead of structured calls: fenced/embedded JSON brace-scan, rehearsal + `name[ARGS]{…}`, Qwen `<function=name>`, Hermes/Qwen3 `<tool_call>`, Mistral + `[TOOL_CALLS]name{…}`, DeepSeek, with `<think>` stripping. +- **Validator** (`guardrails/validator.ss`) — when the model names a tool that + doesn't exist, or returns bare text where a tool was required, inject a nudge + and retry instead of failing. +- **Error budget** (`guardrails/error-tracker.ss`) — after 3 consecutive bad + responses, stop rather than loop forever; tool-execution errors have their own + budget. Nudges are surfaced as **tool results** (`[StepEnforcementError]`, + `[PrerequisiteError]`, `[ToolError]`), because models are pretrained on + "tool failed → try something else". +- **Respond-forcing** (`guardrails/respond.ss`) — optionally inject a synthetic + `respond` tool so a model that wants to answer in prose is kept in + tool-calling mode where the guardrails apply; the reply is unwrapped back to a + normal assistant message. Toggle with `/forge on|off`. +- **Per-model sampling** (`provider/sampling.ss`) — a model card maps each model + to its recommended temperature / top_p / etc. Policy is `off | on | strict`. +- **Compaction** (`core/compaction-strategy.ss`) — pluggable context compaction + (`TieredCompact`, `SlidingWindow`, `NoCompact`, plus jcode's original), always + preserving the system prompt, first user message, and recent steps. + +--- + +## The workflow surface + +`/forge workflow` exercises an optional structured agent loop ported from +forge's `Workflow` + `WorkflowRunner` (`core/workflow.ss`, `core/workflow-runner.ss`). +A workflow constrains the loop with: + +- **required steps** — tools that must run before a terminal tool may fire; +- a **terminal tool** — calling it ends the run and returns its value; +- **prerequisites** — per-tool dependencies (name-only or arg-matched). + +Define one in Jerboa and drive it with a responder (the inference seam): + +```scheme +(def w + (make-workflow + "research" "Search for context, then answer." + (list + (make-tool-def (make-tool-spec "search" "Search." '(("type" . "object"))) + (lambda (args) "results") '()) + (make-tool-def (make-tool-spec "answer" "Final answer." '(("type" . "object"))) + (lambda (args) "done") '("search"))) ; answer requires search + '("search") ; required steps + "answer" ; terminal tool + "You are a research agent.")) + +(run-workflow w "look it up" responder '((max-iterations . 6))) +``` + +The runner enforces premature-terminal nudges (step 3b), prerequisites (3b.2), +batch tool execution (3c–3e), the error budget (3d), and a max-iterations cap +(step 4), raising the matching forge condition when a budget is exhausted. +`core/slot-worker.ss` serializes runs on a single inference slot with +priority + preemption (forge's single-GPU `SlotWorker`). + +--- + +## The OpenAI-compatible proxy + +`jcode proxy` (and the pure, testable `proxy-dispatch` core in +`proxy/server.ss`) speaks the OpenAI chat-completions wire format and runs every +request through the guardrailed handler. + +| Method | Path | Returns | +|---|---|---| +| `GET` | `/health` | `{"status":"ok"}` | +| `GET` | `/v1/models` | OpenAI model list (id `forge`) | +| `POST` | `/v1/chat/completions` | `chat.completion`, or an SSE stream when `stream:true` | + +```bash +jcode proxy --port 8080 & +curl -s localhost:8080/v1/chat/completions \ + -H 'content-type: application/json' \ + -d '{"messages":[{"role":"user","content":"hi"}]}' +``` + +When the client sends `tools`, a `respond` tool is injected so the model stays +in tool-calling mode (where the guardrails apply), then stripped from the reply. +Conversion lives in `proxy/convert.ss`; the guardrailed request handling in +`proxy/handler.ss`. + +--- + +## The eval & ablation harness + +`/forge ablation` runs a **deterministic** harness (`eval/{scenario,ablation,runner}.ss`): +the model is replaced by a scripted responder, so a run depends only on the +script plus the guardrail configuration. This isolates each guardrail's +contribution to outcome quality without a live model. + +- **Scenario** (`eval/scenario.ss`) — a workflow + user message + scripted + responder + a validator; `check-substrings` is forge's case-insensitive, + comma-stripping answer matcher. +- **Ablation presets** (`eval/ablation.ss`): `reforged` (all on), `no_rescue`, + `no_nudge`, `no_steps`, `no_recovery`, `no_compact`, `bare` (all off). +- **Runner** (`eval/runner.ss`) — `run-eval-scenario`, `run-ablation`, + `ablation-pass-rates`, and `analyze-messages` (tool-call / nudge / error / + reasoning tallies). Accuracy = *completeness* ∧ *validate(terminal-args)* ∧ + *validate-state*. + +The built-in demo scripts a model that tries to answer before searching. With +step enforcement **on** it is nudged to search first (accuracy `#t`); with +`no_steps` it answers immediately but ungrounded (accuracy `#f`) — the harness +attributing the quality gain directly to the step-enforcement guardrail: + +``` +reforged complete=#t accuracy=#t iters=3 +no_steps complete=#t accuracy=#f iters=1 +``` + +Only the knobs the runner observes have effect in the deterministic harness +(step-enforcement, max-retries, max-tool-errors); `rescue` / `compaction` live +in the injected responder seam and are recorded for fidelity but do not by +themselves change a scripted run. + +--- + +## Module map + +``` +src/jcode/ + guardrails/ rescue.ss validator.ss error-tracker.ss respond.ss + nudge.ss message-type.ss step-enforcer.ss guardrails.ss + core/ workflow.ss workflow-runner.ss steps.ss slot-worker.ss + compaction-strategy.ss errors.ss (+ message.ss, agent.ss, …) + proxy/ convert.ss handler.ss server.ss + eval/ scenario.ss ablation.ss runner.ss +``` + +All deterministic logic is covered by `test/run.ss` (`make test`). --- a/docs/FORGE_PORT_PLAN.md +++ b/docs/FORGE_PORT_PLAN.md @@ -1,5 +1,10 @@ # Forge → jerboa-code Port Plan +> **Status: shipped.** All 8 phases are implemented, tested, and merged on the +> `forge` branch. This document is the original design plan, kept for rationale +> and forge-source provenance. For how to *use* the result, see +> [FORGE.md](FORGE.md). + **Goal.** Reproduce 100% of [forge](../../forge)'s functionality — a reliability layer for self-hosted LLM tool-calling — natively in jerboa-code (Jerboa Scheme), surfaced as a builtin `/forge` command plus always-on guardrail middleware on every provider call. **Decisions locked with the user:**