forge port phase 0: guardrail scaffolding (nudge + error-tracker)
ober
7309036ead050d1ce598451f6ae2bcdcb69bb2b0
--- a/build-binary.ss +++ b/build-binary.ss @@ -129,6 +129,8 @@ "lib/jcode/core/debug-repl" "lib/jcode/core/skill" "lib/jcode/core/builtin-skills" + "lib/jcode/guardrails/nudge" + "lib/jcode/guardrails/error-tracker" "lib/jcode/provider/provider" "lib/jcode/tool/registry" "lib/jcode/tool/file" @@ -320,6 +322,7 @@ "std/result" "std/datetime" "std/csv" + "std/contract/condition" "std/ergo" "std/misc/string" "std/misc/list" new file mode 100644 --- /dev/null +++ b/docs/FORGE_PORT_PLAN.md @@ -0,0 +1,292 @@ +# Forge → jerboa-code Port Plan + +**Goal.** Reproduce 100% of [forge](../../forge)'s functionality — a reliability layer for self-hosted LLM tool-calling — natively in jerboa-code (Jerboa Scheme), surfaced as a builtin `/forge` command plus always-on guardrail middleware on every provider call. + +**Decisions locked with the user:** +- **Scope: true 100%** — guardrail core + OpenAI-compatible proxy + eval harness. The forge eval *dashboard* is React/Vite/TypeScript and is explicitly **not** ported; we keep the same data (JSONL) and the ASCII/list/markdown report views, which are pure logic. +- **Activation: always-on for all providers** — guardrails wrap every provider (local *and* cloud). The `/forge` command is therefore a **control / status / ablation** surface, not an on/off gate. + +Forge upstream is Python (~6,700 LOC, MIT, Antoine Zambelli). This plan ports behavior, not code; all exact constants/strings below are cited to forge source so the Jerboa implementation can match 1:1. + +--- + +## 1. What jcode already has vs. what forge adds + +This is the load-bearing table. **We extend/unify the left column; we build the right column.** Do not duplicate existing jcode machinery. + +| Forge capability | Already in jcode? | Action | +|---|---|---| +| Rescue-parse tool calls from text | **Partial** — `agent.ss` parses paren-style `tool(k="v")`, Hermes/Qwen3 `<tool_call><function name=…>`, DeepSeek `<｜DSML｜invoke…>`; `provider.ss:641-748` recovers `<tool_call>`/fenced-JSON | **Unify + extend** to forge's full set: fenced/embedded JSON brace-scan, rehearsal `name[ARGS]{…}`, Qwen `<function=name>`, Mistral `[TOOL_CALLS]name{…}`, think-tag stripping | +| Per-model sampling defaults | **Partial** — `apply-mlx-sampling!` hardcodes MLX temp/top_p/rep | **Replace** with full per-model map (3-key identity) + opt-in policy | +| HTTP transient retry (408/429/5xx) | **Yes** — `provider.ss:56-81` `api-call-with-retry` | Keep as-is (orthogonal layer) | +| Context compaction | **Yes but flat** — `compaction.ss`: keep system + last N, stub bulky tool/assistant content over `prune_bytes`, trigger at % of ctx | **Add strategy abstraction** + `TieredCompact` (MessageType-priority 3-phase) + `SlidingWindow` + `NoCompact`; keep current as a 4th strategy | +| Pre/post tool hooks, abort | **Yes** — `hooks.ss`, `registry.ss:99-104` | Reuse as a guardrail hook point | +| Parallel tool exec | **Yes** — `agent.ss execute-tool-calls` (green threads, param re-parameterize) | Reuse | +| Response validation (unknown-tool / bare-text → nudge) | **No** | **Build** (`ResponseValidator`) | +| Retry-with-nudge + error budget | **No** (only HTTP retry) | **Build** (`ErrorTracker`, nudge injection in loop) | +| Synthetic `respond` tool | **No** (jcode treats "no tool calls" as done) | **Build** + resolve coding-loop semantics (§5.A) | +| Step enforcement / `required_steps` / `terminal_tool` / prerequisites | **No** (open-ended loop, no workflow) | **Build** as an *optional* Workflow surface (§5.B) | +| OpenAI-compatible proxy server | **No** (`serve.ss` is JSONL-over-TCP, not HTTP) | **Build** HTTP/1.1 + SSE from scratch on `:std/net/tcp` | +| Eval harness + ablation | **No** | **Build** (scenarios, runner, batch, ablation, metrics, report, significance) | +| Eval React dashboard | n/a | **Out of scope** (keep JSONL + ASCII/markdown) | + +--- + +## 2. Target module layout + +Mirror forge's package structure under `src/jcode/`. New dirs in **bold**. + +``` +src/jcode/ + guardrails/ ← NEW (forge/guardrails + forge/context + forge/tools/respond) + nudge.ss Nudge struct + nudge-text templates (forge/prompts/nudges.py) + rescue.ss unified rescue parser (forge/prompts/templates.py + jcode's existing parsers) + validator.ss ResponseValidator (forge/guardrails/response_validator.py) + step-enforcer.ss StepEnforcer + StepTracker + prerequisites (forge/guardrails/step_enforcer.py, core/steps.py) + error-tracker.ss ErrorTracker (forge/guardrails/error_tracker.py) + guardrails.ss Guardrails facade: check()/record() (forge/guardrails/guardrails.py) + respond.ss synthetic respond tool (forge/tools/respond.py) + compaction-strategy.ss CompactStrategy / NoCompact / SlidingWindow / TieredCompact (forge/context/strategies.py) + hardware.ss detect-hardware + budget tiers (forge/context/hardware.py + server.py VRAM tiers) + provider/ + provider.ss EXTEND: wire validator+rescue+error-budget+respond into chat path + sampling.ss ← NEW MODEL_SAMPLING_DEFAULTS map + get/apply policy (forge/clients/sampling_defaults.py) + core/ + agent.ss EXTEND: guardrail middleware in agent-loop / agent-loop-stream + workflow.ss ← NEW Workflow/ToolSpec/ToolDef/ToolCall structs + validation (forge/core/workflow.py) + workflow-runner.ss ← NEW WorkflowRunner loop (forge/core/runner.py) + SlotWorker (forge/core/slot_worker.py) + guardrails-config.ss ← NEW /forge toggles, ablation state, persisted in jcode.json + proxy/ ← NEW (forge/proxy/* + forge/server.py) + http.ss HTTP/1.1 parse + SSE chunked framing on :std/net/tcp + convert.ss OpenAI ↔ jcode message conversion (forge/proxy/convert.py) + handler.ss per-request guardrail bridge (forge/proxy/handler.py) + server.ss routes, serialize worker, disconnect-cancel (forge/proxy/server.py) + backend.ss ServerManager + BudgetMode (forge/server.py) + eval/ ← NEW (forge/tests/eval/*) — in src so /forge eval compiles into the binary + scenario.ss EvalScenario contract + _check/_validate helpers (_base.py) + scenarios/*.ss the 30 scenarios + runner.ss batch.ss ablation.ss metrics.ss report.ss significance.ss + ui/cli.ss EXTEND: /forge command dispatch in handle-command +``` + +**Layout decisions to confirm during impl:** (a) eval under `src/` (compiled, invokable via `/forge eval`) vs `test/` (forge's choice) — recommend `src/jcode/eval/`; (b) `respond` registered through existing `register-internal-tool!` so it never leaks into normal tool listings. + +--- + +## 3. Faithful behavior spec (the 1:1 contract) + +Everything an implementer needs to match forge exactly. **Cite = forge path:line.** + +### 3.1 Magic constants (quote-exact) + +| Constant | Value | Cite | +|---|---|---| +| WorkflowRunner.max_iterations | **10** (eval scenarios override to 15) | runner.py:36 / eval `_base.py` | +| max_retries_per_step | **3** (eval: 5) | runner.py:37 | +| max_tool_errors | **2** | runner.py:38 / error_tracker.py:19 | +| max_premature_attempts | **3** | step_enforcer.py:49 | +| max_prereq_violations | **2** | step_enforcer.py:49 | +| step nudge tier | `min(attempts, 3)` (1=polite,2=direct,3=aggressive) | step_enforcer.py:77 | +| TieredCompact TRUNCATE_CHARS | **200** | strategies.py:108 | +| TieredCompact keep_recent | **2** | strategies.py:110 | +| compact_threshold | **0.75** of budget | strategies.py:113 | +| token estimate | `chars // 4` | strategies.py:11 | +| context warning tiers | ≥0.80 / ≥0.65 / else | manager.py:33,40 | +| attempt_limit | `max_retries + 1`, clamped to `max_iterations − iteration` | inference.py:163 | +| call_id format | `call_{counter:09d}` (runner) / `call_<8hex>` (proxy) | inference.py:101 / convert.py | +| VRAM budget tiers | ≥48GB→**262144**, ≥24GB→**32768**, else→**4096** | server.py:383 | +| no-hardware fallback | **4096** tokens | hardware.py:80 | +| proxy default host/port | **127.0.0.1 / 8081**; backend **8080** | __main__.py | +| proxy max body | **16 MiB** → 413 | server.py:23 | + +**Exhaustion is strict `>`** everywhere: `max_retries=3` permits 3 failures, trips on the **4th**. Same for premature/prereq/tool-errors. (error_tracker.py:56) + +### 3.2 Nudge text (verbatim — nudges.py) + +- **retry** (bare text): `"Your previous response was not a valid tool call. You must respond with a tool call, not free text. Please try again with a valid tool call."` +- **unknown_tool**: `"Tool '{name}' does not exist. Available tools: {list}. Call one of them."` +- **step tier 1**: `"You cannot call {terminal} yet. You must first complete these required steps: {steps}. Call one of them now."` +- **step tier 2**: `"You must call one of these tools now: {steps}. Pick one."` +- **step tier 3**: `"STOP. You MUST call one of: {steps}. Do NOT call {terminal}. Your next response MUST be a tool call to one of: {steps}."` +- **prerequisite**: `"You cannot call {tool} yet. You must first call: {prereqs}. Call the prerequisite tool now."` + +Nudge struct: `role` (`"user"`|`"tool"`), `content`, `kind` (`retry`/`unknown_tool`/`step`/`prerequisite`), `tier` (0 default). Wire-shape rule: **retry/unknown nudges go on the `user` channel** (model needs a positive instruction); **step/prereq nudges go on the `tool`-error channel** with `[StepEnforcementError]`/`[PrereqError]`/`[UnknownTool]` prefixes (OpenAI-tool-trained models pattern-match "your call failed, retry" better there). (ADR rationale in ARCHITECTURE.md "Guardrails" table; inference.py:245-292) + +### 3.3 Rescue parsing (templates.py) — try strategies **in order**, stop at first hit + +0. **Strip think tags** first: `[THINK]…[/THINK]` and `<think>…</think>` (DOTALL). If empty after, return none. +1. **Embedded/fenced JSON**: strip ```` ```json ```` fences, brace-scan for balanced `{…}`, parse each. Accept forge-style `{"tool","args"}` **or** OpenAI-style `{"name","arguments"}`; tool name must be in the known set. (templates.py:54-114) +2. **Rehearsal**: regex `(\w+)\[ARGS\](\{.*\})` DOTALL, args = JSON. Only if (1) found nothing. (templates.py:120,266) +3. **Qwen Coder XML**: `<function=NAME>` … `<parameter=KEY>value</parameter>` … `</function>`; strip one leading + one trailing newline per value; values stay strings (coerced later). (templates.py:137-187) +4. **Mistral bracket-tag**: `[TOOL_CALLS]NAME` then brace-balanced JSON (string-aware: respects `\` escapes and `"`). (templates.py:152-240) + +jcode already has parsers for #1(partial)/#3 plus paren-style and DeepSeek-invoke that forge lacks — **keep those, add #2 and #4, align #1's dual-key acceptance.** Consolidate all into `guardrails/rescue.ss` so `agent.ss` and the proxy share one parser (forge principle #4: rescue lives at the client/abstraction boundary, not in the loop). + +### 3.4 Synthetic `respond` tool (respond.py) + +- name `"respond"`; single param `message: string` ("The message to send to the user."). +- description (verbatim): `"Respond to the user with a message. Use this when the user is chatting, asking a question, when you need to ask a clarifying question before proceeding, or when no other tool action is needed. Also use this after completing the user's request to report the result."` +- Behavior: injected when tools are present and no `respond` already exists; a pure-`respond` response is **unwrapped to a normal assistant text message** before returning; mixed batches emit only the non-respond calls. (handler.py:112,165) + +### 3.5 ErrorTracker / StepEnforcer state machine (runner.py loop order) + +Per iteration (`while iteration < max_iterations`): +1. **Cancellation** check (once, before inference) → raise `WorkflowCancelledError`. +2. **run_inference** (compact → threshold-check → fold/serialize → inject transient `user` context-warning → send → sync token count → validate; retry-with-nudge internally up to `attempt_limit`). `iteration += attempts` (**retries consume the iteration budget**). Returns `None` when exhausted → `break` → `MaxIterationsError`. +3. **TextResponse** → emit assistant text, `continue`. +4. **Premature-terminal**: if a terminal tool is in the batch and required steps unsatisfied → escalating step nudge on tool channel; raise `StepEnforcementError` past limit. +5. **Prerequisites**: whole-batch blocking — any one violating call blocks the batch; raise `PrerequisiteError` past limit. +6. **Execute batch** — 3 outcomes per call: (a) `ToolResolutionError` is **privileged** — feeds back as `[ToolResolutionError]`, does *not* count against error budget, does *not* record the step; (b) other exception → `[ToolError]`, counts; (c) success → `StepEnforcer.record(tool,args)`, emit result. +7. **Post-batch**: clean batch → reset error/premature/prereq counters; else `record_result(false)`, raise `ToolExecutionError` past limit. +8. **Terminal return**: if a terminal tool succeeded (non-exception result) → return it. + +Control-flow state (`StepTracker.completed_steps`/`executed_tools`, error counters, premature/prereq counters, `iteration`, `tool_call_counter`) lives **on the runner, never in message history** (forge principle #3) so compaction can't corrupt it. + +### 3.6 Compaction strategies (strategies.py) + +`CompactStrategy.compact(messages, budget, step_hint) -> (messages, phase)`; phase 0 = untouched (return the *same* list object so callers' identity check short-circuits). Invariant: never cut `messages[0]` (system) or `messages[1]` (first user). Eligible window = indices `[2, eligible_end)`; `eligible_end` found by counting **distinct consecutive `step_index` values** and keeping the last `keep_recent` iterations (handles variable-size parallel batches). + +**TieredCompact 3-phase priority (exact):** + +| Phase | Drop entirely (in window) | Special | +|---|---|---| +| 1 | step/prereq/retry nudges | tool_result >200 chars → truncate to `[:200]` + `"\n[Truncated — {n} chars removed]"` | +| 2 | + tool_result (whole) | reasoning & text_response **preserved** | +| 3 | + reasoning + text_response | only tool_call skeleton remains | + +Each phase rebuilds from the *original* list (not cumulative) and stops as soon as the estimate drops below the next phase's trigger. Reasoning surviving through phase 2 is the key design choice (the model's interpretation of results matters more than raw results). jcode's current `compaction.ss` ≈ a SlidingWindow+truncate hybrid → keep it as `JcodeLegacyCompact`; default switches to `TieredCompact` once `MessageType` tags exist (§6 phase 1 adds them). + +### 3.7 Sampling defaults (sampling_defaults.py) + +Port `MODEL_SAMPLING_DEFAULTS` **verbatim** (every row, all 3 identity keys per model, inline HF-card URL comments preserved). Two functions: +- `get-sampling-defaults(model)` → fresh copy or `{}` (pure lookup, no logging/raising). +- `apply-sampling-defaults(model, strict)` → 4-quadrant: strict+known→dict; strict+unknown→**raise UnsupportedModelError**; non-strict+known→one-shot INFO log, `{}`; non-strict+unknown→`{}`. + +**Always-on interaction:** because guardrails wrap *all* providers, default to **non-strict** (apply if known, else fall through to backend defaults — never break an unknown model). `/forge sampling strict` opts into strict per-session. This supersedes `apply-mlx-sampling!` (MLX values become rows in the map, or remain a provider default if no card exists). + +### 3.8 Errors (errors.py) + +Port the hierarchy: base `ForgeError` → `UnsupportedModelError`, `ToolCallError(raw_response)`, `ToolExecutionError`, `WorkflowCancelledError`, `MaxIterationsError`, `StepEnforcementError`, `PrerequisiteError`, `ContextBudgetExceeded`, `HardwareDetectionError`, `ContextDiscoveryError`, `BudgetResolutionError`, `BackendError`→`ThinkingNotSupportedError`, `StreamError`. **`ToolResolutionError` is deliberately NOT a ForgeError** (plain exception; the privileged "retry with different args" feedback). Map to Jerboa conditions; watch the `else`-in-`guard` gotcha (see §7). + +--- + +## 4. The `/forge` command (control surface) + +Because guardrails are always-on, `/forge` is stateful control — implement as a real case in `cli.ss handle-command` (lines 283-388), **not** a builtin-skill prompt string (those just feed text to `agent-run`). State persists under a `"forge"` block in `jcode.json` via `guardrails-config.ss`. + +| Subcommand | Action | +|---|---| +| `/forge` or `/forge status` | Show active guardrails, current ablation, retry/error budgets, compaction strategy, detected hardware + budget, proxy state | +| `/forge ablation <preset>` | Apply an ablation preset (reforged/no_rescue/no_nudge/no_steps/no_recovery/no_compact/bare) to the live session — directly reuses eval ablation wiring | +| `/forge rescue on\|off`, `/forge respond on\|off`, `/forge compact <tiered\|sliding\|legacy\|none>`, `/forge retries N`, `/forge budget <mode\|N>` | Per-guardrail toggles/tunes | +| `/forge sampling <off\|on\|strict>` | Sampling-map policy for this session | +| `/forge proxy start [--port …] [--backend …]` / `stop` | Start/stop the OpenAI proxy (§5.C) | +| `/forge eval [--scenario … \|--tags …] [--ablation …] [--runs N]` | Run the eval harness (§5.D), print report | +| `/forge workflow <file.ss>` | Load + run a Workflow definition through WorkflowRunner (§5.B) | + +--- + +## 5. Surfaces (and the workflow-vs-coding-loop resolution) + +### 5.A In-loop guardrail middleware (always-on) + +Wrap `agent.ss` `agent-loop` / `agent-loop-stream`. After the provider returns and tool calls are parsed (the existing 4-format parse stays, now delegating to `rescue.ss`): + +1. `validator.validate(response)` → if bare text and rescue finds a call, use it; if unknown tool or unrescuable text → nudge + `error-tracker.record-retry`; re-infer until success or `retries_exhausted` → raise `ToolCallError`. +2. Inject the nudge on the correct channel (user vs tool) and loop **without** counting an HTTP round as "done". +3. On clean batch, `error-tracker.reset`. + +**Resolving "no terminal tool" in an open coding loop:** jcode's loop ends when the model emits no tool calls (bare text = the answer). Forge forbids bare text via `respond`. Reconcile: +- With guardrails on, **register `respond` as an internal tool** and treat it as the terminal for the *chat* turn. A local model emitting bare text gets **one** rescue/nudge toward `respond`; a `respond` call is unwrapped to the normal assistant message the TUI already renders. +- `required_steps`/`prerequisites` are **empty** in the default coding loop (no fixed workflow), so `StepEnforcer` is a no-op there — exactly forge's "guardrails apply with zero required steps too." Premature-terminal logic only bites when a Workflow is loaded (§5.B). +- Cloud models (Anthropic/OpenAI) reliably choose text vs tool; per the always-on decision they still pass through, but `respond` is effectively never needed — keep it injected for uniformity, cheap. + +### 5.B WorkflowRunner surface (forge's structured workflows) + +For the forge features that have no home in an open loop, port the Workflow concept as an **opt-in** runner (`/forge workflow file.ss`): +- `workflow.ss`: `ToolSpec`/`ToolDef`(+`prerequisites`)/`ToolCall`/`Workflow` structs; `Workflow` validation (every tool key == spec name; required_steps ⊂ tools; terminal ⊂ tools; **terminal ∉ required_steps**; prereqs resolve). Tool params are **JSON-schema dicts** (Jerboa has no Pydantic) — port `from_json_schema`/`get_json_schema`. +- `workflow-runner.ss`: the full loop from §3.5 with step/prereq enforcement + `SlotWorker` (priority queue, lower int = higher priority, default 0 = FIFO, auto-preempt lower-priority running task by setting its cancel event). Streaming + `on_message` map to jcode's existing stream/tool callbacks (`make-stream-cb`/`make-tool-cb` in `serve.ss`). + +This keeps the open coding agent and the structured runner as two front-ends over one shared guardrail core (`guardrails.ss`) — mirroring forge's "middleware is the foundation; proxy and runner compose it." + +### 5.C OpenAI-compatible proxy (forge/proxy + server.py) + +`serve.ss`/`relay.ss` are JSONL-over-raw-TCP with a bespoke schema — **no HTTP, no SSE, no OpenAI shape reusable.** Build on `:std/net/tcp` from scratch, reusing only serve.ss's accept-loop / auth / arg-parse / cancellation *patterns*. + +- **Routes:** `GET /health` → `{"status":"ok"}`; `GET /v1/models` → `{"object":"list","data":[{"id":"forge","object":"model"}]}`; `POST /v1/chat/completions` (JSON or SSE); `OPTIONS *` → 204 + CORS; else 404. +- **Per-request order** (handler.py): convert OpenAI→messages + extract `ToolSpec`s → **inject `respond`** (if tools present, none already) → no-tools fast path forwards untouched (no guardrails) → run the validate/rescue/retry loop (`attempt_limit = max_retries+1`) → **strip `respond`** (pure-respond → text+`finish_reason:"stop"`; mixed → only real calls) → `ToolCallError` caught returns last raw text, not a 5xx. +- **Response shapes** (convert.py): `chatcmpl-<12hex>` ids, `call_<8hex>` tool-call ids, `arguments` **JSON-stringified**, usage zeros. Tool-call finish_reason `"tool_calls"`, text `"stop"`. +- **SSE** (server.py:310-340): send headers immediately (liveness), then HTTP chunked frames `data: <json>\n\n`, terminate `data: [DONE]` + `0\r\n\r\n`. Note forge materializes the whole response then chunks it (no true token streaming) — match unless we wire jcode's real streaming. +- **CLI flags** (`/forge proxy …` and a `jcode proxy` subcommand): `--backend-url` (external) XOR `--backend {llamaserver,llamafile,ollama}` (managed); `--host`/`--port`(8081)/`--backend-port`(8080); `--max-retries`(3); `--no-rescue`; `--budget-mode {backend,manual,forge-full,forge-fast}`; `--budget-tokens`; `--serialize`/`--no-serialize` (auto: managed→on, external→off); `--model`/`--gguf`. +- **BudgetMode** (server.py:26): backend (trust `/props`), manual (`-c N`), forge-full (VRAM tier or `/props` max), forge-fast (half of full). VRAM tiers per §3.1. +- **ServerManager** (backend.ss): managed-mode boot of `llama-server -m GGUF -ngl 999 --port P [--jinja] [-c N] …`; health via polling `/props` for `default_generation_settings` (NOT `/health`), 180s/2s; stop terminates then **sleeps 3s** for VRAM release; ollama uses `ollama stop <model>`. +- **Concurrency:** `--serialize` runs a single inference worker (single-GPU); disconnect-cancel polls writer-closing every 1s. Use `with-mutex` around the shared transcoded socket port (see §7 concurrency gotcha). + +### 5.D Eval harness (forge/tests/eval) + +Port as pure logic; reconcile counts (30 registered = 26 documented [OG-18 + 8 advanced_reasoning] + 4 `compaction_chain_*`). +- **`scenario.ss`** — `EvalScenario`: `name, description, workflow, user_message, budget_tokens=8192, max_iterations=15, max_retries_per_step=5, max_tool_errors=2, validate(args)->bool|#f, validate_state()->bool|#f, build_workflow()->（workflow . validate_state), tags=[], ideal_iterations`. Grading = `validate(captured terminal args) AND validate_state()`; terminal tool wrapped in a capturing closure. `_check(text, substrings)` = lowercase + strip commas + AND-contains. Stateful scenarios use a mutable record (Jerboa `defstruct` + setters) where forge uses a closure-captured DB. +- **Scoring:** `completeness` = runner returned without raising; `accuracy` = grader true (or `#f`/none); `iterations_used` = provider `send` count (wrap provider in a counting shim); `StreamError` → up to 2 fresh re-runs. +- **`ablation.ss`** — 7 presets, exact disables: `reforged`(none), `no_rescue`(rescue off), `no_nudge`(rescue off + retries 0), `no_steps`(required_steps→[]), `no_recovery`(max_tool_errors 0), `no_compact`(force NoCompact, skip compaction scenarios), `bare`(all off). These directly back `/forge ablation`. +- **`metrics.ss`/`report.ss`** — per-scenario completion/correctness/iterations/elapsed/nudges/tool-errors/compaction-phases/wasted-calls; ConfigMetrics score/accuracy/completeness/efficiency/speed; **ASCII table + list + progress + `--markdown` (7 views)** — all portable. **`--html` dashboard NOT ported.** +- **`significance.ss`** — pooled McNemar (exact binomial when discordant ≤25, else continuity-corrected χ²) + Wilson 95% CI, paired on `(scenario,run)` vs reforged. Pure `math` (needs `erfc`, `log1p`) — verify Jerboa stdlib has these or port them. +- **`runner.ss`/`batch.ss`** — single + batch (JSONL append, resume by counting existing `model|backend|mode|ablation|tool_choice|scenario` rows). JSONL row shape per the inventory (model…cost_usd). + +--- + +## 6. Phased roadmap + +Each phase is independently buildable + testable (`make build` → `make binary` → `./jcode …`). Ship guardrails before workflow/proxy/eval. + +**Phase 0 — Scaffolding.** Create `guardrails/` dir + `nudge.ss`, `error-tracker.ss`, errors. Port nudge text + ErrorTracker (pure, trivial unit tests). No loop changes yet. + +**Phase 1 — Message typing + rescue unification.** Add a `MessageType`-equivalent tag to jcode messages (needed by TieredCompact + nudge classification). Consolidate jcode's existing parsers + forge strategies #2/#4 into `guardrails/rescue.ss`; point `agent.ss` and (later) the proxy at it. Unit-test each rescue format against fixtures. + +**Phase 2 — Validator + in-loop retry/respond (the core lift).** `validator.ss`, `respond.ss`, `guardrails.ss`. Wire into `agent.ss agent-loop`/`agent-loop-stream`: validate → nudge → re-infer → error budget; register `respond` internal tool; unwrap respond. This is where always-on guardrails become real for the coding loop. Validate on a local model (ollama/mlx) end-to-end. + +**Phase 3 — Sampling map.** `provider/sampling.ss` with the full verbatim map + policy; replace `apply-mlx-sampling!`; non-strict default. Unit-test lookup + 4-quadrant policy. + +**Phase 4 — Compaction strategies.** `compaction-strategy.ss` (NoCompact/Sliding/Tiered + JcodeLegacy); `hardware.ss` (NVIDIA `nvidia-smi`, AMD sysfs, tiers). Default to Tiered. Test phase transitions against synthetic histories. + +**Phase 5 — Workflow surface.** `workflow.ss` + `workflow-runner.ss` (+SlotWorker). `/forge workflow`. Port forge's step/prereq/runner unit tests. + +**Phase 6 — Proxy.** `proxy/http.ss` (HTTP/1.1 + SSE) → `convert.ss` → `handler.ss` → `server.ss` → `backend.ss`. `/forge proxy` + `jcode proxy`. Test with `curl` (JSON + SSE) and by pointing an external OpenAI client at it. + +**Phase 7 — Eval harness.** `eval/*` + scenarios. `/forge eval`. Reproduce a forge eval row on one model+backend and compare numbers. `significance.ss` last (needs `erfc`). + +**`/forge` command** lands incrementally: status in Phase 2, ablation in Phase 7 (or stub earlier), proxy/eval subcommands as those phases complete. + +--- + +## 7. Jerboa implementation notes (from project memory — non-negotiable) + +- **Write `.ss`, never `.sls`.** Use `(jerboa prelude)`, `def`, `defstruct`, `[...]`/`{...}`, `keyword:` args, `(std …)` modules. Edit `.ss` only via `jerboa_balanced_replace` (dry-run → apply → `jerboa_check_balance`), never raw `Edit`. Verify with `jerboa_compile_check` before declaring done. +- **Build gotchas:** forward references fail in generated `.sls` — define before use. The `else` keyword is captured by `guard` — don't use `else` as a guard clause (bites the errors port directly). Always `(import (std misc string))` for `string-prefix?`/`string-join`/`string-split`. +- **Plugin/binary boot deps:** every transitive `(jerboa prelude)` dependency of new modules must be added to `build-binary.ss` `external-libs` — WPO inlining is not enough. New dirs (`guardrails/`, `proxy/`, `eval/`) need their modules registered there. +- **Concurrency:** a shared transcoded port across green threads corrupts framing — wrap each send+read pair (proxy sockets, any shared stream) in `with-mutex` (`chez-make-mutex`). Chez parameters **don't inherit on spawn** — every spawned thread that logs or serves must re-`parameterize` `current-error-port`/`current-log-level` (the proxy worker + parallel tool exec; `agent.ss execute-tool-calls` already shows the pattern). Blocking FFI/socket calls must declare `__collect_safe` or they pin the thread mutex and starve other green threads. +- **Sleep:** use `thread-sleep!`, not `(sleep (make-time …))` (the prelude shadows `make-time`; it raises silently). Relevant to ServerManager's 3s VRAM wait and health polling. +- **max_tokens:** local provider bodies must set `max_tokens` (jcode already sets 32768 OpenAI / 8192 Anthropic) — keep it; truncated streams break the parser (prior MLX/qwen3 bug). + +--- + +## 8. Testing & acceptance + +- **Unit tests** (`test/run.ss`): forge ships ~865 deterministic tests with no backend. Port the guardrail/validator/step/rescue/compaction/significance suites — they're pure logic and are the fidelity contract. Target: every constant/nudge/phase in §3 has a test. +- **Integration:** `make binary` + `./jcode --help`, then a real local-model session (ollama/mlx) exercising rescue + respond + retry; `curl` against the proxy (JSON + SSE); `/forge eval --scenario basic_2step --runs 5`. +- **Acceptance (the 100% bar):** reproduce a forge eval config (one model + backend + ablation grid) and compare score/completeness/efficiency to forge's JSONL within noise; McNemar significance vs `bare` should reproduce forge's per-guardrail contribution direction. +- **Never commit `jcode.json`** (live keys); scan `git diff --cached` for `sk-`/`AIza`/`Bearer ` before any commit; test the binary before committing (project rule, stated 3+ times). + +--- + +## 9. Open questions / risks + +1. **respond in a coding agent (§5.A).** Forcing `respond` is forge's biggest small-model win, but jcode's TUI/markdown render assumes bare-text finals. Plan: unwrap respond → normal assistant message. Risk: double-handling streaming text. Validate early in Phase 2. +2. **Always-on for cloud models.** Per the decision, frontier models also get the stack. `respond` injection + validate is cheap and harmless, but the retry/nudge loop must not fight a model that legitimately wants to answer in text — the rescue-first-then-respond path covers this. Watch for added latency; `/forge ablation no_nudge` is the escape hatch. +3. **Workflow features vs open loop.** required_steps/prereqs/terminal only meaningful under `/forge workflow`; in the default loop they're inert. Confirm that's the intended "100%" reading (functionality present and reachable, not forced onto every coding turn). +4. **Proxy from scratch.** No HTTP/SSE in jcode today — most net-new code; budget Phase 6 accordingly. Decide whether to expose true token streaming (jcode has it) or match forge's materialize-then-chunk. +5. **`erfc`/`log1p` availability** for significance — confirm in Jerboa stdlib or port the numerics. +6. **Eval location** (`src/` vs `test/`) and **scenario count** doc drift (forge's `report.py` references dead legacy scenario names) — adopt the live 30 and document the 26+4 split. + +--- + +*Provenance: forge `src/forge/{guardrails,core,clients,context,prompts,tools,proxy}/*.py`, `server.py`, `errors.py`, `tests/eval/*`, `docs/ARCHITECTURE.md`, ADRs 005/006/011/012/013/014. jcode integration points: `cli.ss:283-388`, `agent.ss` (agent-loop ~1066, execute-tool-calls ~1167, system-prompt ~37), `provider.ss` (openai-body ~772, rescue ~641-748, sampling ~571, retry ~56-81), `registry.ss:61-104`, `compaction.ss`, `hooks.ss`, `config.ss`, `models.ss`, `serve.ss`.* new file mode 100644 --- /dev/null +++ b/src/jcode/guardrails/error-tracker.ss @@ -0,0 +1,66 @@ +;;; jcode guardrail error tracker +;;; +;;; Verbatim port of forge's ErrorTracker (guardrails/error_tracker.py). +;;; Tracks consecutive retry and tool-error counts against limits. Exhaustion +;;; is strict `>` — a budget of N tolerates N failures; the (N+1)th trips it. +;;; This matches forge exactly. Stateful: instantiate one per session/task. + +(export make-error-tracker + error-tracker? + error-tracker-record-retry! + error-tracker-reset-retries! + error-tracker-record-result! + error-tracker-reset-errors! + error-tracker-retries-exhausted? + error-tracker-tool-errors-exhausted? + error-tracker-consecutive-retries + error-tracker-consecutive-tool-errors) + +;; Private carrier; public API uses the error-tracker-* wrappers below +;; (same convention as message.ss's tool-call-data). +(defstruct etracker + (max-retries max-tool-errors consecutive-retries consecutive-tool-errors)) + +(def (make-error-tracker max-retries max-tool-errors) + (make-etracker max-retries max-tool-errors 0 0)) + +(def (error-tracker? x) (etracker? x)) + +;; Record a validation failure (TextResponse or unknown tool). +(def (error-tracker-record-retry! t) + (etracker-consecutive-retries-set! t (+ 1 (etracker-consecutive-retries t)))) + +;; Reset retry counter (call on successful validation). +(def (error-tracker-reset-retries! t) + (etracker-consecutive-retries-set! t 0)) + +;; Record a tool execution result. +;; success? — #t if the tool executed without error +;; is-soft-error? — optional; #t for resolution/soft errors that do not count +;; toward the budget (default #f). Ignored when success?. +;; A single success does NOT reset the counter — only a fully clean batch does +;; (call error-tracker-reset-errors! after a zero-error batch). +(def (error-tracker-record-result! t success? . opt) + (let ((is-soft-error? (and (pair? opt) (car opt)))) + (unless success? + (unless is-soft-error? + (etracker-consecutive-tool-errors-set! + t (+ 1 (etracker-consecutive-tool-errors t))))))) + +;; Reset tool error counter (call after a fully clean batch). +(def (error-tracker-reset-errors! t) + (etracker-consecutive-tool-errors-set! t 0)) + +;; True if consecutive retries exceed the limit (strict >). +(def (error-tracker-retries-exhausted? t) + (> (etracker-consecutive-retries t) (etracker-max-retries t))) + +;; True if consecutive tool errors exceed the limit (strict >). +(def (error-tracker-tool-errors-exhausted? t) + (> (etracker-consecutive-tool-errors t) (etracker-max-tool-errors t))) + +(def (error-tracker-consecutive-retries t) + (etracker-consecutive-retries t)) + +(def (error-tracker-consecutive-tool-errors t) + (etracker-consecutive-tool-errors t)) new file mode 100644 --- /dev/null +++ b/src/jcode/guardrails/nudge.ss @@ -0,0 +1,92 @@ +;;; jcode guardrail nudges +;;; +;;; Verbatim port of forge's Nudge dataclass (guardrails/nudge.py) plus the +;;; message templates in prompts/nudges.py. A Nudge is a corrective message a +;;; guardrail injects into history when the model misbehaves. The text strings +;;; are byte-for-byte identical to forge for behavioural fidelity — do not +;;; reword them. + +(export make-nudge + nudge? + nudge-role + nudge-content + nudge-kind + nudge-tier + retry-nudge-text + unknown-tool-nudge-text + step-nudge-text + prerequisite-nudge-text + make-retry-nudge + make-unknown-tool-nudge + make-step-nudge + make-prerequisite-nudge) + +(import :std/misc/string) + +;; role: message role for injection ("user", "system", or "tool") +;; content: the nudge text +;; kind: "retry" | "unknown_tool" | "step" | "prerequisite" +;; tier: escalation level for step nudges (0 = N/A, 1-3 = escalating) +(defstruct nudge (role content kind tier)) + +;; ── Verbatim text templates (forge prompts/nudges.py) ──────────────── + +(def (retry-nudge-text raw-response) + ;; raw-response is unused — kept for signature compatibility with forge. + (string-append + "Your previous response was not a valid tool call. " + "You must respond with a tool call, not free text. " + "Please try again with a valid tool call.")) + +(def (unknown-tool-nudge-text tool-name available-tools) + (string-append + "Tool '" tool-name "' does not exist. " + "Available tools: " (string-join available-tools ", ") ". " + "Call one of them.")) + +(def (clamp-tier tier) + (max 1 (min 3 tier))) + +;; Escalating nudge for premature terminal-tool attempts (tier clamped 1-3). +(def (step-nudge-text terminal-tool pending-steps tier) + (let ((tier (clamp-tier tier)) + (steps (string-join pending-steps ", "))) + (cond + ((= tier 1) + (string-append + "You cannot call " terminal-tool " yet. " + "You must first complete these required steps: " steps ". " + "Call one of them now.")) + ((= tier 2) + (string-append + "You must call one of these tools now: " steps ". " + "Pick one.")) + (else + (string-append + "STOP. You MUST call one of: " steps ". " + "Do NOT call " terminal-tool ". " + "Your next response MUST be a tool call to one of: " steps "."))))) + +(def (prerequisite-nudge-text tool-name missing-prereqs) + (string-append + "You cannot call " tool-name " yet. " + "You must first call: " (string-join missing-prereqs ", ") ". " + "Call the prerequisite tool now.")) + +;; ── Nudge constructors (role + kind + tier wired per forge) ────────── + +(def (make-retry-nudge raw-response) + (make-nudge "user" (retry-nudge-text raw-response) "retry" 0)) + +(def (make-unknown-tool-nudge tool-name available-tools) + (make-nudge "user" (unknown-tool-nudge-text tool-name available-tools) + "unknown_tool" 0)) + +;; tier is stored as forge stores it: min(premature_attempts, 3), already 1-3. +(def (make-step-nudge terminal-tool pending-steps tier) + (make-nudge "user" (step-nudge-text terminal-tool pending-steps tier) + "step" tier)) + +(def (make-prerequisite-nudge tool-name missing-prereqs) + (make-nudge "user" (prerequisite-nudge-text tool-name missing-prereqs) + "prerequisite" 0)) --- a/test/run.ss +++ b/test/run.ss @@ -7,7 +7,9 @@ (jcode provider provider) (jcode tool registry) (jcode tool file) - (jcode tool bash)) + (jcode tool bash) + (jcode guardrails nudge) + (jcode guardrails error-tracker)) ;; ── Helpers ────────────────────────────────────────────────────── @@ -218,6 +220,111 @@ (check! "recover: tool-call name" (tool-call-name (car (message-tool-calls fixed))) "bash")) +;; ── Guardrails: nudges ──────────────────────────────────────────── +;; Verbatim port of forge's nudge templates. Texts must be byte-for-byte +;; identical to forge — these checks lock that down. + +(section "=== guardrails: nudges ===") + +(check! "retry-nudge-text verbatim" + (retry-nudge-text "anything") + (string-append + "Your previous response was not a valid tool call. " + "You must respond with a tool call, not free text. " + "Please try again with a valid tool call.")) + +(let ([n (make-retry-nudge "junk")]) + (check! "retry nudge role" (nudge-role n) "user") + (check! "retry nudge kind" (nudge-kind n) "retry") + (check! "retry nudge tier" (nudge-tier n) 0)) + +(check! "unknown-tool-nudge-text verbatim" + (unknown-tool-nudge-text "foo" '("a" "b")) + "Tool 'foo' does not exist. Available tools: a, b. Call one of them.") + +(check! "unknown-tool nudge kind" + (nudge-kind (make-unknown-tool-nudge "foo" '("a" "b"))) "unknown_tool") + +(check! "step-nudge tier 1 verbatim" + (step-nudge-text "respond" '("read") 1) + (string-append + "You cannot call respond yet. " + "You must first complete these required steps: read. " + "Call one of them now.")) + +(check! "step-nudge tier 2 verbatim" + (step-nudge-text "respond" '("read" "grep") 2) + "You must call one of these tools now: read, grep. Pick one.") + +(check! "step-nudge tier 3 verbatim" + (step-nudge-text "respond" '("read") 3) + (string-append + "STOP. You MUST call one of: read. " + "Do NOT call respond. " + "Your next response MUST be a tool call to one of: read.")) + +;; tier clamps to 1..3 +(check! "step-nudge tier 0 clamps to 1" + (step-nudge-text "respond" '("read") 0) + (step-nudge-text "respond" '("read") 1)) +(check! "step-nudge tier 9 clamps to 3" + (step-nudge-text "respond" '("read") 9) + (step-nudge-text "respond" '("read") 3)) + +(let ([n (make-step-nudge "respond" '("read") 2)]) + (check! "step nudge kind" (nudge-kind n) "step") + (check! "step nudge tier preserved" (nudge-tier n) 2)) + +(check! "prerequisite-nudge-text verbatim" + (prerequisite-nudge-text "write" '("read")) + "You cannot call write yet. You must first call: read. Call the prerequisite tool now.") + +(check! "prerequisite nudge kind" + (nudge-kind (make-prerequisite-nudge "write" '("read"))) "prerequisite") + +;; ── Guardrails: error tracker ───────────────────────────────────── +;; Exhaustion is strict `>`: a budget of N tolerates N, the (N+1)th trips it. + +(section "=== guardrails: error tracker ===") + +(let ([t (make-error-tracker 3 2)]) + (check! "fresh retries 0" (error-tracker-consecutive-retries t) 0) + (check! "fresh tool-errs 0" (error-tracker-consecutive-tool-errors t) 0) + (check! "fresh not exhausted" (error-tracker-retries-exhausted? t) #f) + + ;; 3 retries == budget, not yet exhausted (3 > 3 is false) + (error-tracker-record-retry! t) + (error-tracker-record-retry! t) + (error-tracker-record-retry! t) + (check! "retries at budget count" (error-tracker-consecutive-retries t) 3) + (check! "retries at budget ok" (error-tracker-retries-exhausted? t) #f) + ;; 4th trips it + (error-tracker-record-retry! t) + (check! "retries over budget" (error-tracker-retries-exhausted? t) #t) + ;; reset clears + (error-tracker-reset-retries! t) + (check! "retries reset" (error-tracker-retries-exhausted? t) #f) + (check! "retries reset to 0" (error-tracker-consecutive-retries t) 0)) + +(let ([t (make-error-tracker 3 2)]) + ;; success never increments + (error-tracker-record-result! t #t) + (check! "success no tool-err" (error-tracker-consecutive-tool-errors t) 0) + ;; soft errors do not count + (error-tracker-record-result! t #f #t) + (check! "soft-err no count" (error-tracker-consecutive-tool-errors t) 0) + ;; 2 hard errors == budget, not exhausted + (error-tracker-record-result! t #f) + (error-tracker-record-result! t #f) + (check! "tool-errs at budget" (error-tracker-consecutive-tool-errors t) 2) + (check! "tool-errs at budget ok" (error-tracker-tool-errors-exhausted? t) #f) + ;; 3rd trips it + (error-tracker-record-result! t #f) + (check! "tool-errs over budget" (error-tracker-tool-errors-exhausted? t) #t) + ;; clean batch resets + (error-tracker-reset-errors! t) + (check! "tool-errs reset" (error-tracker-tool-errors-exhausted? t) #f)) + ;; ── Results ─────────────────────────────────────────────────────── (printf "~n~a passed, ~a failed~n" pass-count fail-count)