Improve verified Jerboa workflow recovery
ober
22e982475fcb88ac0e39ddfeae43499ea0db0c79
--- a/docs/FORGE.md +++ b/docs/FORGE.md @@ -33,7 +33,7 @@ see [`FORGE_PORT_PLAN.md`](FORGE_PORT_PLAN.md). | `/forge verify` | Describe + self-test the verify-gate (ATLAS verify + repair). | | `/forge bestofk` · `/forge best-of-k` | Describe + self-test best-of-k diverse generation. | | `/forge breaker` · `/forge no-progress` | Describe + self-test the no-progress loop breaker. | -| `/forge run [opts] <task>` | Verify-gated coding on the **live** model (edit → verify → done). Options: `--verify`, `--bestof`, `--cwd`, `--write-scope`, `--guidance-file`, `--json`, `--status-file`, `--run-aliases`. See [verified.md](verified.md). | +| `/forge run [opts] <task>` | Verify-gated coding on the **live** model (edit → verify → done). Options: `--verify`, `--bestof`, `--max-iterations`, `--cwd`, `--write-scope`, `--guidance-file`, `--json`, `--status-file`, `--run-aliases`. See [verified.md](verified.md). | The `workflow` / `proxy` / `ablation` / `verify` / `bestofk` / `breaker` self-tests run real code against scripted inputs (no live model), so they double @@ -52,23 +52,24 @@ HTTP endpoint. See [The proxy](#the-openai-compatible-proxy) below. ### `jcode verified` (subcommand) ```bash -jcode verified "<task>" [--bestof K] [--verify CMD] [--cwd DIR] [--write-scope PATHS] [--guidance-file FILE] [--json] [--status-file FILE] [--run-aliases] +jcode verified "<task>" [--bestof K] [--max-iterations N] [--verify CMD] [--cwd DIR] [--write-scope PATHS] [--guidance-file FILE] [--json] [--status-file FILE] [--run-aliases] ``` Runs the [verify-gate](#verify-gate) on the live model — edit → run `CMD` → -finish only when it passes — optionally drawing the best of `K` candidates. The -non-interactive twin of `/forge run`. +review the caller requirements → finish — optionally drawing the best of `K` +candidates. Editing during the review invalidates the prior passing result and +requires another verify. This is the non-interactive twin of `/forge run`. Use `--json` when another harness owns the terminal: stdout becomes one final JSON object and the human trajectory moves to stderr. Use `--status-file FILE` to write the same object to disk. `jcode verified` exits `0` only after a passing verify and exits `1` when the run stops. The status object carries `ok`, `status`, `verify_passed`, `exit_code`, `exit_reason`, `error_type`, -`error`, `summary`, `provider`, `model`, `verify_command`, `cwd`, and -`write_scope`. +`error`, `summary`, `provider`, `model`, `verify_command`, `cwd`, `write_scope`, +and provider-reported token/cache/reasoning/cost usage. -If MCP servers are initialized, their registered MCP-origin tools are included -in the verified workflow as structured external tools. Code changes still go +If MCP servers are initialized, the CLI includes a focused set of registered +MCP-origin recipe, syntax, failure, and API lookup tools. Code changes still go through the verified workflow's own edit tools, and success still requires the configured `verify` command to pass. `--no-mcp` keeps those external tools out by skipping MCP startup. --- a/docs/cli.md +++ b/docs/cli.md @@ -36,8 +36,8 @@ Parsed before any subcommand. |---|---| | `JCODE_CONTEXT_WINDOW` | Overrides model context-window metadata for the current process. Useful when a configured MLX provider points at a remote host with more KV-cache headroom than the local default. | | `JCODE_MAX_TOKENS` | Overrides the OpenAI-compatible `max_tokens` request cap for the current process. Useful for bounding local model repair turns. | -| `JCODE_READ_ROOTS` | Colon-separated extra read-only roots for verified runs. File tools can inspect those roots, while edits remain limited by write scope. | -| `JCODE_VERIFIED_COMPACT` | When truthy, verified runs expose a smaller MCP tool menu and prompt local-style agents to write an initial complete version earlier, then repair from verifier output. | +| `JCODE_READ_ROOTS` | Colon-separated extra read-only roots for verified runs. File tools can inspect those roots, while edits remain limited by write scope. An existing `$JERBOA_HOME/lib` is included automatically. | +| `JCODE_VERIFIED_COMPACT` | Forces the focused verified MCP menu for programmatic `coding-workflow` users. The `jcode verified` CLI enables this focused menu by default. | ## Subcommands @@ -92,20 +92,22 @@ see [remote.md](remote.md) for complete examples. ### `verified` ``` -jcode verified "<task>" [--bestof K] [--verify CMD] [--cwd DIR] [--write-scope PATHS] [--guidance-file FILE] [--json] [--status-file FILE] [--run-aliases] +jcode verified "<task>" [--bestof K] [--max-iterations N] [--verify CMD] [--cwd DIR] [--write-scope PATHS] [--guidance-file FILE] [--json] [--status-file FILE] [--run-aliases] ``` -Runs an edit→verify→done loop on the live model: it edits, runs `--verify CMD` -(e.g. `make test`), and only declares success once the command passes — -optionally drawing the best of `K` diverse candidates. `--write-scope` accepts +Runs an edit→verify→requirements-review→done loop on the live model: it edits, +runs `--verify CMD` (e.g. `make test`), reviews the full caller requirements +after the tests pass, and only then declares success — optionally drawing the +best of `K` diverse candidates. An edit during review invalidates the prior +pass and requires another verify. `--write-scope` accepts `all`, `none`, or comma-separated path prefixes such as `src/tetris/,tests/`. `--guidance-file` appends caller-supplied task context to the verified workflow prompt; use it for cookbook-generated task bundles or other external examples without baking task-specific knowledge into jcode. -When MCP is initialized, MCP-origin tools are also exposed to the verified -workflow as structured tools, so validators and project-specific MCP helpers are -available without bypassing the verified edit/verify gate. Use `--no-mcp` to -skip MCP initialization for the process. +When MCP is initialized, a focused set of MCP-origin recipe, syntax, failure, +and API lookup tools is exposed as structured tools. The menu narrows further +between edits and verification so discovery cannot displace the verifier. Use +`--no-mcp` to skip MCP initialization for the process. By default, `run`, `bash`, and `shell` are not exposed in verified mode. If a local model calls them anyway, jcode returns corrective tool-resolution feedback; repeated attempts get stronger wording and do not count against the @@ -113,13 +115,20 @@ normal tool-error budget. `--run-aliases` opts back into the legacy narrow inspection aliases, but `verify` remains the authority for build and behavior checks. +Verified runs allow 72 model turns by default. Use `--max-iterations N` for a +larger caller-owned wall-clock budget; `N` must be a positive integer. The +existing no-progress and tool-error breakers still stop stagnant trajectories. + Automation can use `--json` to emit a single final status object on stdout; the human trajectory is written to stderr in that mode. `--status-file FILE` writes the same final status object to disk in both human and JSON modes. The command exits `0` only when verification passed, and exits `1` when the verified run stopped. Status fields include `ok`, `status`, `verify_passed`, `exit_code`, `exit_reason`, `error_type`, `error`, `summary`, `provider`, `model`, -`verify_command`, `cwd`, and `write_scope`. +`verify_command`, `cwd`, `write_scope`, token/cache/reasoning usage, and +provider cost when reported by the provider. Typed provider failures also add +`error_provider` and `provider_http_status`, avoiding error-string parsing in +benchmark automation. See [verified.md](verified.md) for the full workflow, MCP cookbook guidance, JSON status shape, and troubleshooting notes. @@ -189,7 +198,7 @@ Useful built-in workflow skill: | `/forge verify` | Describe + self-test the verify-gate. | | `/forge bestofk` · `/forge best-of-k` | Describe + self-test best-of-k generation. | | `/forge breaker` · `/forge no-progress` | Describe + self-test the no-progress breaker. | -| `/forge run [opts] <task>` | Verify-gated coding on the live model. Supports `--verify`, `--bestof`, `--cwd`, `--write-scope`, `--guidance-file`, `--json`, `--status-file`, and `--run-aliases`. | +| `/forge run [opts] <task>` | Verify-gated coding on the live model. Supports `--verify`, `--bestof`, `--max-iterations`, `--cwd`, `--write-scope`, `--guidance-file`, `--json`, `--status-file`, and `--run-aliases`. | Each `/forge` self-test runs real code against scripted inputs (no live model), so it doubles as a smoke test. Full semantics in [FORGE.md](FORGE.md). --- a/docs/escalation.md +++ b/docs/escalation.md @@ -23,7 +23,9 @@ model is only used when both `expert.provider` and `expert.model` are set. "escalation": { "request_logprobs": true, "min_mean_logprob": -2.5, - "max_mean_entropy": 1.3 + "max_mean_entropy": 1.3, + "max_recoverable_tool_errors": 4, + "max_verified_prose_retries": 2 } } } @@ -135,16 +137,29 @@ verification or an expert model." Do not treat them as a factuality score. | `identical-tool-loop` | The assistant repeated the same tool call and arguments several times. | | `no-text-rounds` | The assistant produced repeated empty text turns. | | `tool-error-streak` | Several consecutive tool results looked like errors. | +| `recoverable-tool-error-streak` | Several recoverable mutation refusals occurred since the last successful edit. | | `no-progress` | Several tool results canonicalized to the same output. | | `low-mean-logprob` | `mean_logprob` was below `min_mean_logprob`. | | `high-mean-entropy` | `mean_entropy` was above `max_mean_entropy`. | | `truncated-response` | The provider stopped because the reply hit the token cap. | +| `provider-aborted` | The provider returned `finish_reason: error`. | +| `verified-prose-drift` | A verified workflow returned prose again after its structured tool-call retry nudge. | + +Read-only observations do not reset recoverable refusal pressure. A successful +edit/write/create or passing verifier does. This catches models that alternate +rejected mutations with `read` or `balance` calls without changing the +workspace. Lookup misses from repository exploration, such as `File not found`, `Path not found`, and `Not a directory`, do not count toward `tool-error-streak`. They are treated as recoverable path discovery rather than evidence that the model is stuck. +`max_verified_prose_retries` defaults to `2`: the first prose-only miss gets a +normal structured-call nudge, and a second prose-only response routes to the +expert. The signal is verified-workflow-specific and does not classify normal +chat responses as failures. Set it to `false` to disable this route. + The primary model can also explicitly request the expert by emitting: ```text @@ -153,8 +168,13 @@ The primary model can also explicitly request the expert by emitting: When escalation happens, `jcode` sends the conversation to the configured expert model with a short handoff note explaining why the primary model was escalated. -The expert response replaces the primary response. If the expert call fails, -`jcode` falls back to the primary response instead of dropping a usable answer. +For tool-driven workflows, the note explicitly asks the expert to continue from +the current state with the next valid tool call instead of returning a prose-only +summary. It also preserves a bounded, sentinel-free excerpt of the primary +model's final note, so an explicit request can explain the missing fact without +creating an unresolved assistant tool-call turn. The expert response replaces +the primary response. If the expert call fails, `jcode` falls back to the +primary response instead of dropping a usable answer. For provider evaluation runs where the primary model must be the only model called, pass `--no-expert` or set `JCODE_NO_EXPERT=1`. In the TUI, you can also new file mode 100644 --- /dev/null +++ b/docs/rle-benchmark-optimization.md @@ -0,0 +1,1181 @@ +# Jcode Jerboa Benchmark Optimization + +## Objective + +Improve `jcode verified` so OpenRouter models reliably produce correct Jerboa +programs on parser-heavy, multi-step tasks. Changes are evaluated against the +hard RLE Life benchmark in `~/mine/kratistos` using only jcode with: + +- `z-ai/glm-5.2` +- `deepseek/deepseek-v4-pro` +- `moonshotai/kimi-k2.7-code` +- `qwen/qwen3.7-plus` + +Each model receives a 1,800-second wall-clock budget. Jerboa MCP and expert mode +remain enabled. The authoritative gate is `make test`; CLI exit alone is not a +pass. Passing artifacts also receive the six additional specification probes +recorded in `~/mine/kratistos/results.md`. + +## Baseline + +Run: + +```text +~/mine/kratistos/runs/20260715-030909-life-rle-hard-opencode-vs-jcode-four-models +``` + +| Model | Result | Wall time | Failure mode | +|---|---|---:|---| +| DeepSeek V4 Pro | PASS 22/22, hidden 6/6 | 8m58s | None | +| GLM 5.2 | PASS 22/22, hidden 5/6 | 23m10s | Accepted illegal `2!` input | +| Qwen 3.7 Plus | FAIL | 25m56s | Fixed 48-iteration limit during active repair | +| Kimi K2.7 Code | FAIL | 5m56s | Truncated provider JSON after repeated rejected drafts | + +Baseline pass rate is 2/4 for the bundled suite and 1/4 for full specification +compliance. + +## Current Outcome + +The retained implementation passes 1,166 tests with zero failures and one +environment-dependent image-backend skip. Provider-level local integration now +proves that an explicit primary escalation reaches a separately configured +expert, preserves the bounded help note and active tool protocol, returns a +structured edit call, and records both usage events. + +Live exact-model scoring after the early successful cohorts is blocked by the +OpenRouter account balance: every current GLM 5.2, DeepSeek V4 Pro, Kimi K2.7 +Code, and Qwen 3.7 Plus request returns HTTP 402 before inference. These are +recorded as typed `provider_api` failures with zero tokens and unknown cost, not +as model failures. Consequently the latest retained changes use deterministic +trajectory replay, full-suite coverage, installed dispatch probes, and earlier +live run evidence; this report does not invent post-402 pass-rate deltas. + +Final task-identical cohort: + +```text +~/mine/kratistos/runs/20260715-092614-life-rle-jcode-stream-usage +``` + +All four candidates used the final installed binary, the complete hard RLE +task, expert mode, and the 72-turn default. The archived `command.txt` files do +not contain `--no-expert`. Each stopped in about one second at the account-level +402 before producing a model turn or workspace diff. + +## Experiment Rules + +1. Change one workflow behavior at a time. +2. Add deterministic regression coverage before or with each implementation. +3. Run the full jcode unit suite after each code change. +4. Rebuild and install the tested binary before each model cohort. +5. Run the same four models with the same task, seed, MCP setup, and timeout. +6. Keep a change only if it improves pass rate, specification quality, useful + progress, or wall time without introducing a correctness regression. +7. Record unsuccessful experiments and revert their code while preserving the + result here. +8. Once the benchmark reaches 4/4, move to a harder Jerboa benchmark. + +## Recommendations + +### 1. Make rejected drafts fully editable + +Status: implemented and retained. See **Editable Rejected Drafts** and the +subsequent staged-repair experiments. + +Baseline behavior retained a rejected new-file draft, but only `read`, `balance`, +full-file rewrite, and exact `old_str` replacement can use it. `line_edit`, +`replace_range`, and `replace_def` require an on-disk target. After the small +inspection budget is exhausted, exact replacement is disabled too. + +Treat the rejected draft as an explicit staging buffer. Apply local edit tools +to that buffer, rerun the syntax guard, and promote it to the workspace only +when it passes. If a staged repair remains invalid, retain the repaired draft +and return the new focused balance diagnostic. Do not force another full-file +generation. + +Expected benefit: fewer repeated 250-400 line generations and better recovery +for Kimi, Qwen, and GLM. + +### 2. Use progress-aware iteration budgets + +Status: partially implemented. Jcode now defaults to 72 iterations and exposes +`--max-iterations`; adaptive extension based on objective progress remains a +future option. + +The baseline defaulted to 48 iterations independently of the caller's wall +budget. Qwen exhausted that at 25m56s while edits and verifier diagnostics were +still changing, leaving over four minutes unused. + +Expose a CLI `--max-iterations` option and let the benchmark pass a budget +appropriate to 30-minute tasks. Longer term, distinguish productive iterations +from repeated/no-progress iterations and extend only when there is objective +progress such as a successful edit or changed verifier failure. + +Expected benefit: allow active repair trajectories to consume the requested +time without making stagnant loops unbounded. + +### 3. Retry truncated OpenAI-compatible responses + +Status: implemented and retained. See **Retry Truncated Provider JSON**. + +Kimi terminated on `read-json: unexpected EOF`. A complete HTTP 200 body can +still contain truncated JSON due to an interrupted upstream response. + +Retry narrowly classified JSON parse failures with bounded backoff. Reuse the +same request and preserve the conversation because no assistant message was +accepted. Do not retry valid API errors or arbitrary model/tool failures. + +Expected benefit: transport truncation becomes a recoverable provider event +instead of discarding an otherwise viable trajectory. + +### 4. Relax structural repair gating only for relevant edits + +Status: implemented and retained. See **Structural Repair Overlap** and +**`replace_def` Structural Span Enforcement**. + +At baseline, a pending structural diagnosis required a replacement +range to fully cover the diagnosed span. A model can select a smaller line or +subrange that overlaps and directly fixes the error, but that edit is refused. + +Allow `line_edit`, exact replacement, `replace_def`, and overlapping +`replace_range` operations on the diagnosed file. Keep unrelated files and +non-overlapping edits blocked. Verification remains mandatory after the edit. + +Expected benefit: remove recovery deadlocks while retaining focused repair. + +### 5. Stop charging repeated discovery refusals + +Status: implemented and retained through recoverable refusals and phase-aware +MCP/shell schema menus. + +At baseline, after the pre-edit MCP discovery limit, models could continue calling discovery +tools and receive repeated refusal messages that consume context and workflow +iterations. + +After the first refusal, make the next response concise and route the model +toward a concrete draft. Consider omitting discovery tools from subsequent +tool schemas for that phase if the provider supports dynamic tool lists. + +Expected benefit: faster first edits and fewer wasted iterations. + +### 6. Prefer balanced incremental construction for large tasks + +Status: implemented in verified prompt guidance, focused mutation tools, and +staged draft repair. Large complete writes remain available when appropriate. + +The baseline prompt recommended a small first version but still permitted one large +first write. For parser-heavy tasks, encourage a balanced executable skeleton, +then one complete definition or transformation at a time with verification. + +Expected benefit: syntax errors stay local and model repairs remain small. + +### 7. Complete usage accounting + +Status: implemented and retained. See **Expert Routing and OpenRouter Usage +Accounting** and **Honest Unknown-Cost Reporting**. + +Baseline jcode status files reported zero token usage and no cost for these OpenRouter +runs. Aggregate per-response provider usage into verified status output. + +Expected benefit: reliable quality/speed/cost comparisons. This does not +directly raise pass rate and should follow correctness work. + +## Experiment Log + +| Experiment | Unit tests | Four-model result | Decision | +|---|---|---|---| +| Baseline at `041e6ca` | 999 passed, 1 skipped | 2/4 bundled, 1/4 full spec | Reference | +| Editable rejected drafts | 1005 passed, 1 skipped | 2/4 bundled, 1/4 full spec | Keep: GLM directly used staged repair, improved from 23m10s and 5/6 hidden to 12m19s and 6/6 hidden | +| Retry truncated JSON | 1009 passed, 1 skipped | 1/4 bundled and full spec | Keep: both prior EOF trajectories survived and made substantial progress | +| Explicit 72-turn budget | 1009 passed, 1 skipped | 2/4 bundled, 1/4 full spec | Keep: GLM recovered from the immediately preceding 48-turn failure to a 6/6 hidden pass; two other runs progressed until an unrelated OpenRouter 402 | +| Affordable OpenRouter output cap | 1013 passed, 1 skipped | Cohort blocked by exhausted account balance | Keep: deterministic replay converts the measured affordability 402 from 32,768 to 8,100 tokens and succeeds | +| Structural repair overlap | 1021 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: removes measured false refusals while preserving file/span scope | +| Phase-aware MCP schemas | 1027 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: stale discovery tools disappear from model requests while remaining executable | +| Leading-range full-body normalization | 1032 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: converts Qwen's measured full-file body sent to line 1 into the intended whole-file edit | +| Existing-file rejected-draft staging | 1033 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: invalid rewrites remain locally repairable without changing the original until promotion | +| UUID/schema tool-call recovery | 1044 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: exact Kimi trace replay reaches `provider-chat` as a structured `read` call | +| Protocol-safe verified history compaction | 1052 passed, 1 skipped | Four 64-token probes blocked by account-level 402 | Keep: caps repeated-draft history while preserving the task and complete recent tool/result batches | +| Empty inspection fallback | 1052 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: measured GLM `read()`/`head()` omissions now return a useful directory observation instead of a rejection | +| Expert routing and OpenRouter usage accounting | 1061 passed, 1 skipped | Four installed 64-token probes blocked by account-level 402 | Keep: verified mode now uses the expert-aware provider path and records non-stream usage/cost | +| Expert prompt and Jerboa binding rule | 1063 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: verified models can self-escalate and are warned against the dominant Qwen internal-`def` syntax failure | +| Phase-aware shell schemas | 1066 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: exhausted aliases and aliases between edit/verify disappear from provider requests | +| Verified-only subsystem startup | 1066 passed, 1 skipped | Installed smoke test improved startup to about 0.7s; model call blocked by 402 | Keep: plugins/LSP cannot enter this workflow and no longer add latency or irrelevant load errors | +| Post-pass requirements review | 1069 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: addresses the repeated hidden `2!` miss and prevents shipping edits against stale verification | +| Focused CLI MCP menu | 1072 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: retains measured recipe/syntax/API recovery tools while reducing initial prompt/tool ambiguity | +| Named Kimi tool-call recovery | 1077 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: exact trace form `type=functions.read arguments={...}` becomes a schema-validated call | +| Provider-abort expert escalation | 1079 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: Qwen's measured `finish_reason: error` routes immediately to the enabled expert | +| Targeted post-failure API schemas | 1079 passed, 1 skipped | Live cohort blocked by account-level 402 | Keep: preserves GLM's successful `apropos`/signature recovery without restoring broad discovery | +| Canonical compact inspection tools | 1085 passed, 1 skipped | Installed request reached OpenRouter; inference blocked by account-level 402 | Keep: removes redundant `ls`/`cat`/`head`/`tail`/`wc` schemas and reduces the first request from 29,392 to 22,543 bytes | +| Bounded automatic MCP preflight | 1086 passed, 1 skipped | Exact hard-task envelope reached OpenRouter; inference blocked by account-level 402 | Keep: retains the top two recipes and top anti-pattern while reducing the hard-task first request to 22,315 bytes | +| Internal-definition diagnostic classifier | 1088 passed, 1 skipped | Live inference blocked by account-level 402; exact failure shape covered deterministically | Keep: indented internal `def` errors now recommend local bindings instead of an unrelated delimiter repair | +| `replace_def` structural span enforcement | 1091 passed, 1 skipped | Live inference blocked by account-level 402; refusal/recovery trajectory covered deterministically | Keep: unrelated function replacements can no longer bypass a pending verifier diagnosis | +| Automatic Jerboa runtime read root | 1093 passed, 1 skipped | Live inference blocked by account-level 402; archived Qwen path covered deterministically | Keep: `$JERBOA_HOME/lib` reads no longer consume refusal turns, while writes remain workspace-scoped | +| Trailing zero-argument tool recovery | 1101 passed, 1 skipped | Live inference blocked by account-level 402; exact Kimi response covered through local OpenRouter HTTP | Keep: trailing `verify()` becomes a schema-validated call without heuristic argument parsing | +| Verified prose-drift expert escalation | 1105 passed, 1 skipped | Live inference blocked by account-level 402; exact retry sequence covered deterministically | Keep: a second prose response after the tool-call nudge routes to the enabled expert instead of terminating the workflow | +| Ranged retained-draft replay | 1107 passed, 1 skipped | Exact Kimi `read(start=250,end=290)` trajectory covered deterministically | Confirmed: recovered numeric arguments return the requested draft region rather than the top of the file | +| Requirements-review prose recovery | 1109 passed, 1 skipped | Green-verify/prose/done trajectory covered deterministically | Keep: prose after a successful review is redirected to `done` or corrective edit instead of redundant verification | +| Failed-verifier prose recovery | 1111 passed, 1 skipped | Failed-verify/prose/edit/reverify trajectory covered deterministically | Keep: prose analysis after a failed test now receives a repair action rather than a contradictory repeat-verify recommendation | +| Ordered local syntax repairs | 1113 passed, 1 skipped | Three-error line/exact/line trajectory covered deterministically | Keep: a local fix may persist when the next syntax diagnostic moves beyond the pending span | +| Recoverable `replace_def` name lookup | 1115 passed, 1 skipped | Exact DeepSeek guessed-name failure covered with zero hard-error budget | Keep: missing names list actual top-level definitions and route unbalanced forms to `replace_range` | +| Protocol-preserving expert handoff | 1141 passed, 1 skipped | Installed exact-model probes reached OpenRouter; inference blocked by account-level 402 | Keep: an escalated expert is explicitly told to resume the active structured workflow instead of ending with prose | +| Recoverable phase-gate refusals | 1141 passed, 1 skipped | Installed exact-model probes reached OpenRouter; inference blocked by account-level 402 | Keep: repeated forbidden inspection/shell actions preserve the one genuine verifier-error budget and remain visible to expert escalation | +| Preserve the primary expert note | 1142 passed, 1 skipped | Installed exact-model probes reached OpenRouter; inference blocked by account-level 402 | Keep: the expert receives the primary model's bounded help request without an orphaned assistant tool-call turn | +| Promote 72-turn jcode default | 1143 passed, 1 skipped | Installed banner confirms 72 for all exact-model probes; inference blocked by account-level 402 | Keep: normal jcode use now gets the benchmark-proven repair budget without harness-only configuration | +| Refusal pressure across read-only calls | 1145 passed, 1 skipped | Installed exact-model probes reached OpenRouter; inference blocked by account-level 402 | Keep: archived Qwen-style reject/read/reject loops now reach the expert, while successful mutations reset pressure | +| Bounded single failure advisor | 1149 passed, 1 skipped | Installed exact-model probes reached OpenRouter; inference blocked by account-level 402 | Keep: each failed verify makes one bounded automatic MCP call instead of three; targeted lookups remain model-callable | +| Structured provider failure status | 1153 passed, 1 skipped | All four installed statuses report OpenRouter and HTTP 402 structurally | Keep: benchmark automation no longer parses human error text to distinguish external failures | +| Typed streaming provider failures | 1157 passed, 1 skipped | Direct local streaming 402 retains provider, status, and body; installed exact-model cohort remains account-blocked | Keep: TUI/agent streams no longer collapse upstream HTTP rejection into a generic internal failure | +| Unified streaming protocol recovery | 1161 passed, 1 skipped | Direct streaming `verify()` and UUID+JSON calls recover under advertised schemas; exact-model cohort remains account-blocked | Keep: normal TUI/agent use gets the strict Kimi recovery already proven in verified mode | +| Complete built-in verified help | 1162 passed, 1 skipped | Installed help renders the 72-turn option and requirements review; exact-model cohort remains account-blocked | Keep: users can discover the benchmark-proven budget without consulting separate docs | +| Missing iteration-value guard | 1163 passed, 1 skipped | Incomplete CLI invocation exits locally; standard four-model cohort remains account-blocked | Keep: prevents an incomplete option from becoming task text and making an unintended provider request | +| Complete streaming expert usage | 1166 passed, 1 skipped | Local dual-provider stream records both attempts; standard four-model cohort remains account-blocked | Keep: interactive expert escalation no longer undercounts the discarded primary request | + +### Editable Rejected Drafts + +Run: + +```text +~/mine/kratistos/runs/20260715-040716-life-rle-jcode-four-models-editable-drafts-valid +``` + +| Model | Result | Wall time | Hidden probes | Observation | +|---|---|---:|---:|---| +| DeepSeek V4 Pro | PASS 22/22 | 5m01s | 5/6 | Faster, but this sample accepted illegal `2!` | +| GLM 5.2 | PASS 22/22 | 12m19s | 6/6 | Repaired and promoted the retained missing-file draft | +| Kimi K2.7 Code | FAIL | 12m02s | not run | Reached retained-draft repair, then received truncated JSON | +| Qwen 3.7 Plus | FAIL | 34s | not run | Received whitespace-only HTTP 200 JSON during discovery | + +The bundled and full-spec pass rates are unchanged, but the GLM trajectory is +direct evidence for the feature: it used exact replacement on the retained +draft after the inspection lock, continued repairing the updated staging +buffer, and completed in roughly half the baseline time with better hidden +correctness. The deterministic regression tests cover the same behavior with +`line_edit`, `replace_range`, `replace_def`, and exact replacement. Keep this +change. + +Both failed candidates independently exposed `read-json: unexpected EOF` from +HTTP 200 responses. Qwen's trace contains a whitespace-only response body; +Kimi failed on the first response after inspecting its rejected draft. This is +the measured motivation for the next bounded provider-response retry. + +### Retry Truncated Provider JSON + +Run: + +```text +~/mine/kratistos/runs/20260715-042229-life-rle-jcode-four-models-json-retry +``` + +| Model | Result | Wall time | Hidden probes | Observation | +|---|---|---:|---:|---| +| DeepSeek V4 Pro | PASS 22/22 | 20m10s | 6/6 | Correct but slower stochastic sample | +| GLM 5.2 | FAIL | 9m49s | not run | Hit 48 iterations during changing verifier repairs | +| Kimi K2.7 Code | FAIL | 19m04s | not run | Produced 268 lines; later lost native tool-call formatting | +| Qwen 3.7 Plus | FAIL | 20m52s | not run | Produced 219 lines; hit 48 iterations during active repair | + +The public pass count fell from 2/4 to 1/4 because this is a stochastic model +sample, not because either transport retry caused a semantic failure. The +important controlled observation is that both candidates which previously +terminated on `read-json: unexpected EOF` survived that phase. Kimi progressed +from no artifact to a large implementation and Qwen progressed from a 34-second +transport exit to 20 minutes of drafting and verifier repair. The retry resends +only when the provider call raises both `read-json` and `unexpected EOF`, and +the integration test proves that the serialized request is unchanged. Keep +this change. + +GLM and Qwen both stopped on the fixed 48-iteration ceiling while verifier +failures and source edits were still changing. This repeats Qwen's baseline +failure and makes the iteration-budget experiment the next priority. + +### Explicit 72-Turn Budget + +Run: + +```text +~/mine/kratistos/runs/20260715-045117-life-rle-jcode-four-models-72-iterations +``` + +| Model | Result | Wall time | Hidden probes | Observation | +|---|---|---:|---:|---| +| DeepSeek V4 Pro | PASS 22/22 | 20m15s | 5/6 | Completed, but accepted illegal `2!` | +| GLM 5.2 | PASS 22/22 | 15m27s | 6/6 | Recovered from its 48-turn failure in the prior cohort | +| Kimi K2.7 Code | FAIL | 24m17s | not run | Reached active drafting, then OpenRouter rejected the 32,768-token reservation with HTTP 402 | +| Qwen 3.7 Plus | FAIL | 22m20s | not run | Produced 97 lines, then hit the same OpenRouter reservation 402 | + +The bundled pass rate returned to 2/4 and full-spec compliance remained 1/4. +The option is still a controlled improvement: GLM's immediately preceding run +stopped at the fixed 48-turn ceiling, while the same model with 72 turns +completed and passed every hidden probe. DeepSeek also completed. Neither +remaining trajectory stopped on the iteration ceiling; both exposed a separate +provider integration failure after substantial work. + +At this experiment stage, keep the configurable budget while retaining 48 as +the normal CLI default; the later **Promote the 72-Turn Default** experiment +supersedes that interim decision. Here the benchmark harness opts into 72 +through `KRATISTOS_JCODE_MAX_ITERATIONS`. The next experiment should recover the +OpenRouter 402 response by lowering only the rejected request's output-token +reservation to the affordable value advertised by the server. + +### OpenRouter Credit Recovery And External Blocker + +The 72-turn Kimi and Qwen traces ended with an actionable OpenRouter 402: +the request reserved 32,768 output tokens while the account could afford a +smaller amount. Jcode now recognizes only the OpenRouter error form containing +`can only afford N`, retries the same request with 10 percent headroom, and +limits the adjustment loop to three attempts. A local HTTP integration test +proves a 32,768-token request is retried at 8,100 when the server advertises +9,000, with the prompt unchanged. + +Before a live cohort could validate this behavior, the OpenRouter balance was +fully exhausted. The attempted cohort is recorded at: + +```text +~/mine/kratistos/runs/20260715-051923-life-rle-jcode-four-models-credit-cap-retry +``` + +Every candidate failed before generation with the generic account-level +`402 Insufficient credits`. After the later UUID recovery experiment, direct +64-token probes against all four exact model IDs produced the same response in +under one second. Those calls are external-blocker checks, not benchmark model +failures. No live quality or pass-rate comparison is valid until the account +has enough balance for at least one request. + +### Focused Repair And Schema Experiments + +Four deterministic changes address concrete refusals in the saved traces: + +- Structural repair mode now permits same-file `line_edit`, exact replacement, + `replace_def`, and any overlapping `replace_range`; unrelated or + non-overlapping edits remain blocked. +- MCP schemas are phase-aware. Broad discovery disappears after the pre-edit + allowance, all MCP schemas disappear between a successful edit and verify, + and only targeted syntax/failure helpers remain after a failed verify. +- A complete multi-line Jerboa body sent by Qwen as a leading line-1 range is + normalized to a whole-file replacement. Short line-1 edits remain local. +- Invalid rewrites of existing Jerboa files are retained as staging drafts. + Local edits repair the staged candidate, verification is blocked while it + would test the unchanged original, and promotion occurs only after the + syntax guard accepts the candidate. + +These are retained because each removes a measured false rejection or wasted +loop and has focused regression coverage. Live four-model comparison remains +blocked by the account-level 402. + +### Kimi UUID Tool-Call Recovery + +Kimi's JSON-retry run reached 93,759 prompt tokens and then returned +`finish_reason: stop` with no native tool call. Its final text ended in: + +```text +7395D0C8-4A55-4C5A-BD2B-8F8E6D6D4C0E {"path":"life-rle.ss","start":250,"end":290} +``` + +Jcode now handles this in the shared OpenAI-compatible provider response path, +not only in proxy mode. It parses a complete trailing JSON object, requires a +valid UUID immediately before it, checks required and declared argument keys, +and recovers the call only when exactly one currently advertised tool schema +matches. Empty arguments and path-only arguments that match multiple tools are +left as prose. The exact Kimi response shape is covered through a local HTTP +`provider-chat` integration test and becomes a structured `read` call. + +The prompt-token sequence before failure was `25,334 -> 58,459 -> 85,972`, +followed by smaller turns up to 93,759. This points to repeated full-file draft +arguments as the next target: compact complete old tool exchanges before long +contexts cause protocol drift, while retaining the original task and recent +verifier repair turns. + +### Protocol-Safe Verified History Compaction + +Verified mode previously sent its entire workflow history on every turn and did +not use the normal chat compaction path. Repeated complete draft arguments made +this especially expensive and pushed Kimi into protocol drift near 94k prompt +tokens. + +The workflow runner now accepts a provider-message transformation while keeping +its complete internal history and step state. Verified mode activates a focused +transform at 48,000 estimated history tokens. It preserves the system prompt, +the original task, a short state notice, and the three newest complete +tool-call batches. The boundary always begins on an assistant tool-call message, +so retained calls keep their matching tool results and OpenAI message ordering +remains valid. Exact old source remains available from the workspace or retained +staging draft. + +The implementation also fixed the token estimator to handle association-list +tool arguments, which is the representation used by the workflow runner. Tests +cover the provider-facing hook, task preservation, newest-batch selection, and +tool-call/result ID pairing. This should improve speed, cost, and protocol +stability; live measurement remains blocked by the empty OpenRouter balance. + +### Empty Inspection Fallback + +An earlier GLM run repeatedly sent structured `read()`, `cat()`, `head()`, and +`wc()` calls without `path`. Each call produced another rejection, consuming +turns and context without teaching the model anything new. The schemas still +advertise concrete required paths, but malformed empty inspection calls now +return a successful `list(path=".")` observation with a concise instruction to +select a file next. Empty edit/write calls remain strict and recoverable because +jcode cannot safely invent a target or source body. + +This change is retained because it converts a harmless omission into useful +forward progress while preserving every write and verification guard. + +### Expert Routing, Usage, And Verifier Integrity + +Verified mode previously constructed its provider backend with raw +`provider-chat`. This bypassed `chat-with-expert`, so omitting `--no-expert` did +not actually enable expert escalation for benchmark runs. The verified backend +now injects the expert-aware chat function, includes the self-escalation +instruction in its system prompt, records both primary and expert usage, and +parses OpenRouter's non-streaming token/cache/reasoning/cost object into the CLI +status accumulator. A local OpenRouter-compatible integration test drives this +exact responder path and verifies all usage fields. + +Rapid verifier subprocesses also exposed a native process-status race: one +call could observe the previous child's status. `run-verify-command` now emits +a private random exit marker from the verifier shell and treats that marker as +authoritative, with the native collected status only as a fallback. The full +suite reproduces back-to-back pass/fail verifiers and confirms exact output is +preserved. + +### Trace-Guided Prompt And Tool Phases + +Qwen spent much of its failed 48-turn trajectory repairing `invalid context for +definition` because it repeatedly emitted `def`/`define` inside procedure and +`let-values` bodies. Verified prompts now state the Jerboa top-level binding +rule explicitly, overriding misleading examples that use internal definitions. + +GLM's traces showed two complementary schema requirements. Shell aliases must +disappear once their pre-edit allowance is exhausted and between an edit and +the mandatory verify. Conversely, narrow API tools such as `apropos`, +`function_signature`, `module_exists`, and `module_exports` must remain after a +failed verify because the successful GLM run used them to resolve an unbound +command-line API. The phase filter now makes that distinction. + +The CLI also uses the focused MCP menu by default. It retains recipes, syntax +and compile checks, failure/error lookup, and narrow API lookup while omitting +broad security and workflow discovery schemas. Automatic request-advisor +preflight still runs before the first model request. Verified startup skips LSP +and plugin initialization because those tool origins cannot enter the verified +bridge; an installed-binary smoke test removed the recurring plugin load error +and completed startup plus the blocked provider request in about 0.7 seconds. + +### Post-Pass Specification Review + +Both GLM public-suite passes in the original comparison accepted illegal `2!` +because the 22-check verifier omitted that explicit caller requirement. CLI +verified mode had made `verify` terminal, leaving no turn for a specification +review. It now returns a requirements-review observation after the first pass +and requires `done` afterward. The model is told to compare the implementation +against every caller requirement, especially validation and edge cases the +tests may omit. If it edits during review, jcode invalidates the old pass and +refuses `done` until a fresh verify succeeds. A deterministic trajectory covers +pass, review edit, refused stale completion, reverify, and final completion. + +### Additional Protocol Recovery + +Before Kimi's final UUID-shaped pseudo-call, the saved response used this form: + +```text +工具调用: type=functions.read arguments={"path":"life-rle.ss","start":250,"end":290} +``` + +The shared provider path now recovers this only when the named function is +currently advertised, its argument keys satisfy that function's schema, the +JSON object is complete, and only whitespace follows it. A string-aware brace +scanner prevents the permissive JSON reader from accepting trailing prose. The +same strict scanner now hardens UUID recovery. Separately, OpenRouter's measured +`finish_reason: error` is treated as a provider abort and immediately routes to +the configured expert instead of spending generic prose-retry turns. + +### Current Live Blocker + +After rebuilding and installing these changes, all four exact model IDs were +probed again with `JCODE_MAX_TOKENS=64` and expert mode enabled. Each failed +before inference with the same account-level response: + +```text +HTTP 402: Insufficient credits. Add more using https://openrouter.ai/settings/credits +``` + +This is not the affordable-reservation variant and cannot be repaired by +lowering `max_tokens`. Live pass-rate, speed, and quality deltas remain unknown +until the OpenRouter account is funded; deterministic tests and exact saved +trace replays are the only valid evidence in the meantime. + +### Canonical Compact Inspection Tools + +The archived cohort used `ls`, `cat`, and `tail` only for operations already +covered by `list` and `read`; no trajectory used `head` or `wc`. Advertising all +eight inspection aliases gave the model duplicate choices and charged their +schemas on every request. Compact verified mode now exposes only `read`, +`list`, and `balance`. Non-compact workflows retain every alias for backward +compatibility, and malformed empty inspection calls still use the existing +directory-list fallback. + +Installed-binary traces against the exact Qwen model show the progression: + +| Request | Tools | Tool-schema bytes | Serialized body bytes | +|---|---:|---:|---:| +| Original hard-benchmark binary | 39 | 21,780 | 29,392 | +| Focused MCP only | 26 | 13,503 | 24,566 | +| Focused MCP plus canonical inspection | 21 | 11,523 | 22,543 | + +The final request has 46% fewer tool choices, 47% fewer schema bytes, and a 23% +smaller serialized body than the baseline. The system prompt is intentionally +larger because it contains the later safety, expert, compaction, and +requirements-review guidance. The provider accepted the request envelope and +then returned the same account-level 402 before inference. + +### Bounded Automatic MCP Preflight + +The request advisor originally injected four recipes and five full +anti-pattern entries before every model turn. On a small script request this +made the system prompt 10,642 characters, and several selected anti-patterns +concerned unrelated release-evidence workflows. Jcode now asks for the two +highest-ranked recipes and one anti-pattern. The hard RLE query still selects +CLI-script recipes and `random-paren-pokes`, which directly addresses the +archived structural-repair failures. + +With the complete hard benchmark guidance attached, the installed Qwen request +contains 21 tools, 11,513 bytes of schemas, and a 22,315-byte serialized body. +The original hard-benchmark request was 29,392 bytes, so the final envelope is +24% smaller even after adding expert escalation, binding guidance, automatic +preflight, and post-pass requirements review. The deterministic suite passes +1,086 tests; OpenRouter returned the account-level 402 only after accepting the +request envelope. + +### Internal Definition Versus Swallowed Form + +The verifier advisor previously treated every `invalid context for definition` +error as a balanced function that had swallowed the next top-level definition. +That diagnosis is correct when the offending definition begins in column one, +but Qwen also generated genuinely indented internal `def` forms. In that case +the old response incorrectly prescribed a broad delimiter repair. + +The classifier now inspects the reported source line. An indented `def` or +`define` gets a focused internal-binding diagnosis: use `let`, `let*`, named +`let`, move the helper to top level, or replace the enclosing function. A +column-one definition retains the existing swallowed-form range repair. Both +branches have regression coverage, and the full suite passes 1,088 tests. + +### Definition Replacement Span Enforcement + +The structural repair gate checked line edits, exact replacements, and ranges, +but `replace_def` bypassed the pending diagnosis. A model could therefore edit +an unrelated function while jcode reported that only overlapping repairs were +allowed. Jcode now resolves the selected definition's source line span and +requires that span to overlap the diagnosed range in the diagnosed file. + +A deterministic trajectory first attempts an unrelated function replacement, +asserts that the file remains unchanged, then replaces the implicated function, +reverifies, and completes. The positive overlap cases remain valid and the full +suite passes 1,091 tests. + +### Jerboa Runtime Read Root + +Qwen's 72-turn trace requested both a directory listing and a concrete source +file under `$JERBOA_HOME/lib`; jcode refused both even though the benchmark +explicitly allowed standard modules from that runtime. Verified mode now adds +an existing `$JERBOA_HOME/lib` to its read-only roots and advertises it in the +prompt. `JCODE_READ_ROOTS` continues to add caller-selected roots. + +This changes only explicit `read`/`list` access. Edit tools still apply the +workspace write-scope and reject the same external paths. Regression tests +cover the automatic read and the unchanged write refusal; the full suite passes +1,093 tests. + +### Zero-Argument Protocol Recovery + +Kimi emitted a literal trailing `verify()` after losing native tool-call +formatting. Jcode now recovers that form only when the name exactly matches one +currently advertised tool whose schema has no properties and no required +arguments. Calls such as `read()`, unknown names, and calls followed by prose +remain text and continue through the normal retry/escalation path. + +Tests cover the pure parser constraints and the complete local +OpenRouter-compatible `provider-chat` path using Kimi's measured response form. +The full suite passes 1,101 tests. + +### Prose-Drift Expert Escalation + +After Kimi lost native tool formatting, several prose-only pseudo-calls reached +the workflow retry limit and ended the run. Verified mode still gives the +primary model one normal structured-call correction. If the next response is +also prose-only, the expert escalation layer now recognizes the verified retry +marker and routes that turn to the configured expert. Ordinary chat prose and +the first recoverable miss do not trigger this signal, and `--no-expert` +continues to disable every expert route. Only immediately trailing retry pairs +count, so an old nudge cannot escalate later prose after useful tool progress. +The full suite passes 1,105 tests. + +### Retained Draft Range Verification + +Kimi reported that repeated reads near line 272 returned the top of its rejected +draft. The current staged-draft reader already applies `start`/`end`; the lost +range came from prose-shaped calls that never delivered structured arguments. +A 260-line trajectory now combines a retained syntax-invalid draft with Kimi's +exact `start=250,end=290` request and asserts that the result includes the +target region and excludes the file header. Together with named/UUID recovery, +this locks the complete repair path. The full suite passes 1,107 tests. + +### Requirements Review Completion Nudge + +After a green verify, a weak model may describe its requirements review in +prose rather than call `done`. The old retry logic saw that an edit had occurred +earlier and incorrectly recommended another `verify()`. The retry message now +examines the latest tool result: a successful `REQUIREMENTS REVIEW` observation +recommends `done(summary=...)` when no gap remains, or an edit followed by +verification when one does. If an edit is actually the latest result, the +normal verify recommendation remains. A complete deterministic trajectory +passes, bringing the suite to 1,109 tests. + +The inverse phase is also covered. When the latest tool result says +`verification failed`, the retry now recommends a concrete +`edit`/`line_edit`/`replace_def`/`replace_range` repair followed by verification +and includes the balance hint for structural errors. A deterministic +failed-verify, prose, edit, reverify, done trajectory passes. The suite reaches +1,111 tests. The generic footer is phase-aware as well: it no longer tells a +model to rerun an unchanged failure or redundantly reverify after a green +requirements review. + +### Ordered Local Syntax Repairs + +`replace_range` already allowed a diagnosed repair to persist when it exposed a +later independent reader error, but `line_edit` and exact replacement still +required the entire file to become valid in one operation. The local repair +guard now accepts the edit only when the new syntax diagnostic moves strictly +after the pending diagnosed span in the same file. The edit is then verified, +which establishes the next authoritative repair target. + +A three-error fixture repairs line 3 with `line_edit`, exposes line 5, repairs +that definition by exact replacement, exposes line 7, and completes with a +final line edit and passing verifier. Diagnostics that remain at or before the +pending span are still rejected. The suite passes 1,113 tests. + +### Definition Name Recovery + +DeepSeek incurred hard tool errors after guessing `replace_def` targets named +`fail` and `remove-spaces` that were absent from its current draft. A missing +name now returns a recoverable result listing up to 20 actual top-level +definitions. It asks the model to retry an exact listed name or use a ranged +repair when the intended form is unbalanced; an unbalanced named definition is +recoverable for the same reason. + +A zero-hard-error-budget trajectory guesses `fail`, receives +`actual-failure, main`, retries the real name, verifies, and completes. The +suite passes 1,115 tests. + +### Focused Post-Pass Tool Menu + +The workflow previously represented a green verifier only through the absence +of failure/edit state. During the required final requirements review, that +caused broad MCP discovery schemas to reappear. A model that had already built +and passed the artifact could therefore leave the completion path for another +research loop. + +Jcode now tracks a distinct requirements-review-pending phase. In that phase it +keeps local inspection, edit, verify, and `done` actions, but omits every MCP +discovery schema from the provider request. Any edit clears the review phase +and makes a fresh verify authoritative; a subsequent pass enters a new review. +An end-to-end responder test captures the actual post-pass schema list and +asserts both sides of this contract. The full suite passes 1,117 tests. + +The installed revision was then probed against all four exact model IDs with +expert routing enabled. GLM 5.2, DeepSeek V4 Pro, Kimi K2.7 Code, and Qwen 3.7 +Plus each reached OpenRouter and failed before inference with the same +account-level `402 Insufficient credits`. The deterministic phase test is the +available quality evidence; no live pass-rate claim is made. + +### Verify-Focused Post-Edit Menu + +In Qwen's 72-turn trace, a successful whole-file structural repair returned +`Call verify now`, but the following request still advertised inspection, +discovery, scaffold, shell, and `done` schemas. Qwen chose `cat` and `read` +instead of the authoritative verifier and then encountered the provider credit +limit. This is a workflow detour independent of the later provider failure. + +Once an edit succeeds, jcode now advertises exactly the five edit/repair tools +plus `verify`. This still lets a model correct a mistake it notices immediately, +but removes stale observations and a premature finish action. Hidden calls stay +executable, so a provider replaying an already-generated call gets the existing +recoverable guidance rather than a hard unknown-tool failure. The actual schema +list is asserted end to end and the full suite passes 1,118 tests. + +The installed revision again reached each of the four exact OpenRouter model +IDs with expert routing enabled. Every endpoint returned the account-level +`402 Insufficient credits` before inference, so this experiment is retained on +the deterministic workflow and archived-trace evidence, not a live score delta. + +### Canonical Requirements-Review Menu + +Removing MCP discovery did not fully focus the new post-pass phase: non-compact +inspection aliases, the script scaffold, and enabled shell aliases could still +be advertised. The review filter now has an explicit allowlist: canonical +`read`, `list`, and `balance`; edit/repair tools; `verify`; and `done`. This +prevents a green run from reopening construction or duplicate-observation paths +while preserving everything needed to find and fix a missing requirement. + +The end-to-end requirements-review test now asserts the positive and negative +schema sets. The full suite remains at 1,118 passing tests, zero failures, and +one environment skip. After installation, all four exact OpenRouter probes +again failed before inference with the account-level credit response. + +### Hard Workflow Error Escalation + +Recoverable guard refusals correctly bypass the hard-error budget, and the +identical-call detector can already route a loop to the configured expert. +However, the separate tool-error-streak detector recognized only results +beginning with plain `Error` or `MCP Error`. Verified workflows emit genuine +failures as `[ToolError]`, `[StepEnforcementError]`, and +`[PrerequisiteError]`, so the detector was blind to its own canonical format. + +Those three hard wrappers now count toward expert escalation. Privileged +`[ToolRecoverableError]` and `[ToolResolutionError]` messages deliberately do +not count, preserving normal model self-repair. Regression tests cover both +the hard and recoverable sequences; the suite passes 1,120 tests. The installed +four-model probe was accepted through dispatch for every exact model and then +blocked before inference by the same account-level OpenRouter 402. + +### Isolated Best-of Attempts + +Verified state was parameterized once around the entire `best-of > 1` loop. +Consequently a failed candidate could leave its rejected draft, pending repair, +diagnostic range, counters, or review phase visible to the next supposedly +fresh sample. Several state fields added during this benchmark were also absent +from the top-level reset list. + +Jcode now resets every verified attempt parameter when the responder factory is +entered, which happens once per best-of candidate, while retaining dynamic +scope restoration after the run. A regression makes candidate one leave an +invalid missing-file draft; candidate two first lists the workspace, proves no +pending-repair message leaked, writes valid source, and verifies. The suite +passes 1,122 tests. Best-of-one benchmark behavior is unchanged, and the +installed four-model probe remains blocked before inference by account credit. + +### Recoverable Exact-Replacement Misses + +An exact replacement whose `old_str` was absent returned useful guidance as a +normal successful edit result. The file did not change, but the workflow +recorded the edit step and reset its hard-error streak. Both on-disk and staged +draft misses now raise the privileged recoverable condition: the model still +receives the same `read`/`replace_range` repair route, the event consumes no +hard-error budget, and it is no longer represented as a completed mutation. + +The existing zero-hard-error-budget trajectory now asserts the recoverable +wrapper, repairs the file with `replace_range`, verifies, and finishes. The +full suite remains at 1,127 passing tests. The installed binary again reached +all four exact OpenRouter model routes with expert mode enabled; every request +was blocked before inference by the account-level 402. + +### Honest No-Op Mutation Results + +Several structural focus gates returned repair instructions as ordinary tool +successes even though they refused the requested mutation. Jcode could then +record `line_edit`, `replace_def`, `replace_range`, or `edit` as completed and +reset an unrelated hard-error streak without changing either the staging draft +or workspace. + +Blocked mutation calls now raise the privileged recoverable condition. They +remain free of the hard-error budget and preserve their focused repair text, +but are no longer represented as successful edits. An end-to-end structural +trajectory attempts unrelated line, definition, and range mutations, asserts +all three recoverable wrappers, applies the correct overlapping repair, and +passes verification. The suite remains at 1,127 passes. All four installed +model probes again reached OpenRouter and stopped on the account-level 402 +before inference. + +### Recoverable Range Content Omission + +`replace_range` with a valid path and range but no replacement body returned a +normal result saying that it did nothing. It now returns the same exact +corrected call shape as a recoverable tool error, so no mutation is recorded +and no hard-error budget is consumed. Its end-to-end test now runs with a zero +hard-error budget, follows the correction, preserves surrounding source, and +verifies. The full suite remains green at 1,127 passes. Installed probes for all +four exact models were again blocked before inference by the account-level +OpenRouter 402. + +### Recoverable Local-Mutation Schemas + +Empty or incomplete `line_edit`, `replace_def`, and `replace_range` calls used +to consume the hard execution-error budget, unlike an equally malformed +`edit()` call. These local mutation tools now return recoverable guidance with +one exact complete call shape. The repeated-call breaker still terminates an +unchanged malformed loop, while a first omission does not make the model give +up early. + +A zero-hard-error-budget trajectory sends all three malformed calls, follows +with a valid edit, verifies, and finishes. The suite passes 1,129 tests, zero +failures, and one environment skip. The installed four-model probe again +reached all exact OpenRouter routes and was blocked before inference by the +account-level 402. + +### Reject Unchanged Mutations + +Byte-identical full-file, line, range, and exact replacements were previously +written and recorded as successful edits. That manufactured progress, cleared +failure state, and encouraged another verify even though the program had not +changed. On-disk unchanged mutations now return a recoverable result and leave +the previous verifier state authoritative. Invalid staging drafts keep their +existing syntax-repair behavior. + +The zero-hard-error-budget regression makes three different unchanged edits, +then applies a real exact replacement and verifies. The suite passes 1,131 +tests, zero failures, and one environment skip. Installed probes against all