Use MCP advisor guidance in verified runs
ober
4a76ccd2efc4ff3af491c3e2eb74b473922fbd52
new file mode 100644 --- /dev/null +++ b/docs/verified.md @@ -0,0 +1,543 @@ +# Verify-Gated Runs + +`jcode verified` is the non-interactive mode for tasks where "the model said it +worked" is not good enough. It forces the agent through a small coding workflow: + +```text +read context -> edit files -> run a real verify command -> repair on failure +``` + +The run succeeds only after the configured verify command exits successfully. +If verification fails, the failure output is returned to the model as a tool +result and the model must repair the code and verify again. + +This mode is especially useful for local or small models. They may still make +poor first attempts, but the workflow keeps them tied to concrete compiler, +test, or runtime output. + +## Quick Start + +```bash +jcode verified "fix the parser test" \ + --cwd ~/mine/project \ + --verify "make test" +``` + +For a Jerboa task: + +```bash +jcode verified "write a text Conway's Game of Life in Jerboa" \ + --cwd ~/mine/scratch \ + --verify "scheme life.ss" \ + --write-scope "life.ss" +``` + +For automation: + +```bash +jcode verified "$TASK" \ + --cwd "$REPO" \ + --verify "make test" \ + --json \ + --status-file /tmp/jcode-verified-status.json +``` + +With `--json`, the final status object is written to stdout and the human +trajectory moves to stderr. `--status-file` writes the same status object to a +file in both human and JSON modes. + +## Mental Model + +Normal chat is advisory: the model can read files, edit files, and then claim it +is done. `jcode verified` makes completion conditional on execution: + +```text +1. The model reads the files it needs. +2. The model edits with the verified workflow's edit tools. +3. The model calls verify. +4. jcode runs your verify command in the requested working directory. +5. If the command fails, jcode returns the error text to the model. +6. The model repairs and verifies again. +7. The run ends only when verify passes. +``` + +The terminal success condition is not the `done` tool in CLI mode; it is a +passing `verify`. Internally, the workflow may still have a `done` tool for +non-terminal variants, but the live `jcode verified` command uses verify as the +completion gate. + +## Command Surface + +```bash +jcode verified <task> [options] +``` + +| Option | Default | Meaning | +|---|---:|---| +| `--verify CMD` | `make build` | Shell command that must pass. Run from `--cwd`. | +| `--cwd DIR` | `.` | Working directory for file access and verification. | +| `--write-scope PATHS` | `all` | Writable paths: `all`, `none`, or comma-separated prefixes/files. Alias: `--scope`. | +| `--bestof K` | `1` | Try up to `K` trajectories; a passing verified trajectory wins. | +| `--guidance-file FILE` | none | Append external task guidance to the workflow prompt. Aliases: `--guide-file`, `--task-guidance`. | +| `--json` | off | Emit only final status JSON on stdout; human trace goes to stderr. | +| `--status-file FILE` | none | Write final status JSON to a file. | +| `--no-run-aliases` | off | Omit `run`, `bash`, and `shell` inspection aliases from the workflow. | + +Global flags still apply before the subcommand: + +```bash +jcode --provider mlx2 --model /Users/user/models/qwen3-coder-next-mlx \ + verified "fix the failing test" --verify "make test" +``` + +Use `--no-mcp` as a global flag if you want to skip MCP startup: + +```bash +jcode --no-mcp verified "fix the failing test" --verify "make test" +``` + +## Tools Available To The Model + +The verified workflow is intentionally narrower than the normal chat toolbox. +It gives the model enough power to inspect and repair code without turning the +run into arbitrary shell automation. + +Built-in inspection tools: + +```text +read(path) +list(path) +ls(path) +cat(path) +head(path, lines?) +tail(path, lines?) +wc(path) +balance(path) +``` + +Edit tools: + +```text +edit(path, content) +edit(path, old_str, new_str) +write(path, content) +line_edit(path, line, content) +replace_def(path, name, content) +replace_range(path, start, end, content) +``` + +Verification and completion: + +```text +verify() +done(summary) +``` + +By default, the workflow also includes narrow shell-habit aliases: + +```text +run(command) +bash(command) +shell(command) +``` + +These aliases are not general-purpose shell access. They cover simple +inspection habits such as `ls`, `cat`, `head`, `tail`, `wc`, `grep`, `rg`, plain +file reads, directory listings, and `mkdir -p` inside the write scope. The real +build or test command must go through `verify`. + +Use `--no-run-aliases` when a local model keeps wasting turns trying shell +commands instead of reading, editing, and verifying. + +## Write Scope + +`--write-scope` limits where the verified workflow can write. This is useful +when a task should only touch a small part of a repository. + +```bash +jcode verified "fix the parser regression" \ + --verify "make test" \ + --write-scope "src/parser/,test/parser/" +``` + +Common values: + +| Value | Meaning | +|---|---| +| `all` | Allow writes anywhere under `--cwd`. This is the default. | +| `none` | Read-only. Useful for diagnosis runs. | +| `src/foo.ss` | Allow exactly that file. | +| `src/foo/` | Allow that directory and its children. | +| `src/,test/` | Allow multiple prefixes. | + +The verify command itself is still a shell command. Write scope controls the +verified workflow's edit tools; it is not an operating-system sandbox for the +verify command. + +## Choosing A Verify Command + +The verify command is the oracle. A weak verify command produces weak results. + +Good verify commands are specific enough to catch the task: + +```bash +--verify "make test" +--verify "make build" +--verify "scheme life.ss" +--verify "jerbuild compile --libdirs ./lib src/my-module.ss" +``` + +For a one-file script, choose a command that both loads the file and exercises +the expected behavior: + +```bash +--verify "scheme life.ss | grep -q 'Generation 3'" +``` + +For a repo change, prefer the smallest command that is still meaningful. A fast +target gives the model more repair cycles before it runs out of iterations. + +Avoid verify commands that can pass before the requested behavior exists. For +example, `scheme life.ss` is enough for "file must load", but not enough for +"prints a correct Conway's Game of Life board" unless the script itself checks +or demonstrates that behavior. + +## JSON Status + +With `--json`, stdout contains a single JSON object. Human trace lines are +written to stderr. + +Example: + +```bash +jcode verified "fix parser" --verify "make test" --json +``` + +Representative success shape: + +```json +{ + "ok": true, + "status": "passed", + "verify_passed": true, + "exit_code": 0, + "exit_reason": "verified", + "error_type": false, + "error": false, + "summary": "VERIFIED: exit 0", + "task": "fix parser", + "provider": "mlx2", + "model": "/Users/user/models/qwen3-coder-next-mlx", + "best_of": 1, + "verify_command": "make test", + "cwd": ".", + "write_scope": "all", + "run_aliases": true, + "guidance_file": false, + "status_file": false +} +``` + +On failure, `ok` is false, `exit_code` is `1`, and `error_type` describes the +runner stop condition when available. + +Common failure types include: + +| Error type | Meaning | +|---|---| +| `max_iterations` | The model used all workflow turns without a verified pass. | +| `tool_execution` | Tool failures exceeded the error budget. | +| `step_enforcement` | The model repeatedly tried to finish before required steps. | +| `no_progress` | The model repeated the same tool calls without making progress. | + +## Cookbook-Guided Runs + +`jcode verified` can be much stronger when a local model receives concise, +task-specific cookbook guidance before it starts. For Jerboa tasks, that +guidance should come from `jerboa-mcp`, because the MCP server owns Jerboa +language facts, recipes, syntax checks, and repair patterns. + +There are four current integration paths: + +1. Automatic preflight: for Jerboa-looking tasks, `jcode verified` calls a + registered MCP `jerboa_request_advisor` before the first model turn. +2. Failure repair: failed verifier output is augmented with + `jerboa_failure_advisor` when that MCP tool is available. +3. Manual guidance: pre-generate a cookbook/task bundle and pass it with + `--guidance-file`. +4. Let the model call MCP tools during the verified workflow. + +The automatic path is preferred for weak local models because they do not need +to discover the right MCP tool on their own. `--guidance-file` still works and +is appended after automatic guidance when both are present. + +### Guidance File Flow + +Use an MCP client to call a cookbook bundle tool such as: + +```text +jerboa_cookbook_task_bundle({ + "task": "write a text Conway's Game of Life in Jerboa", + "project_path": "/Users/user/mine/scratch", + "file_path": "life.ss", + "max_recipes": 4, + "include_code": true +}) +``` + +Write the returned compact guidance to a file, then run: + +```bash +jcode verified "write a text Conway's Game of Life in Jerboa" \ + --cwd /Users/user/mine/scratch \ + --verify "scheme life.ss" \ + --write-scope "life.ss" \ + --guidance-file /tmp/jerboa-life-guidance.md +``` + +The guidance text is injected into the verified workflow prompt after any +automatic MCP preflight guidance. The model sees both before it starts reading +and editing. + +### MCP Tools During Verified Runs + +When MCP is initialized, `jcode verified` exposes registered MCP-origin tools as +structured workflow tools. This means a model can call Jerboa MCP helpers during +the run, for example to: + +- fetch cookbook recipes, +- check syntax, +- check delimiter balance, +- run Jerboa-specific verification helpers, +- inspect exported symbols or module availability. + +The exact tool names depend on the MCP server prefix configured in `jcode.json`. +For example, one setup might expose tools under `mcp_jerboa_...`; another might +use `jerboa_...`. + +This is useful, but it should not be the only plan for small local models. A +weak model may not decide to call the cookbook tool at the right time. For hard +Jerboa tasks, rely on the automatic preflight and add `--guidance-file` when +you have extra task-specific context that the MCP advisor cannot infer. + +### What Belongs In A Cookbook Bundle + +Good verified-run guidance is short and operational. It should say what to do +when the model hits a known failure mode. + +Useful cookbook content: + +- minimal working imports, +- known-good file/script shapes, +- exact verifier commands, +- common compiler errors and the likely repair, +- API signatures that are easy to hallucinate, +- anti-patterns that look plausible but fail, +- examples small enough to copy safely. + +Less useful content: + +- long prose tutorials, +- broad language reference dumps, +- unrelated recipes, +- multiple competing ways to solve the same tiny task, +- huge examples that consume the context window. + +A practical guidance file shape: + +````markdown +# Task Guidance: Text Conway's Game of Life in Jerboa + +## Use This Shape + +- Put the whole script in `life.ss`. +- Use `(import (jerboa prelude))`. +- Keep state as lists/vectors; avoid relying on unimported SRFI helpers. +- Provide a deterministic demo from a fixed seed board. + +## Verify + +Run: + +```bash +scheme life.ss +``` + +The script should exit 0 and print at least three generations. + +## Anti-Patterns + +- Do not use imports that are not available in this project. +- Do not leave top-level test expressions that depend on undefined variables. +- Do not repair unbalanced code with tiny `old_str` edits; use + `replace_range` after checking the broken span. + +## Common Failure Repairs + +- `unexpected close parenthesis`: call `balance(path)` or MCP balance helper, + then replace the whole broken span. +- `unbound identifier`: either define it locally or remove the dependency. +```` + +Keep guidance specific to the task. For local models, a small high-signal file +often works better than a comprehensive language manual. + +## Recommended Jerboa Workflow + +For Jerboa coding tasks, use this pattern: + +```text +1. Ask jerboa-mcp for a task bundle or relevant recipes. +2. Save the bundle as a guidance file. +3. Run jcode verified with: + - a narrow write scope, + - a real Jerboa verify command, + - the guidance file, + - MCP enabled. +4. Let the model repair against actual Scheme/Jerboa errors. +5. If it fails, improve the cookbook with the failure pattern you saw. +``` + +Example: + +```bash +jcode --provider mlx2 \ + --model /Users/user/models/qwen3-coder-next-mlx \ + verified "write a text Conway's Game of Life in Jerboa" \ + --cwd /Users/user/mine/scratch \ + --verify "scheme life.ss" \ + --write-scope "life.ss" \ + --guidance-file /tmp/jerboa-life-guidance.md \ + --json \ + --status-file /tmp/jerboa-life-status.json +``` + +## Best-Of-K + +`--bestof K` asks the runner to try multiple trajectories. The verify gate is +the selector: a candidate that cannot pass verification is rejected. + +```bash +jcode verified "fix the failing parser test" \ + --verify "make test" \ + --bestof 3 +``` + +Use this when first attempts vary a lot. It costs more tokens and time, but it +can help local models when the task has several plausible starts and only one +gets into a repairable path. + +Use a fast verify command with `--bestof`. Slow verification multiplied by `K` +can become expensive quickly. + +## Interactive Equivalent + +Inside the REPL or TUI, use `/forge run`: + +```text +/forge run --verify "make test" --write-scope "src/,test/" fix parser +``` + +It uses the same verified runner as `jcode verified`. The subcommand is better +for harnesses and scripted experiments; `/forge run` is convenient when you are +already in a session. + +## The `verified` Tool + +The normal agent also has a `verified` tool. Skills and sub-agents can call it +to launch a scoped verified sub-run from inside a larger conversation. + +Use cases: + +- a planning agent delegates a bounded implementation rung, +- a skill wants a hard edit/test gate for one file, +- a local model needs a smaller verified task rather than a broad repo task. + +The same rule applies: the sub-run is only successful after its verify command +passes. + +## Safety Notes + +`jcode verified` is a reliability tool, not a full sandbox. + +- `--write-scope` limits jcode's verified edit tools. +- The `verify` command is a shell command you choose. +- MCP tools come from configured MCP servers; use trusted servers. +- If an MCP server exposes tools with side effects, those side effects are the + server's responsibility. jcode only imports tools marked as MCP-origin into + the verified workflow. +- Use `--no-mcp` when you want a run with no external MCP tools. +- Use `--no-run-aliases` when a model keeps trying shell habits instead of the + verified edit/verify loop. + +For high-risk repositories, combine: + +```bash +--write-scope "specific/file.ss,specific/tests/" +--verify "make targeted-test" +--json +--status-file /tmp/status.json +``` + +and inspect the diff before committing. + +## Troubleshooting + +### The model keeps inspecting instead of fixing + +The verified workflow has inspection-pressure guards. After failed verifies or +rejected drafts, repeated reads eventually become tool errors that tell the +model to make a concrete repair. + +If this still loops: + +- narrow the task, +- add a better `--guidance-file`, +- use `--no-run-aliases`, +- increase cookbook anti-pattern coverage for the failure you saw. + +### The run passes but the behavior is wrong + +Your verify command was too weak. Strengthen it so the requested behavior is +actually checked. + +### The model cannot edit the file + +Check `--write-scope`. A file path scope permits exactly that file; a directory +scope should end with `/` if you want children. + +### MCP tools are missing + +Check: + +- you did not pass global `--no-mcp`, +- the MCP server is configured in `jcode.json`, +- the MCP server starts successfully, +- the tool was registered as MCP-origin by jcode's MCP client. + +Tool prefixes are configurable, so do not assume every MCP tool starts with +`mcp_`. + +### JSON output contains no trace + +That is intentional. In `--json` mode stdout is reserved for the final machine +status. Read stderr for the human trajectory, or use `--status-file` for the +machine result. + +## Implementation Map + +Key modules: + +| Module | Responsibility | +|---|---| +| `src/jcode/ui/cli.ss` | Parses `jcode verified`, prints the trajectory, emits status JSON. | +| `src/jcode/core/verified-run.ss` | Builds the verified coding workflow and runs provider-backed verification. | +| `src/jcode/core/verified.ss` | Defines the verify callable and verified workflow primitives. | +| `src/jcode/core/workflow-runner.ss` | Executes workflow turns, enforces steps, tracks tool errors, and stops loops. | +| `src/jcode/mcp/client.ss` | Starts MCP servers, registers MCP tools, tags them as MCP-origin. | +| `src/jcode/tool/registry.ss` | Stores tool metadata and schemas used by normal and verified runs. | + +The important design boundary is that Jerboa expertise belongs in +`jerboa-mcp` cookbooks and validators. `jcode verified` should retrieve or +receive the relevant guidance, expose MCP tools, and enforce the edit/verify +loop. --- a/src/jcode/core/verified-run.ss +++ b/src/jcode/core/verified-run.ss @@ -34,6 +34,111 @@ (def (opt-get o key) (let ((p (assoc key o))) (and p (cdr p)))) +(def (hash-args . kvs) + (let ((h (make-hash-table))) + (let loop ((xs kvs)) + (unless (null? xs) + (let ((key (car xs)) + (value (cadr xs))) + (when (and value + (or (not (string? value)) + (> (string-length value) 0))) + (hash-put! h key value))) + (loop (cddr xs)))) + h)) + +(def (string-has-any? text needles) + (let ((lower (string-downcase (or text "")))) + (let loop ((xs needles)) + (cond + ((null? xs) #f) + ((string-contains lower (car xs)) #t) + (else (loop (cdr xs))))))) + +(def (mcp-tool-name-by-suffix suffix) + (or (and (tool-exists? suffix) + (eq? (tool-origin suffix) 'mcp) + suffix) + (let loop ((names (list-tools))) + (cond + ((null? names) #f) + ((and (eq? (tool-origin (car names)) 'mcp) + (string-suffix? suffix (car names))) + (car names)) + (else (loop (cdr names))))))) + +(def (usable-mcp-guidance? text) + (and (string? text) + (> (string-length (string-trim text)) 0) + (not (string-prefix? "Unknown tool" text)) + (not (string-prefix? "Error executing" text)) + (not (string-prefix? "MCP Error" text)))) + +(def (safe-mcp-tool-call suffix args) + (let ((name (mcp-tool-name-by-suffix suffix))) + (and name + (let ((result (try (tool-execute name args) + (catch (_) #f)))) + (and (usable-mcp-guidance? result) result))))) + +(def (scope-primary-file scope) + (and (list? scope) + (= (length scope) 1) + (let ((path (car scope))) + (and (string? path) + (or (string-suffix? ".ss" path) + (string-suffix? ".sls" path) + (string-suffix? ".scm" path) + (string-suffix? ".md" path)) + path)))) + +(def (jerboa-project-cwd? cwd) + (or (file-exists? (string-append cwd "/jerboa.pkg")) + (file-exists? (string-append cwd "/mcp/server.ss")) + (file-exists? (string-append cwd "/lib/jerboa/prelude.ss")))) + +(def (jerboa-looking-task? task verify-cmd cwd scope) + (or (jerboa-project-cwd? cwd) + (scope-primary-file scope) + (string-has-any? + (string-append task " " verify-cmd) + '("jerboa" ".ss" ".sls" ".scm" "jmcp" "jerbuild" "scheme" "chez")))) + +(def (verified-preflight-guidance task cwd verify-cmd scope) + (and (jerboa-looking-task? task verify-cmd cwd scope) + (safe-mcp-tool-call + "jerboa_request_advisor" + (hash-args "task" task + "project_path" cwd + "file_path" (scope-primary-file scope) + "max_recipes" 4 + "max_anti_patterns" 5)))) + +(def (combine-task-guidance auto caller) + (let ((parts '())) + (when (usable-mcp-guidance? auto) + (set! parts + (cons (string-append + "Automatic Jerboa MCP preflight guidance:\n" + auto + "\nEnd automatic Jerboa MCP preflight guidance.") + parts))) + (when (usable-mcp-guidance? caller) + (set! parts + (cons (string-append + "Caller-provided guidance:\n" + caller + "\nEnd caller-provided guidance.") + parts))) + (and (pair? parts) + (string-join (reverse parts) "\n\n")))) + +(def (mcp-failure-guidance detail cwd) + (safe-mcp-tool-call + "jerboa_failure_advisor" + (hash-args "verify_output" detail + "project_path" cwd))) + ;; ── provider-backed responder ───────────────────────────────────────── ;; make-provider-backend gives the (messages tool-specs sampling) seam; the ;; runner wants (messages tool-specs step-index). Ignore the step index and pass @@ -276,12 +381,16 @@ (best-repair-candidate-text cwd repair label)))))) (def (augment-verify-detail detail cwd) - (let ((diagnosis (or (invalid-context-diagnosis detail cwd) - (invalid-syntax-diagnosis detail cwd) - (unexpected-close-diagnosis detail cwd)))) - (if diagnosis - (string-append detail diagnosis) - detail))) + (let* ((diagnosis (or (invalid-context-diagnosis detail cwd) + (invalid-syntax-diagnosis detail cwd) + (unexpected-close-diagnosis detail cwd))) + (base (if diagnosis + (string-append detail diagnosis) + detail)) + (mcp-advice (mcp-failure-guidance base cwd))) + (if (usable-mcp-guidance? mcp-advice) + (string-append base "\n\nMCP failure advisor:\n" mcp-advice) + base))) (def (verify-output-forced-failure? out) (or (string-contains out "invalid context for definition") @@ -2429,12 +2538,16 @@ (cwd (or (opt-get o 'cwd) ".")) (k (or (opt-get o 'best-of) 1)) (scope (parse-write-scope (opt-get o 'write-scope))) + (task-guidance + (combine-task-guidance + (verified-preflight-guidance task cwd vcmd scope) + (opt-get o 'task-guidance))) (wf (coding-workflow vcmd cwd (list (cons 'write-scope scope) (cons 'run-aliases? (let ((p (assoc 'run-aliases? o))) (if p (cdr p) #t))) - (cons 'task-guidance (opt-get o 'task-guidance)) + (cons 'task-guidance task-guidance) (cons 'terminal-on-verify #t)))) (ropt (list (cons 'max-iterations (or (opt-get o 'max-iterations) 48)) (cons 'max-repeated-calls (or (opt-get o 'max-repeated-calls) 6)) --- a/test/run.ss +++ b/test/run.ss @@ -1966,6 +1966,20 @@ (and (str-contains? s "External MCP tools") (str-contains? s "jerboa_test_lookup"))))) +(register-tool! "jerboa_failure_advisor" + "Fake MCP failure advisor for verified-run tests." + '(("type" . "object")) + (lambda (a) "MCP-FAILURE-ADVICE")) +(set-tool-origin! "jerboa_failure_advisor" 'mcp) +(let ([result (run-verify-command "sh -c 'echo mcp-check; exit 1'" "/tmp")]) + (check! "verified-run: MCP failure advisor keeps verify failing" + (car result) #f) + (check-pred! "verified-run: MCP failure advisor augments detail" + (cdr result) + (lambda (s) + (and (str-contains? s "MCP failure advisor") + (str-contains? s "MCP-FAILURE-ADVICE"))))) + (let* ([wf (coding-workflow "true" "/tmp" (list (cons 'task-guidance "recipe-context-marker")))]) (check-pred! "verified-run: caller guidance appears in prompt"