Update Jerboa guidance data
ober
90490f35b0077c27045aeb8f9899ef7ed7163e2f
--- a/data/anti-patterns.sexp +++ b/data/anti-patterns.sexp @@ -839,4 +839,47 @@ ("tools" "jerboa_check_balance" "jerboa_verify" - "jerboa_run_tests"))) + "jerboa_run_tests")) + (("advice" + . + "Use at most one or two targeted MCP calls to remove concrete language/API uncertainty, then write the first complete draft with edit/write and run the configured verifier. After a verify failure, use the verifier output as the repair target and only call MCP for a specific syntax/API question.") + ("avoid" + . + "Do not spend repeated pre-edit turns on MCP, cookbook, or API discovery when the task is a small greenfield script or file creation. Local models can loop in discovery and never create the required file.") + ("id" . "pre-edit-mcp-discovery-loop") + ("kinds" "all" "script" "test") + ("pattern" + . + "pre-edit MCP discovery|mcp_.*howto|mcp_.*module_exports|mcp_.*request_advisor") + ("severity" . "high") + ("tags" "mcp" "verified" "greenfield" "local-model" + "tool-routing" "first-edit") + ("title" + . + "Pre-edit MCP discovery loop on greenfield tasks") + ("tools" + "jerboa_request_advisor" + "jerboa_howto" + "verify" + "jerboa_failure_advisor")) + (("advice" + . + "Keep a small invalid-tool-call budget for local-model verified runs. If the model emits prose after a required-tool-call correction, shrink context and force an immediate edit/write, or stop the benchmark and switch to a provider with stronger tool-call discipline. Treat local models as experimental for serious agentic coding until they prove reliable tool use.") + ("avoid" + . + "Do not assume a weak local model will recover from a tool-error correction by emitting a valid structured tool call. It may produce long prose plans, consume slow provider turns, and never make an accepted edit.") + ("id" . "local-model-prose-after-tool-error") + ("kinds" "all" "script" "test" "debug-error") + ("pattern" + . + "Your previous response was not a valid tool call|must respond with a tool call|ToolError.*Stop routing to MCP") + ("severity" . "high") + ("tags" "local-model" "tool-calls" "verified" "jcode" "mlx" + "recovery") + ("title" + . + "Local model replies with prose after a required tool-call error") + ("tools" + "jerboa_request_advisor" + "verify" + "jerboa_anti_pattern_lookup"))) --- a/data/features.sexp +++ b/data/features.sexp @@ -2972,4 +2972,59 @@ ("use_case" . "Use when build scripts generate Scheme or C source via display/fprintf. The scanner should still catch issues in emitted code, but it should evaluate cleanup patterns within the emitted code instead of treating each string literal line as executable source in the generator.") + ("votes" . 0)) + (("description" + . + "When a verified edit/write is rejected by the Jerboa syntax guard, save the rejected draft to a quarantine path or expose it through a read-only recovery tool, and include the exact syntax/balance diagnostics in the tool result. This gives the next model turn something concrete to repair instead of forcing a complete regeneration from memory.") + ("estimated_token_reduction" + . + "~1500-5000 tokens per rejected-write repair, and often avoids one full regeneration turn") + ("example_scenario" + . + "In the DS4 TUI Tetris benchmark, the model eventually called write for tetris.ss, but the syntax guard rejected the file. Because no draft was saved on disk, the repair turn had only the error in context and stalled in another long provider call.") + ("id" . "preserve-rejected-write-draft") ("impact" . "high") + ("tags" "jcode" "verified" "syntax-guard" "local-model" + "repair") + ("title" . "Preserve syntax-guard rejected write drafts") + ("use_case" + . + "Verified coding runs where a model attempts to create or rewrite a Jerboa file and the guard rejects it before the file reaches disk.") + ("votes" . 0)) + (("description" + . + "Add a verified-run policy for local/experimental providers that counts invalid non-tool replies after a tool call is required. After a small budget, stop the run with a structured failure reason instead of continuing slow provider calls indefinitely. Optionally retry once with a minimal prompt that exposes only write/edit/verify.") + ("estimated_token_reduction" + . + "Eliminates repeated multi-minute recovery turns and saves ~2000-8000 prompt/completion tokens per failed local-model run") + ("example_scenario" + . + "After jcode blocked pre-edit MCP discovery, DS4 returned prose instead of a tool call, was corrected, eventually tried one rejected write, then spent more than three minutes in another repair turn with no accepted edit.") + ("id" . "verified-local-model-tool-call-hard-stop") + ("impact" . "high") + ("tags" "jcode" "verified" "local-model" "tool-calls" + "benchmark") + ("title" + . + "Hard-stop local verified runs after repeated invalid tool-call replies") + ("use_case" + . + "Benchmarking or using local models whose OpenAI-compatible tool-call behavior is unreliable under large prompts or after tool errors.") + ("votes" . 0)) + (("description" + . + "Improve request_advisor write-scope recommendations for greenfield tasks. If the task says to add a root source file, implement a script, or the acceptance criteria explicitly forbid editing tests, avoid recommending tests/ as the write scope. Prefer the repo root or the named file path.") + ("estimated_token_reduction" + . + "~500-1500 tokens per affected run by avoiding scope confusion and unnecessary clarification turns") + ("example_scenario" + . + "The TUI Tetris problem required adding root tetris.ss and not modifying test/run-tests.sh, but the advisor guidance said Recommended write scope: tests/. That conflicts with the task and can mislead constrained models.") + ("id" . "request-advisor-greenfield-write-scope") + ("impact" . "medium") + ("tags" "request-advisor" "write-scope" "greenfield" "tests" + "verified") + ("title" . "Infer write scope for greenfield source tasks") + ("use_case" + . + "jcode verified runs that use request_advisor output to set or reinforce allowed edit scope before starting a task.") ("votes" . 0)))