Document v6 agent eval findings
Jaime Fournier <jaimef@linbsd.org>
d0f2eaf4474a7e6a0fea4be6edbf800a2fe7ac2c
diff --git a/v6-agent-eval-findings.md b/v6-agent-eval-findings.md new file mode 100644 index 0000000..0dc91ed --- /dev/null +++ b/v6-agent-eval-findings.md @@ -0,0 +1,228 @@ +# v6 Agent Eval Findings + +Date: 2026-06-17 + +This note records the Conway's Game of Life isolation test run from +`~/mine/kratistos` against the local MLX models and Jerboa adapter variants. +The purpose was to determine whether failures came from the base models, the +LoRA adapters, MCP/tool wiring, or their interaction. + +## Short conclusion + +LoRA/adapters remain the right mechanism for adding Jerboa-specific behavior, +but the current adapter is not improving this agent workflow. In this test, it +usually made behavior worse. + +The clean signal was: + +```text +qwen80 base + Jerboa MCP: passed Conway +qwen80 adapter + MCP: failed / looped +gpt-oss base no-MCP: substantial edit/test attempt +gpt-oss adapter no-MCP: no edit +``` + +So the failure is not simply "local models cannot use tools" or "MCP is +broken". The MCP path can work. The current adapter appears to damage agentic +behavior: instruction following, edit discipline, tool-loop control, or runtime +debugging. + +## Harness correction + +The first no-MCP runs were invalid because OpenCode inherited the user's global +MCP config. `KRATISTOS_MCP=0` did not actually remove top-level `mcp` from the +isolated OpenCode config. + +I patched `kratistos/runners.ss` so isolated OpenCode configs remove top-level +`mcp` when MCP is disabled, then reran the no-MCP axes. The clean no-MCP run +configs were verified with: + +```text +has_mcp=False +``` + +## Clean matrix + +Task: implement `life.ss` for text Conway's Game of Life in Jerboa. + +Acceptance: `make test` prints `OK: 8 checks, 0 failures`. + +Timeouts: 600s wall, 180s idle for the clean reruns. + +| Config | Model | Result | Notes | +| --- | --- | --- | --- | +| base/no-MCP | gpt-oss 120B | fail | 105 added lines, substantial edit/test loop, ended in syntax/runtime failure. | +| base/no-MCP | qwen80 | fail | 91 added lines, wall timeout, repeated non-passing tests. | +| base/no-MCP | gemma31 | fail | 11 added lines, poor Jerboa probing, wall timeout. | +| base+MCP | gpt-oss 120B | fail | 1 added line, much worse than base/no-MCP. | +| base+MCP | qwen80 | pass | 88 added lines, passed acceptance. | +| base+MCP | gemma31 | fail | 0 added lines, idle timeout after limited context/tool use. | +| adapter/no-MCP | gpt-oss 120B | fail | 0 files changed. Adapter suppressed useful editing. | +| adapter/no-MCP | qwen80 | fail | 78 added lines, wall timeout, stuck on argument parsing. | +| adapter/no-MCP | gemma31 | fail | 5 added lines, wrote invalid probe, dumped large tree, idle timeout. | +| adapter+MCP | gpt-oss 120B | fail | 0 files changed. | +| adapter+MCP | qwen80 | fail | 0 files changed / repeated tool loop. | +| adapter+MCP | gemma31 | fail | 0 files changed / idle timeout. | + +Run directories used for the clean/valid data: + +```text +base/no-MCP: +~/mine/kratistos/runs/20260617-210343-conway-matrix-base-true-nomcp + +adapter/no-MCP: +~/mine/kratistos/runs/20260617-214042-conway-matrix-adapter-true-nomcp-rerun + +base+MCP: +~/mine/kratistos/runs/20260617-203207-conway-matrix-base-mcp + +adapter+MCP: +~/mine/kratistos/runs/20260617-193016-conway-adapters-mcp +~/mine/kratistos/runs/20260617-194255-conway-gemma31-adapter-mcp +``` + +## What failed + +### GPT-OSS + +Base/no-MCP can act like a coding agent: it writes code, runs tests, and tries +to repair. It did not land a passing Jerboa solution, but it made real progress. + +With the adapter, GPT-OSS stopped being useful in this task: + +```text +base/no-MCP: 105 added lines +adapter/no-MCP: 0 added lines +adapter+MCP: 0 added lines +``` + +This points to an adapter-induced behavior change, not just missing Jerboa +knowledge. + +### Qwen80 + +Qwen80 base+MCP passed. That is the best observed local configuration for this +task. + +The adapter variants regressed: + +```text +base+MCP: passed +adapter+MCP: looped / no useful diff +adapter/no-MCP: wrote code but timed out on argument parsing +``` + +This suggests Qwen80 benefits from MCP guidance, but the adapter changes the +model's tool/use-repair behavior enough to lose the pass. + +### Gemma31 + +Gemma was poor in every configuration. It repeatedly guessed invalid Jerboa +syntax such as: + +```scheme +(jerboa prelude) +``` + +instead of: + +```scheme +(import (jerboa prelude)) +``` + +It also responded to concrete runtime errors by dumping huge directory trees +from `~/mine/jerboa`, which wastes context and does not repair the issue. + +## Likely cause + +The adapter may have learned Jerboa-looking tokens and docs, but not successful +agent behavior. If the training data is mostly syntax, docs, snippets, or +completion-style examples, the LoRA can increase Jerboa-flavored output while +hurting: + +- following the task brief; +- using OpenCode edit/read/bash tools productively; +- calling MCP only when it helps; +- running the exact acceptance test; +- repairing from concrete errors; +- stopping after success. + +For agent use, Jerboa knowledge is not enough. The model needs traces of the +workflow: + +```text +read task -> inspect files -> write small implementation -> run make test +-> read exact error -> apply focused patch -> rerun tests -> stop on pass +``` + +## v6 implications + +1. Keep `qwen80 base+MCP` as the current local baseline. + +2. Do not call a v6 adapter better unless it beats `qwen80 base+MCP` on this + exact Conway benchmark and at least a few other Jerboa coding tasks. + +3. Evaluate the final deployable path, not just the trained checkpoint: + + ```text + quantized base + adapter + OpenCode + MCP setting + ``` + +4. Add agentic training examples, not only Jerboa text. Include successful + traces with file reads, edits, `make test`, focused error repair, and final + stopping behavior. + +5. Add negative/preference examples for repeated observed failures: + + - do not use `(jerboa prelude)` as an expression; + - do not invent `command-line-args`; + - do not dump `ls -R ~/mine/jerboa` after one concrete error; + - do not repeat the same failing manual test without editing; + - do not loop on MCP calls instead of writing the target file; + - do not modify test files to pass the benchmark. + +6. Keep ablations mandatory: + + ```text + base/no-MCP + base+MCP + adapter/no-MCP + adapter+MCP + ``` + + Without these four axes, it is too easy to blame the wrong component. + +7. The next adapter should be trained and selected for agent success, not just + Jerboa syntax familiarity. + +## Practical next eval set + +Use Conway as the first gate, then add a small fixed battery: + +```text +1. Conway text CLI in Jerboa. +2. A command-line stdin parser task. +3. A small module using `(jerboa prelude)` and one `(std ...)` import. +4. A task with one intentional failing test requiring focused repair. +5. A task where MCP has the exact answer, to test useful MCP usage. +``` + +Each task should record: + +```text +pass/fail +files changed +lines added/deleted +turn count +wall time +whether tests were run +last failure mode +whether MCP was used productively +``` + +## Bottom line + +The lesson is not "LoRA is wrong". The lesson is that this adapter was trained +or selected for the wrong observable behavior. v6 should keep the adapter +architecture, but optimize and evaluate it against complete agentic Jerboa task +success.