Improve local verifier repair loops
ober
2687973ff5b0d5c89e10b9c0a5ea7a1a5f14945c
new file mode 100644 --- /dev/null +++ b/docs/tui-tetris-local-ds4-optimization.md @@ -0,0 +1,95 @@ +# Local DS4 TUI Tetris Optimization + +## Benchmark + +The harder follow-up benchmark asks DeepSeek V4 Pro (`deepseek-v4-pro`) to +create a one-file Jerboa terminal Tetris implementation. Acceptance covers a +10x20 snapshot, seven pieces, collision, rotation, hard drop, line clearing, +scoring, game over, ANSI terminal control, and an executable self-test. The +public gate is `make test`, with 12 checks. + +The benchmark exposed more than source-generation difficulty. A candidate can +be delimiter-balanced and still fail on Jerboa-specific record syntax, +unsupported keyword-style calls, binding scope, or a later self-test path. +Each repair can expose another runtime failure. + +## Baseline Failure + +The first jcode run used local DS4, expert fallback enabled, a 30-minute wall +limit, and 72 model turns. It retained one verifier-visible `tetris.ss`, but +timed out at 5 failures. Its first central failure was: + +```text +Exception: invalid syntax (defstruct game ...) at line 146 +``` + +Jcode diagnosed a structural span but did not explain Jerboa's required +`(defstruct name (fields ...))` form. The model then spent its final turns +trying unrelated prelude and shell inspection calls even though the runtime +blocked them. + +The artifact later reached 10/12 through verifier-guided continuation, but a +new plateau appeared: `read -> verify unchanged -> read`. Every unchanged +failed verify reset the local post-failure inspection counter, so the model +could renew its observation allowance indefinitely. + +## Retained Changes + +1. Invalid `defstruct` diagnostics now state the exact Jerboa grammar and show + `(defstruct game (board score))` as the replacement shape. +2. A repeated failed verify with no successful edit no longer resets the local + inspection or required-range inspection counters. Exhausted read/list/ + balance schemas remain hidden while exact repairs and verify stay visible. +3. `bracketed list terminated by parenthesis` is recognized as a structural + repair. Jcode identifies the reported line/character, generates a line with + `)` changed to `]`, and explains that `[` must close with `]`. +4. Regression coverage reproduces all three cases, including the exact + unchanged-verify inspection-renewal loop. + +## Live Result + +After the first diagnostic change, local DS4 continued the preserved artifact. +It repaired unsupported keyword-like constructor declarations and calls, +placement logic, self-test setup, and rendering. The run reached 10/12 but +eventually plateaued on unchanged verify/read cycles. + +After installing the inspection-renewal fix, another continuation started from +that 10/12 artifact. It repaired the remaining unbound-cell path, added ANSI +rendering, corrected self-test assumptions, and iterated on hard-drop/line-clear +behavior. Jcode automatically verified after each successful edit. The run +finished normally with `status: passed` and `OK: 12 checks, 0 failures`. + +Independent checks confirmed 20 snapshot rows, score/lines/level output, and a +self-test ending in `OK: tui tetris self-test passed`. The status recorded +121,219 input tokens, 13,026 output tokens, 34,096 cache-read tokens, 6,883 +reasoning tokens, and $0.0421 total provider cost. Expert fallback remained +enabled as required. + +This was an iterative recovery result, not a fresh post-change jcode run. The +measured claim is that jcode retained and repaired a difficult local-model +artifact to a full pass instead of discarding it or looping until timeout. + +## OpenCode Comparison + +A fresh corrected OpenCode run used the same local DS4 model and the same +30-minute limit. It timed out after 60 turns with 7/12 checks passing. The final +artifact had a bracket/parenthesis mismatch at line 250 and lacked the ANSI +source hook. Metrics were 25,631 input tokens, 25,113 output tokens, and +1,673,785 cache-read tokens. + +OpenCode made 43 shell calls, 13 reads, four edits, and two writes. It spent a +large part of the run rediscovering `defstruct` syntax and later bisecting a +delimiter error with temporary files, partial-file execution, and Python +delimiter counts. Those observations directly motivated the exact jcode +diagnostics above. + +An earlier OpenCode run produced an artifact that rescored 12/12 after the +benchmark's malformed ANSI grep was corrected, but that agent timed out and +created an out-of-scope `bin/grep` workaround against the bad harness. It is +not treated as a clean terminal pass. + +## Verification + +The final jcode suite is `1223 passed, 0 failed, 1 skipped`. The skip remains +the existing environment-dependent image backend test. The final binary was +rebuilt and installed to `~/.local/bin/jcode`. --- a/src/jcode/core/verified-run.ss +++ b/src/jcode/core/verified-run.ss @@ -522,17 +522,34 @@ (list->string suffix))) (else #f))))) +(def (replace-mismatched-bracket-close line char-no) + (let ((idx (and char-no (- char-no 1)))) + (and idx + (>= idx 0) + (< idx (string-length line)) + (char=? (string-ref line idx) #\)) + (string-append + (substring line 0 idx) + "]" + (substring line (+ idx 1) (string-length line)))))) + (def (unexpected-close-repair detail cwd) (if (not (or (string-contains detail "unexpected close parenthesis") - (string-contains detail "Unexpected close"))) + (string-contains detail "Unexpected close") + (string-contains detail "bracketed list terminated by parenthesis"))) #f - (let* ((diag-start (or (find-diagnostic-start + (let* ((bracket-mismatch? + (string-contains detail "bracketed list terminated by parenthesis")) + (diag-start (or (find-diagnostic-start detail - '("Exception in read: unexpected close" + '("Exception in read: bracketed list terminated by parenthesis" + "bracketed list terminated by parenthesis" + "Exception in read: unexpected close" "unexpected close parenthesis" "Unexpected close")) 0)) - (line-no (find-line-number-after-from detail " at line " diag-start))) + (line-no (find-line-number-after-from detail " at line " diag-start)) + (char-no (find-line-number-after-from detail ", char " diag-start))) (if (not line-no) #f (let ((path (verification-source-path cwd detail line-no diag-start))) @@ -543,7 +560,11 @@ #f (let* ((lines (string-split (read-file-string p) #\newline)) (line (line-at lines line-no)) - (candidate (and line (drop-last-close-delim line)))) + (candidate + (and line + (if bracket-mismatch? + (replace-mismatched-bracket-close line char-no) + (drop-last-close-delim line))))) (and candidate (make-required-range-repair path line-no line-no line-no 'delimiter candidate))))))))))) @@ -676,6 +697,13 @@ (repair-ref repair 'path) (repair-ref repair 'start) (repair-ref repair 'end)) + (if (and (string-contains detail "invalid syntax") + (string-contains detail "(defstruct")) + (string-append + "\n\nJerboa defstruct diagnosis: all fields must be inside one parenthesized field list. " + "Replace `(defstruct game board score)` with `(defstruct game (board score))`. " + "Apply that exact shape to the diagnosed span, then call verify.") + "") (if (and span (<= (string-length span) 4000)) (string-append "\n\nSuspect span:\n" span) "") @@ -691,7 +719,7 @@ (let ((label (required-repair-label repair))) (string-append (format - "\n\nDelimiter diagnosis: verifier reported an unexpected close delimiter in ~a at line ~a. Repair that exact line with replace_range, then call verify again:\n read(path=\"~a\", start=~a, end=~a)\n replace_range(path=\"~a\", start=~a, end=~a, content=<corrected line>)" + "\n\nDelimiter diagnosis: verifier reported a mismatched or unexpected close delimiter in ~a at line ~a. Repair that exact line with replace_range, then call verify again:\n read(path=\"~a\", start=~a, end=~a)\n replace_range(path=\"~a\", start=~a, end=~a, content=<corrected line>)" (repair-ref repair 'path) (repair-ref repair 'line) (repair-ref repair 'path) @@ -700,6 +728,9 @@ (repair-ref repair 'path) (repair-ref repair 'start) (repair-ref repair 'end)) + (if (string-contains detail "bracketed list terminated by parenthesis") + "\n\nJerboa bracket diagnosis: a list opened with `[` must close with `]`. Replace the reported `)` at that line and character with `]`, then verify." + "") (best-repair-candidate-text cwd repair label)))))) (def (augment-verify-detail detail cwd) @@ -1417,7 +1448,13 @@ (current-required-repair-inspections 0)) (def (record-verify-result! result . review-after-verify) - (let ((passed? (and (pair? result) (car result)))) + (let ((passed? (and (pair? result) (car result))) + ;; Re-running the same failing verifier without an edit must not renew + ;; the local model's inspection budget. Otherwise it can loop forever + ;; through read -> unchanged verify -> read. + (unchanged-reverify? + (and (current-after-failed-verify?) + (not (current-edited-since-verify?))))) (current-after-failed-verify? (not passed?)) (if passed? (begin @@ -1428,7 +1465,7 @@ (when (and (current-verified-local-model?) (>= failures local-expert-after-verify-failures) (config-expert-enabled?)) - (current-force-expert-next? #t))))) + (current-force-expert-next? #t)))) (current-requirements-review-pending? (and (pair? result) (car result) @@ -1441,9 +1478,10 @@ (when (and (pair? result) (car result)) (reset-existing-ss-rewrite-state!) (current-required-range-repair #f)) - (current-required-repair-inspections 0) - (current-inspections-after-failed-verify 0) - result) + (unless unchanged-reverify? + (current-required-repair-inspections 0) + (current-inspections-after-failed-verify 0)) + result)) (def (record-successful-edit! cwd path) (clear-pending-ss-create-repair! cwd path) --- a/test/run.ss +++ b/test/run.ss @@ -4690,6 +4690,30 @@ (safe-delete-test-file! target-path)) (let* ([vr-dir "/tmp"] + [target "jcode-invalid-defstruct-diagnosis.ss"] + [target-path (string-append vr-dir "/" target)] + [bad-source "(import (jerboa prelude))\n(defstruct game\n board\n score)\n\n(define (next) 2)\n"]) + (safe-delete-test-file! target-path) + (write-test-output-file target-path + (lambda (o) (display bad-source o)) + 'replace) + (let* ([cmd (string-append + "printf '%s\\n' 'Exception: invalid syntax (defstruct game board score) at line 2, char 1 of " + target + "' >&2; exit 255")] + [result (run-verify-command cmd vr-dir)] + [detail (cdr result)]) + (check! "verified-run: invalid defstruct fake verify fails" + (car result) #f) + (check-pred! "verified-run: invalid defstruct detail gives exact Jerboa shape" + detail + (lambda (s) + (and (str-contains? s "Jerboa defstruct diagnosis") + (str-contains? s "(defstruct game (board score))") + (str-contains? s "then call verify"))))) + (safe-delete-test-file! target-path)) + + (let* ([vr-dir "/tmp"] [target "jcode-invalid-syntax-after-warning.ss"] [target-path (string-append vr-dir "/" target)] [bad-source "(import (jerboa prelude))\n(define (warn)\n (error \"bad\"))\n\n(define (bad row)\n (do ((x 0 (+ x 1))\n ((= x 3))\n (displayln x)))\n\n(define (next) 2)\n"]) @@ -4828,6 +4852,32 @@ (safe-delete-test-file! target-path)) (let* ([vr-dir "/tmp"] + [target "jcode-mismatched-bracket-diagnosis.ss"] + [target-path (string-append vr-dir "/" target)] + [bad-source "(import (jerboa prelude))\n(define (bad)\n (displayln [1 2 3)))\n"]) + (safe-delete-test-file! target-path) + (write-test-output-file target-path + (lambda (o) (display bad-source o)) + 'replace) + (let* ([cmd (string-append + "printf '%s\\n' 'Exception in read: bracketed list terminated by parenthesis at line 3, char 20 of " + target + "' >&2; exit 255")] + [result (run-verify-command cmd vr-dir)] + [detail (cdr result)]) + (check! "verified-run: mismatched bracket fake verify fails" + (car result) #f) + (check-pred! "verified-run: mismatched bracket detail gives exact rule" + detail + (lambda (s) + (and (str-contains? s "Jerboa bracket diagnosis") + (str-contains? s "opened with `[` must close with `]`") + (str-contains? s "replace_range") + (str-contains? s "start=3") + (str-contains? s "end=3"))))) + (safe-delete-test-file! target-path)) + + (let* ([vr-dir "/tmp"] [target "jcode-unexpected-close-line.ss"] [target-path (string-append vr-dir "/" target)] [bad-source "(import (jerboa prelude))\n(define (bad)\n (displayln \"x\")))\n"] @@ -7232,6 +7282,54 @@ [else (loop (cdr ys))]))))) (safe-delete-test-file! target-path)) + (let* ([vr-dir "/tmp"] + [target "jcode-verified-local-unchanged-reverify-budget.txt"] + [target-path (string-append vr-dir "/" target)] + [inspections-stay-hidden? #f]) + (safe-delete-test-file! target-path) + (write-test-output-file target-path + (lambda (o) (display "broken\n" o)) 'replace) + (let* ([read-call + (lambda () + (list (make-wtool-call "read" + (list (cons "path" target)) #f)))] + [resp + (lambda (_messages specs step) + (case step + [(0) (list (make-wtool-call "verify" '() #f))] + [(1) (read-call)] + [(2) (read-call)] + [(3) (list (make-wtool-call "verify" '() #f))] + [(4) + (let ([names (map tool-spec-name specs)]) + (set! inspections-stay-hidden? + (and (not (member "read" names)) + (not (member "list" names)) + (not (member "balance" names)) + (member "edit" names) + (member "verify" names) + #t))) + (list + (make-wtool-call "edit" + (list (cons "path" target) + (cons "content" "fixed\n")) #f))] + [(5) (list (make-wtool-call "verify" '() #f))] + [else (error 'test "unexpected unchanged reverify step")]))] + [result + (verified-run resp "do not renew reads on unchanged verify" + (list + (cons 'cwd vr-dir) + (cons 'verify-command + (string-append "grep -q fixed " target)) + (cons 'write-scope (parse-write-scope target)) + (cons 'local-model? #t) + (cons 'max-iterations 8)))]) + (check! "verified-run: unchanged local reverify still reaches pass" + result "VERIFIED: exit 0\n") + (check! "verified-run: unchanged local reverify does not renew inspections" + inspections-stay-hidden? #t)) + (safe-delete-test-file! target-path)) + (let* ([vr-dir "/tmp"] [target "jcode-verified-read-after-failed-verify.txt"] [target-path (string-append vr-dir "/" target)]