Complete focused local model repair workflow
ober
b787bee72c41c2a3f976f676651ff7657bd62bd6
--- a/docs/rle-benchmark-optimization.md +++ b/docs/rle-benchmark-optimization.md @@ -36,19 +36,22 @@ compliance. ## Current Outcome -The retained implementation passes 1,203 tests with zero failures and one +The retained implementation passes 1,217 tests with zero failures and one environment-dependent image-backend skip. The final installed binary has separate local-origin behavior: bounded inspection, strict transactional edits, sticky expert escalation after repeated rejection, an 8,192-token first-draft -cap, and a 4,096-token cap after the first successful edit or verifier failure. +cap, and a 2,048-token focused cap after the first successful edit or verifier +failure. The local-origin cap follows a handoff to a cloud expert; ordinary cloud-origin workflows remain unchanged. The latest local DeepSeek V4 Pro hard-RLE run is documented under **Local DeepSeek V4 Pro Iteration (2026-07-15)** below. It improved the earlier local hard result from a globally broken 0/22 artifact to 20/22 after 971 seconds. -A continuation on the promoted artifact reached 21/22 before the combined -30-minute budget expired. This is substantial progress, but not a pass. +A continuation on the promoted artifact first reached 21/22 before the combined +30-minute budget expired. After the focused-repair changes described below, a +fresh continuation on that exact dirty artifact repaired the remaining spaced +rule input and passed all 22 checks. ### Earlier OpenRouter Cohort @@ -101,9 +104,19 @@ expert turn was stopped and the result remains a fail. This trajectory identified the final retained optimization: 8K is useful while a local model may need to emit the first complete implementation, but excessive for a localized repair. Post-edit and post-verifier local-origin turns now use a -4K cap, including expert handoffs. Deterministic tests verify the 8K-to-4K -transition; a new full cold benchmark was not run after this last timing-only -change. +2K cap, including expert handoffs. Dirty local worktrees are verified before the +first model turn, literal failing inputs are promoted near the start of the +diagnostic, full-file rewrites are hidden and rejected once a valid `.ss` +artifact exists, and every successful local edit immediately triggers the +authoritative verifier. + +The final focused continuation reused the 21/22 artifact in the same run +directory. Jcode ran `make test` before contacting the model, exposed the exact +`B3 / S23` failure, accepted one exact replacement that trimmed the two rule +halves around `/`, and automatically verified `OK: 22 checks, 0 failures`. +Because the CLI intentionally enables a post-pass requirements review, the +model then made one bounded 2K completion call and selected `done` without +changing the file. The status file records a pass and 1,984 output tokens. ## Experiment Rules @@ -1267,3 +1280,37 @@ ended with no promoted file and zero public positive cases after 39 provider calls, costing $0.5718. Together with the earlier unsuccessful 8K experiment, this is negative evidence for retaining a model-specific cap. The cap was removed; explicit `JCODE_MAX_TOKENS` and configured limits remain available. + +### Local Focused Repair Completion + +The final local-only workflow pass retained five related changes: + +1. A dirty local git worktree is verified before the first provider request. +2. Local repair turns use a 2,048-token cap instead of 4,096 tokens. +3. After a verifier-visible `.ss` artifact exists, the provider no longer sees + `write`; `edit` is narrowed to required `path`, `old_str`, and `new_str`, and + runtime enforcement rejects broad rewrites that bypass the schema. +4. A successful local edit automatically invokes the configured verifier + without asking the model to spend another turn on `verify()`. +5. Verifier diagnostics promote unique literal failure lines before longer + advisory output, preserving exact inputs such as `B3 / S23`. + +The same patch also retains transport hardening contributed during the +iteration: an incomplete OpenAI-compatible stream is rejected unless it ends +with a finish reason or `[DONE]`, retry classification includes aborted streams, +and the first truncated local response is allowed one constrained continuation +before expert escalation. Local requests hide Jerboa MCP schemas when the task +guidance already contains the required language patterns. + +Deterministic coverage exercises dirty preflight verification, the 2K cap, +schema narrowing, runtime rewrite rejection, literal failure evidence, exact +repair, automatic verification, truncated-stream retry, and deferred expert +handoff. The complete suite reports `1217 passed, 0 failed, 1 skipped`. + +Live validation used the existing 21/22 artifact under +`20260715-212140-life-rle-local-ds4-jcode-expertcap`. The installed binary +verified the dirty tree first, DeepSeek V4 Pro read the existing implementation, +made one exact replacement, and jcode automatically reached `22/22`. The CLI's +intentional requirements-review turn then called `done` without another edit. +This converts the local hard-RLE benchmark from the original 0/22 failure, via +20/22 and 21/22 intermediate artifacts, to a full public pass. --- a/src/jcode/core/expert.ss +++ b/src/jcode/core/expert.ss @@ -33,6 +33,7 @@ chat-with-expert-via-stream chat-direct-via-stream stream-chat-with-expert + current-defer-truncated-escalation current-expert-cb) (import :std/misc/string @@ -50,6 +51,11 @@ ;; JCODE_NO_EXPERT provides the same behavior for scripts. (def current-expert-disabled (make-parameter #f)) +;; Verified local workflows should get one direct continuation after spending +;; their output budget on reasoning. Replacing that first attempt immediately +;; discards useful task analysis and often sends the expert back to discovery. +(def current-defer-truncated-escalation (make-parameter #f)) + ;; Optional escalation hook. When set (e.g. by the TUI worker), it is invoked ;; on the streaming path INSTEAD of pushing the text banner through token-cb, ;; so the UI can render a structured indicator and colour the expert's reply @@ -258,12 +264,30 @@ ;; Build a synthetic assistant message so should-escalate? can ;; inspect tool-call shapes without touching the live session. (synth (make-assistant-message content tcs)) - (auto-reason (should-escalate? messages synth stats)) + (raw-auto-reason (should-escalate? messages synth stats)) + (truncated? + (and raw-auto-reason + (equal? (cdr (assq 'signal raw-auto-reason)) + 'truncated-response))) + (provider-aborted? + (and raw-auto-reason + (equal? (cdr (assq 'signal raw-auto-reason)) + 'provider-aborted))) + ;; Ignore only the truncation signal. Re-run the detectors without + ;; finish_reason so a repeated verified prose response can still + ;; escalate after the workflow's explicit tool-call nudge. + (auto-reason + (if (and truncated? (current-defer-truncated-escalation)) + (should-escalate? messages synth '()) + raw-auto-reason)) (escalate? (or sentinel? auto-reason)) (reason-text (if auto-reason (format-escalation-reason auto-reason) "sentinel"))) (cond + ((and provider-aborted? (current-defer-truncated-escalation)) + (error 'stream-chat-with-expert + "provider aborted the stream before producing a usable tool call")) ((and escalate? (config-expert-enabled?)) (let ((expert (get-expert-provider))) (log-info logger "escalating-to-expert" @@ -313,14 +337,32 @@ (values (strip-expert-sentinel content) tcs usage)) (else (values content tcs usage)))))) +(def verified-stream-retry-limit 2) + +(def (verified-stream-call-with-retry thunk) + (let loop ((attempt 0)) + (guard (e [#t + (if (and (< attempt verified-stream-retry-limit) + (provider-retryable-error? e)) + (begin + (log-warn logger "verified-stream-retry" + `((attempt . ,(+ attempt 1)) + (err . ,(err->string e)))) + (thread-sleep! 1) + (loop (+ attempt 1))) + (raise e))]) + (thunk)))) + (def (chat-with-expert-via-stream provider messages tools) ;; Verified workflows do not display incremental model text, but using the ;; streaming transport still lets gateways send progress bytes during long ;; generations. Collect the response into the same message shape expected by ;; the workflow backend and record the usage returned by the wrapper. (let-values (((content tool-calls usage) - (stream-chat-with-expert - provider messages tools (lambda (_token) (void))))) + (verified-stream-call-with-retry + (lambda () + (stream-chat-with-expert + provider messages tools (lambda (_token) (void))))))) (record-current-usage! usage) (make-assistant-message content tool-calls))) @@ -330,11 +372,13 @@ repeated verifier failures; unlike chat-with-expert-via-stream, this does not run escalation detection recursively." (let-values (((content tool-calls usage) - (provider-stream-chat - provider - (with-expert-handoff - messages "repeated verifier failures" "") - tools - (lambda (_token) (void))))) + (verified-stream-call-with-retry + (lambda () + (provider-stream-chat + provider + (with-expert-handoff + messages "repeated verifier failures" "") + tools + (lambda (_token) (void))))))) (record-current-usage! usage) (make-assistant-message content tool-calls))) --- a/src/jcode/core/verified-run.ss +++ b/src/jcode/core/verified-run.ss @@ -42,7 +42,7 @@ (def default-local-verified-history-token-limit 24000) (def default-local-verified-history-keep-batches 2) (def default-local-verified-max-completion-tokens 8192) -(def default-local-verified-repair-max-completion-tokens 4096) +(def default-local-verified-repair-max-completion-tokens 2048) (def (opt-get o key) (let ((p (assoc key o))) (and p (cdr p)))) @@ -177,10 +177,21 @@ (verified-mcp-tool-spec? spec))) (def (verified-staged-repair-tool-spec? spec) - (or (member (tool-spec-name spec) - '("line_edit" "replace_def" "replace_range")) - (and (current-serving-forced-expert?) - (member (tool-spec-name spec) '("edit" "write"))))) + (member (tool-spec-name spec) + '("line_edit" "replace_def" "replace_range"))) + +(def (local-focused-repair-mode?) + (and (current-verified-local-model?) + (or (current-after-failed-verify?) + (> (current-successful-edit-count) 0)))) + +(def (verified-provider-tool-spec spec) + (if (and (local-focused-repair-mode?) + (string=? (tool-spec-name spec) "edit")) + (make-tool-spec "edit" + "Replace exact existing text only. args: {\"path\": string, \"old_str\": string, \"new_str\": string}. Full-file rewrites are disabled after a valid local artifact exists; use line_edit, replace_def, or replace_range for bounded repairs." + *exact-edit-schema*) + spec)) (def (verified-post-edit-tool-spec? spec) (member (tool-spec-name spec) @@ -194,16 +205,29 @@ "verify" "done"))) (def (verified-provider-tool-specs specs) - (filter + (map verified-provider-tool-spec + (filter (lambda (spec) (let ((name (tool-spec-name spec))) (cond + ;; Constrained local mode already carries the Jerboa CLI patterns in + ;; task guidance. Advertising discovery schemas contradicts that + ;; guidance, increases prefill, and invites another inspection turn. + ((and (current-verified-local-model?) + (verified-mcp-tool-spec? spec)) + #f) ;; Requirements review is a local evidence check, not a new build or ;; discovery phase. Keep one canonical inspection surface and the ;; actions needed to repair, reverify, or finish. ((and (current-requirements-review-pending?) (not (verified-requirements-review-tool-spec? spec))) #f) + ;; Once a local run has a verifier-visible artifact, keep exact edit + ;; but remove the broad write alias. The edit schema is narrowed + ;; below to old_str/new_str so the provider cannot request a rewrite. + ((and (local-focused-repair-mode?) + (string=? name "write")) + #f) ;; After a successful edit, another observation should come from the ;; authoritative verifier. Keep repair tools in case the model spots ;; a mistake, but remove inspection, discovery, scaffold, and done. @@ -275,7 +299,7 @@ (>= (current-pre-edit-mcp-count) pre-edit-mcp-limit)) #f) (else #t)))))) - specs)) + specs))) (def (provider-responder provider) (let ((backend (if (procedure? provider) @@ -284,7 +308,9 @@ provider chat-with-expert-via-stream)))) (lambda (messages tool-specs step) (parameterize - ((current-max-tokens-cap + ((current-defer-truncated-escalation + (current-verified-local-model?)) + (current-max-tokens-cap (if (and (current-verified-local-model?) (or (current-after-failed-verify?) (> (current-successful-edit-count) 0))) @@ -689,6 +715,13 @@ "- Treat the verification output below as the behavioral authority.\n" "- Make the smallest code edit for the concrete failing path, then call verify again.\n" "- Do not call broad MCP or API discovery tools before that edit unless the failure is a missing API or syntax question.\n\n" + (let ((evidence (failure-evidence detail))) + (if evidence + (string-append + "Literal failure evidence (preserve these inputs and messages exactly):\n" + evidence + "\n\n") + "")) base)) (source-guidance (runtime-source-guidance base cwd)) (with-source (append-guidance-section @@ -703,6 +736,31 @@ with-source))) with-failure)) +(def failure-evidence-line-limit 12) + +(def (failure-evidence-line? line) + (let ((lower (string-downcase (string-trim line)))) + (or (string-prefix? "fail:" lower) + (string-prefix? "failed:" lower) + (string-contains lower "exception") + (string-contains lower "irritant") + (string-contains lower "expected:") + (string-contains lower "got:")))) + +(def (failure-evidence detail) + (let loop ((lines (string-split (or detail "") #\newline)) + (seen '()) + (out '()) + (left failure-evidence-line-limit)) + (cond + ((or (null? lines) (<= left 0)) + (and (pair? out) (string-join (reverse out) "\n"))) + ((and (failure-evidence-line? (car lines)) + (not (member (car lines) seen))) + (loop (cdr lines) (cons (car lines) seen) + (cons (car lines) out) (- left 1))) + (else (loop (cdr lines) seen out left))))) + (def (verify-output-forced-failure? out) (or (string-contains out "invalid context for definition") (string-contains out "Exception:") @@ -760,6 +818,16 @@ detail (augment-verify-detail detail cwd)))))))) +(def (verified-worktree-dirty? cwd) + (guard (_ [else #f]) + (let-values (((stdout _stderr exit-code) + (aproc-run/status + "git status --porcelain --untracked-files=all" + dir: cwd))) + (and (= exit-code 0) + (string? stdout) + (not (string=? (string-trim stdout) "")))))) + ;; ── coding workflow tools ────────────────────────────────────────────── (def (arg-ref args key default) (let ((p (assoc key args))) (if p (cdr p) default))) @@ -4113,6 +4181,16 @@ (raise-recoverable-tool-error (existing-ss-rewrite-lock-message cwd path) 'edit)) + ((and (file-exists? p) + (source-ss-path? path) + (local-focused-repair-mode?) + (not required-full-rewrite?)) + (raise-recoverable-tool-error + (string-append + "full-file rewrite disabled for verifier-visible local artifact " + path + ". Preserve the working implementation and use edit(path,old_str,new_str), line_edit, replace_def, or replace_range for the smallest diagnosed repair.") + 'edit)) (else (let ((dir (path-directory p))) (when (and dir (not (equal? dir "")) (not (file-exists? dir))) @@ -4237,6 +4315,16 @@ (cons "line" (number-schema "1-based line number for line replacement"))))) (cons "required" (list "path"))))) +(def *exact-edit-schema* + (schema-object + (list (cons "type" "object") + (cons "properties" + (schema-object + (list (cons "path" (string-schema "File path to edit")) + (cons "old_str" (string-schema "Exact existing text to replace")) + (cons "new_str" (string-schema "Replacement text"))))) + (cons "required" (list "path" "old_str" "new_str"))))) + (def *write-schema* (schema-object (list (cons "type" "object") @@ -4767,6 +4855,12 @@ (scope (if scope-pair (normalize-write-scope-option (cdr scope-pair)) #f)) + (initial-verify-pair (assoc 'initial-verify? o)) + (initial-auto-verify? + (and local-model? + (if initial-verify-pair + (and (cdr initial-verify-pair) #t) + (verified-worktree-dirty? cwd)))) (task-guidance (combine-task-guidance (verified-preflight-guidance task cwd vcmd scope) @@ -4792,6 +4886,16 @@ (cons 'max-repeated-calls (or (opt-get o 'max-repeated-calls) 6)) (cons 'max-no-progress-retries (or (opt-get o 'max-no-progress-retries) 2)) (cons 'retry-text-responses? #t) + (cons 'automatic-tool-calls + (lambda () + (cond + (initial-auto-verify? + (set! initial-auto-verify? #f) + (list (make-wtool-call "verify" '() #f))) + ((and local-model? + (current-edited-since-verify?)) + (list (make-wtool-call "verify" '() #f))) + (else #f)))) (cons 'prepare-messages (lambda (messages) (compact-verified-history --- a/src/jcode/core/workflow-runner.ss +++ b/src/jcode/core/workflow-runner.ss @@ -396,6 +396,10 @@ (else "If a scaffold or file already exists, call verify() before more prose. ")) "If code must change, call edit/write/line_edit/replace_def/replace_range with complete concrete code, never placeholders.\n" + (if (and (string? raw) + (string-prefix? "<think>" (string-trim raw))) + "Your prior turn exhausted its output budget in reasoning. Do not restart or repeat that analysis; use the task and tool results already present and issue the pending edit now.\n" + "") (if (> retry-count 1) "This is a repeated prose-only response. Stop free-form text and issue the tool call now.\n" "") @@ -417,6 +421,7 @@ max-repeated-calls (#f = off) max-no-progress-retries (0) retry-text-responses? (#f) prepare-messages (#f; optional messages -> provider-messages transform) + automatic-tool-calls (#f; optional thunk -> tool-call list) on-message (#f) prompt-vars ('()) initial-messages (#f) cancel? (thunk -> bool, default never). Raises MaxIterationsError / StepEnforcementError / PrerequisiteError / @@ -430,6 +435,7 @@ (max-no-progress-retries (or (opt-ref o 'max-no-progress-retries) 0)) (retry-text-responses? (and (opt-ref o 'retry-text-responses?) #t)) (prepare-messages (opt-ref o 'prepare-messages)) + (automatic-tool-calls (opt-ref o 'automatic-tool-calls)) (on-message (opt-ref o 'on-message)) (prompt-vars (or (opt-ref o 'prompt-vars) '())) (initial-msgs (opt-ref o 'initial-messages)) @@ -463,7 +469,19 @@ ((cancel?) (raise-workflow-cancelled (step-enforcer-completed enforcer) iteration)) (else - (let* ((provider-messages + (let ((automatic-calls + (and automatic-tool-calls (automatic-tool-calls)))) + (if (and (list? automatic-calls) (pair? automatic-calls)) + ;; Verified workflows use this to run their authoritative + ;; verifier without spending a model turn after an edit or + ;; when resuming a dirty workspace. + (let ((outcome + (execute-batch! emit! workflow enforcer + error-tracker automatic-calls))) + (if (and (pair? outcome) (eq? (car outcome) 'terminal)) + (cdr outcome) + (loop iteration))) + (let* ((provider-messages (if prepare-messages (prepare-messages messages) messages)) @@ -536,4 +554,4 @@ error-tracker tool-calls))) (if (and (pair? outcome) (eq? (car outcome) 'terminal)) (cdr outcome) - (loop (+ iteration 1)))))))))))))))))))))) + (loop (+ iteration 1)))))))))))))))))))))))) --- a/src/jcode/provider/provider.ss +++ b/src/jcode/provider/provider.ss @@ -116,6 +116,8 @@ ;; identical request is safe. (string-contains msg "HTTP read timed out") (string-contains msg "stream read timed out") + (string-contains msg "stream closed before a terminal event") + (string-contains msg "provider aborted the stream") ;; Some OpenAI-compatible gateways occasionally return HTTP 200 with ;; a truncated or whitespace-only body. No assistant response was ;; accepted, so retrying the identical request is safe. @@ -2286,7 +2288,8 @@ (usage-acc (make-hash-table)) (logprob-box (box '())) (entropy-box (box '())) - (finish-reason-box (box #f))) + (finish-reason-box (box #f)) + (saw-done-box (box #f))) (let* ((body-json (json-object->string body)) (dummy (begin (log-info logger "stream-request" @@ -2310,7 +2313,7 @@ (substring event-str 6 (string-length event-str)) event-str))) (cond - ((equal? data "[DONE]") (void)) + ((equal? data "[DONE]") (set-box! saw-done-box #t)) (else (let ((json (guard (e [(error? e) #f]) (string->json-object data)))) @@ -2375,6 +2378,13 @@ (unless (= http-status 200) (log-error logger "stream-http-error" `((status . ,http-status) (url . ,url))))) + ;; A clean HTTP EOF is not a complete model response unless the provider + ;; sent either the OpenAI [DONE] sentinel or a choice finish_reason. Local + ;; gateways can close mid-generation while still returning HTTP 200; do + ;; not turn that partial reasoning into an assistant turn. + (unless (or (unbox saw-done-box) (unbox finish-reason-box)) + (error 'openai-stream-chat + "stream closed before a terminal event (finish_reason or [DONE])")) ;; Build result (let* ((raw-content (get-output-string text-acc)) (reasoning (get-output-string reasoning-acc)) --- a/test/run.ss +++ b/test/run.ss @@ -82,11 +82,23 @@ (guard (e [#t e]) (error 'http-post-json "HTTP read timed out after 600s of silence (host: openrouter.ai)"))] + [incomplete-stream-error + (guard (e [#t e]) + (error 'openai-stream-chat + "stream closed before a terminal event (finish_reason or [DONE])"))] + [provider-aborted-error + (guard (e [#t e]) + (error 'stream-chat-with-expert + "provider aborted the stream before producing a usable tool call"))] [ordinary-error (guard (e [#t e]) (error 'http-post-json "invalid request body"))]) (check! "non-stream HTTP silence is retryable" (provider-retryable-error? timeout-error) #t) + (check! "incomplete model stream is retryable" + (provider-retryable-error? incomplete-stream-error) #t) + (check! "provider-aborted model stream is retryable" + (provider-retryable-error? provider-aborted-error) #t) (check! "ordinary provider errors remain terminal" (provider-retryable-error? ordinary-error) #f)) @@ -2449,6 +2461,58 @@ (tcp-close primary-srv) (tcp-close expert-srv)))) +(let* ([primary-srv (tcp-listen "127.0.0.1" 0)] + [expert-srv (tcp-listen "127.0.0.1" 0)] + [primary-url (format "http://127.0.0.1:~a/v1" + (tcp-server-port primary-srv))] + [expert-url (format "http://127.0.0.1:~a/v1" + (tcp-server-port expert-srv))] + [primary-captured (vector #f)] + [expert-captured (vector #f)] + [primary-body + (string-append + "data: {\"choices\":[{\"delta\":{\"reasoning_content\":\"useful analysis\"},\"finish_reason\":null}]}\n\n" + "data: {\"choices\":[{\"delta\":{},\"finish_reason\":\"length\"}]}\n\n" + "data: [DONE]\n\n")] + [expert-body + (string-append + "data: {\"choices\":[{\"delta\":{\"content\":\"expert replacement\"},\"finish_reason\":null}]}\n\n" + "data: {\"choices\":[{\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n" + "data: [DONE]\n\n")] + [cfg (make-hashtable equal-hash equal?)] + [expert-config (make-hashtable equal-hash equal?)] + [providers (make-hashtable equal-hash equal?)] + [openrouter-config (make-hashtable equal-hash equal?)]) + (hashtable-set! expert-config "provider" "openrouter") + (hashtable-set! expert-config "model" "expert-stream-unit") + (hashtable-set! openrouter-config "base_url" expert-url) + (hashtable-set! openrouter-config "api_key" "unit-key") + (hashtable-set! providers "openrouter" openrouter-config) + (hashtable-set! cfg "expert" expert-config) + (hashtable-set! cfg "providers" providers) + (dynamic-wind + (lambda () (void)) + (lambda () + (serve-one-captured-sse! primary-srv primary-captured primary-body) + (serve-one-captured-sse! expert-srv expert-captured expert-body) + (let ([primary (make-provider "mlx" "" "primary-local-unit" primary-url)]) + (parameterize ([*config* cfg] + [current-expert-disabled #f] + [current-defer-truncated-escalation #t]) + (let-values (((content _calls _usage) + (stream-chat-with-expert + primary (list (make-user-message "write now")) '() + (lambda (_token) (void))))) + (check-pred! "local first truncation preserves primary analysis" + content + (lambda (s) (and (string? s) + (str-contains? s "useful analysis")))) + (check! "local first truncation defers expert replacement" + (vector-ref expert-captured 0) #f))))) + (lambda () + (tcp-close primary-srv) + (tcp-close expert-srv)))) + ;; Guessed paths during repository exploration are normal recovery steps for a ;; local model. They should not consume expert budget as a tool-error streak, ;; while real tool failures still should. @@ -2896,24 +2960,107 @@ [(2) (list (make-wtool-call "edit" (list (cons "path" target) (cons "content" "repaired\n")) #f))] - [(3) (list (make-wtool-call "verify" '() #f))] - [else (list (make-wtool-call "done" - (list (cons "summary" "fixed")) #f))]))]) + [else (error 'test "automatic local verify should avoid another provider call")]))]) (safe-delete-test-file! target-path) - (verified-run responder "repair with a smaller local completion cap" - (list - (cons 'cwd vr-dir) - (cons 'verify-command (string-append "grep -q repaired " target)) - (cons 'write-scope (parse-write-scope target)) - (cons 'local-model? #t) - (cons 'max-iterations 8))) + (let ([result + (verified-run responder "repair with a smaller local completion cap" + (list + (cons 'cwd vr-dir) + (cons 'verify-command (string-append "grep -q repaired " target)) + (cons 'write-scope (parse-write-scope target)) + (cons 'local-model? #t) + (cons 'max-iterations 8)))]) + (check! "verified-run: local edit is automatically verified" + result "VERIFIED: exit 0\n")) (let ([observed (reverse caps)]) (check! "verified-run: local first-draft completion cap is 8k" (car observed) 8192) - (check! "verified-run: local post-verifier repair cap is 4k" - (cadr observed) 4096) - (check! "verified-run: local post-edit verify turn stays at 4k" - (caddr observed) 4096)) + (check! "verified-run: local post-verifier repair cap is 2k" + (cadr observed) 2048)) + (check! "verified-run: automatic verify avoids a provider round" + calls 2) + (safe-delete-test-file! target-path)) + + (let* ([vr-dir "/tmp"] + [target "jcode-local-dirty-preflight.ss"] + [target-path (string-append vr-dir "/" target)] + [initial "(import (jerboa prelude))\n(def rule \"B3 / S23\")\n"] + [calls 0] + [first-tool-names '()] + [first-history ""] + [tool-results '()] + [responder + (lambda (messages specs _step) + (set! calls (+ calls 1)) + (when (= calls 1) + (set! first-tool-names (map tool-spec-name specs)) + (set! first-history + (apply string-append + (map (lambda (m) + (string-append (message-content m) "\n")) + messages)))) + (check! "verified-run: dirty preflight repair uses 2k cap" + (current-max-tokens-cap) 2048) + (case calls + [(1) + ;; Simulate a stale full-write call from a provider that did + ;; not honor the narrowed exact-edit schema. + (list (make-wtool-call "edit" + (list (cons "path" target) + (cons "content" "(import (jerboa prelude))\n")) #f))] + [(2) + (list (make-wtool-call "edit" + (list (cons "path" target) + (cons "old_str" "B3 / S23") + (cons "new_str" "B3/S23")) #f))] + [else (error 'test "dirty local repair should finish after exact edit")]))] + [slurp (lambda (p) + (call-with-input-file p (lambda (i) (get-string-all i))))]) + (safe-delete-test-file! target-path) + (call-with-output-file target-path + (lambda (p) (display initial p)) 'replace) + (let ([result + (verified-run responder "finish a dirty local workspace" + (list + (cons 'cwd vr-dir) + (cons 'verify-command + (string-append + "grep -q 'B3/S23' " target + " || { echo 'FAIL: spaced rule input B3 / S23'; exit 1; }")) + (cons 'write-scope (parse-write-scope target)) + (cons 'local-model? #t) + (cons 'initial-verify? #t) + (cons 'max-iterations 8) + (cons 'on-message + (lambda (m) + (when (equal? (message-role m) "tool") + (set! tool-results + (cons (message-content m) tool-results)))))))]) + (check! "verified-run: dirty preflight exact repair verifies" + result "VERIFIED: exit 0\n")) + (check-pred! "verified-run: focused local repair hides broad write" + first-tool-names + (lambda (names) + (and (member "edit" names) + (member "line_edit" names) + (not (member "write" names))))) + (check-pred! "verified-run: dirty preflight highlights literal failing input" + first-history + (lambda (s) + (and (str-contains? s "Literal failure evidence") + (str-contains? s "FAIL: spaced rule input B3 / S23")))) + (check-pred! "verified-run: stale local full rewrite is rejected" + (reverse tool-results) + (lambda (xs) + (let loop ([rest xs]) + (and (pair? rest) + (or (str-contains? (car rest) "full-file rewrite disabled") + (loop (cdr rest))))))) + (check! "verified-run: dirty preflight uses two provider repair turns" + calls 2) + (check! "verified-run: exact repair preserves valid artifact" + (slurp target-path) + "(import (jerboa prelude))\n(def rule \"B3/S23\")\n") (safe-delete-test-file! target-path)) (let* ([vr-dir "/tmp"] @@ -8739,6 +8886,25 @@ (putenv "JCODE_MAX_TOKENS" (or old-max-tokens "")) (tcp-close srv)))) +(let* ([sse-body + "data: {\"choices\":[{\"index\":0,\"delta\":{\"reasoning_content\":\"partial\"},\"finish_reason\":null}]}\n\n"] + [srv (tcp-listen "127.0.0.1" 0)] + [base-url (format "http://127.0.0.1:~a/v1" (tcp-server-port srv))] + [captured (vector #f)]) + (dynamic-wind + (lambda () (void)) + (lambda () + (serve-one-captured-sse! srv captured sse-body) + (let* ([p (make-provider "mlx" "" "unit-test-model" base-url)] + [err (guard (e [#t e]) + (provider-stream-chat + p (list (make-user-message "hi")) '() + (lambda (_token) (void))) + #f)]) + (check! "mlx incomplete stream raises instead of accepting reasoning" + (and err (provider-retryable-error? err)) #t))) + (lambda () (tcp-close srv)))) + (let* ([sse-body (string-append "data: {\"id\":\"chatcmpl-test\",\"object\":\"chat.completion.chunk\",\"created\":0,\"model\":\"unit\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"ok\"},\"finish_reason\":null}]}\n\n" "data: {\"id\":\"chatcmpl-test\",\"object\":\"chat.completion.chunk\",\"created\":0,\"model\":\"unit\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n"