failures
ober
7351e0e0611c5edc77c4c1c8ad0e41e14b5a62ab
--- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ **A portable AI coding agent written in [Jerboa](https://git.sr.ht/~lisp/jerboa) Scheme.** `jcode` is a terminal coding agent — think opencode / aider / Claude Code — that -compiles to a standalone binary without a Node or Python runtime. It talks to 13 +compiles to a standalone binary without a Node or Python runtime. It talks to 14 LLM providers (cloud and local), drives your editor through a real tool loop, and ships a reliability layer that makes even small self-hosted models call tools dependably. @@ -68,7 +68,7 @@ make binary # produce the standalone ./jcode | **[Getting started](docs/getting-started.md)** | Prerequisites, building on every platform, configuration, keys, first run. | | **[CLI reference](docs/cli.md)** | Every subcommand, flag, slash command, and environment variable. | | **[Architecture](docs/architecture.md)** | The agent loop, module map, and how a turn flows through the guardrails. | -| **[Providers & models](docs/providers.md)** | The 13 providers, model registry, per-model sampling, and hardware tiers. | +| **[Providers & models](docs/providers.md)** | The 14 providers, model registry, per-model sampling, and hardware tiers. | | **[Tools](docs/tools.md)** | The agent's toolbox (file, bash, web, git, patch, task, MCP) and the safety model. | | **[Named agents & plans](docs/agents.md)** | Sub-agent roles with enforced write scopes, per-agent models, gated execution, and the plan-lifecycle skills. | | **[TUI](docs/tui.md)** | Layout, keybindings, themes, and the rendering features. | @@ -89,7 +89,7 @@ Run `make help` for the complete list. main.ss entry point src/jcode/ core/ agent loop, config, sessions, guardrails plumbing, ATLAS - provider/ the 13 LLM providers + per-model sampling + provider/ the 14 LLM providers + per-model sampling tool/ file, bash, web, git, patch, task, repomap, lsp guardrails/ rescue, validator, step-enforcer, respond, error budget proxy/ OpenAI-compatible guardrail proxy --- a/docs/escalation.md +++ b/docs/escalation.md @@ -159,14 +159,16 @@ The expert response replaces the primary response. If the expert call fails, ## Provider availability `jcode` requests and parses logprobs on its OpenAI-compatible provider path. -That includes OpenAI-compatible cloud providers as well as local `mlx` and -`ollama` when they expose compatible fields. - -Current Ollama documentation lists `logprobs` and `top_logprobs` support on its -native `/api/chat` and `/api/generate` APIs, and lists logprobs as supported on -the OpenAI-compatible `/v1/chat/completions` API. Since `jcode` uses Ollama's -OpenAI-compatible endpoint, Ollama can provide the same kind of confidence data -as MLX when the installed Ollama version returns those fields. +That includes OpenAI-compatible cloud providers as well as local `mlx` when they +expose compatible fields. + +Ollama defaults to native `/api/chat` so `jcode` can send `keep_alive` and keep +local prompt/KV state resident across repeated test loops. Native Ollama +streaming currently contributes usage counters (`prompt_eval_count` and +`eval_count`) but not jcode logprob confidence stats. If you explicitly set +`providers.ollama.wire` to `"openai"`, Ollama uses the OpenAI-compatible path +and can provide the same kind of confidence data as MLX when the installed +Ollama version returns those fields. Anthropic streaming does not expose token logprobs through the path `jcode` uses, so only non-logprob escalation signals apply there. Google currently uses --- a/docs/providers.md +++ b/docs/providers.md @@ -22,13 +22,15 @@ an existing `~/.grok/auth.json` session token). | `together` | OpenAI-compatible | `https://api.together.xyz/v1` | Bearer | | `cerebras` | OpenAI-compatible | `https://api.cerebras.ai/v1` | Bearer | | `perplexity` | OpenAI-compatible | `https://api.perplexity.ai` | Bearer | -| `ollama` | OpenAI-compatible (local) | `http://localhost:11434/v1` | none | +| `ollama` | Native Ollama chat (local) | `http://localhost:11434/v1` (`/api/chat` for generation) | none | | `mlx` | OpenAI-compatible (local) | `http://127.0.0.1:8080/v1` | none | -The non-Anthropic, non-Google, non-Grok providers all share the OpenAI -chat-completions code path, so adding a new OpenAI-compatible endpoint is mostly -a table entry. `grok` speaks the **OpenAI Responses API** to the Grok CLI proxy -and is dispatched separately — see [Grok CLI session](#grok-cli-session) below. +The non-Anthropic, non-Google, non-Grok, non-Ollama providers all share the +OpenAI chat-completions code path, so adding a new OpenAI-compatible endpoint is +mostly a table entry. Ollama uses its native `/api/chat` endpoint for generation +so local prompt residency controls are available. `grok` speaks the **OpenAI +Responses API** to the Grok CLI proxy and is dispatched separately — see +[Grok CLI session](#grok-cli-session) below. ## Models @@ -70,6 +72,23 @@ Run `ollama serve`, then point `jcode` at it: No key required. `jcode` probes your GPU to size the context budget (see [hardware tiers](#hardware-tiers)). +Ollama chat defaults to native `/api/chat`, even though the configured base URL +stays `http://localhost:11434/v1` for compatibility with model listing and old +configs. To keep the model and its reusable prompt/KV state resident across +test loops, `jcode` sends: + +```json +{ + "providers": { + "ollama": { "keep_alive": "30m" } + } +} +``` + +Set a longer value such as `"24h"` for sustained local coding sessions, or set +`"wire": "openai"` under `providers.ollama` if you explicitly need Ollama's +OpenAI-compatible `/v1/chat/completions` path. + ### Local models — MLX Serve a model with `mlx_lm.server` (default `http://127.0.0.1:8080/v1`), then: @@ -95,17 +114,37 @@ every OpenAI-compatible endpoint: model families, `jcode` also asks for `prompt_cache_retention: "24h"`; older model families stay on OpenAI's default retention so requests do not fail on unsupported cache-retention options. +- **Google / Gemini**: Gemini 2.5+ implicit caching is automatic server-side. + `jcode` records `usageMetadata.cachedContentTokenCount` as `cache-read` for + both implicit and explicit cache hits. If you create an explicit Gemini cache + with `cachedContents.create`, set the handle in config and `jcode` will pass it + as `cachedContent`: + + ```json + { + "providers": { + "google": { "cached_content": "cachedContents/..." } + } + } + ``` + + Explicit Gemini caches can have storage billing, so `jcode` does not create + them automatically. - **MLX / omlx**: sends the same stable `prompt_cache_key` on the local OpenAI-compatible path and records `usage.prompt_tokens_details.cached_tokens` as `cache-read` when the server reports it. Stock `mlx_lm.server` and `omlx` already do prefix-cache matching server-side; the cache key is a routing hint for compatible frontends and is ignored by servers that do not need it. -- **Ollama**: the `/v1/chat/completions` compatibility endpoint is strict and - does not document `prompt_cache_key`, so `jcode` deliberately omits it instead - of risking 4xx failures. Keep model residency and Ollama KV-cache settings on - the Ollama side (`OLLAMA_KEEP_ALIVE`, `OLLAMA_KV_CACHE_TYPE`, etc.) while - `jcode` keeps using `/v1` for tool-call parity. +- **Ollama**: uses native `/api/chat` by default and sends `keep_alive` + (default `"30m"`, configurable under `providers.ollama.keep_alive`) so repeated + jcode/opencode-style test loops keep the model resident instead of unloading + between turns. Ollama does not expose an OpenAI-style `prompt_cache_key`, so + `jcode` does not send one. Usage is recorded from native + `prompt_eval_count`/`eval_count`; Ollama currently does not report a separate + cached-token count, so `cache-read` is `0` even when server-side prefix/KV + reuse reduces latency. For explicit compatibility testing, set + `providers.ollama.wire` to `"openai"`. ### Grok CLI session --- a/src/jcode/provider/provider.ss +++ b/src/jcode/provider/provider.ss @@ -21,6 +21,7 @@ message->responses-input responses-body responses-headers + google-usage->alist grok-backend grok-backend-from grok-list-models) @@ -1268,6 +1269,10 @@ ;;; Google Gemini API ;;; (def (google-chat provider messages tools) + (let-values (((message _usage) (google-chat-with-usage provider messages tools))) + message)) + +(def (google-chat-with-usage provider messages tools) (let* ((url (string-append (provider-base-url provider) "/models/" (provider-model provider) @@ -1285,13 +1290,21 @@ (log-trace logger "google-response" `((status . ,status) (body . ,text)))) (if (= status 200) - (google-parse-response (string->json-object text)) + (let ((json (string->json-object text))) + (values (google-parse-response json) + (google-usage->alist provider (hash-get json "usageMetadata")))) (error 'google-chat (format "API error ~a: ~a" status text)))))) (def (google-body provider messages tools) (let ((body (make-hash-table)) (system-msg (find-system-message messages)) - (other-msgs (remove-system-messages messages))) + (other-msgs (remove-system-messages messages)) + (cached-content (config-ref "providers" "google" "cached_content"))) + (when (and (string? cached-content) + (> (string-length cached-content) 0)) + ;; Gemini explicit context caching uses cachedContent handles created via + ;; cachedContents.create. Implicit caching remains automatic on Gemini 2.5+. + (hash-put! body "cachedContent" cached-content)) (when system-msg (let ((si (make-hash-table)) (part (make-hash-table))) @@ -1378,7 +1391,32 @@ (restore-tool-call name name args)))) fn-calls))))) -;;; Ollama API (OpenAI-compatible) ;;; +(def (json-number-or-zero x) + (if (number? x) x 0)) + +(def (google-usage->alist provider usage) + "Convert Gemini usageMetadata into jcode's common usage shape. Gemini reports + cachedContentTokenCount for both implicit and explicit cache hits; prompt + token count includes cached tokens, matching OpenAI's accounting model." + (let* ((u (if (and usage (hash-table? usage)) usage (make-hash-table))) + (prompt (json-number-or-zero (hash-get u "promptTokenCount"))) + (cached (json-number-or-zero (hash-get u "cachedContentTokenCount"))) + (candidates (json-number-or-zero (hash-get u "candidatesTokenCount"))) + (thoughts (json-number-or-zero (hash-get u "thoughtsTokenCount"))) + (out (+ candidates thoughts)) + (cost-usage (make-hash-table)) + (details (make-hash-table))) + (hash-put! details "cached_tokens" cached) + (hash-put! cost-usage "prompt_tokens" prompt) + (hash-put! cost-usage "completion_tokens" out) + (hash-put! cost-usage "prompt_tokens_details" details) + (list (cons 'tokens-in prompt) + (cons 'tokens-out out) + (cons 'cache-read cached) + (cons 'cache-creation 0) + (cons 'cost (compute-cost (provider-model provider) cost-usage))))) + +;;; Ollama API ;;; (def (ollama-tools-present? tools) (and tools (not (null? tools)))) @@ -1404,17 +1442,220 @@ (substring msg 0 240) msg))) +(def (trim-trailing-slash s) + (if (and (> (string-length s) 1) + (string-suffix? "/" s)) + (trim-trailing-slash (substring s 0 (- (string-length s) 1))) + s)) + +(def (ollama-native-base-url provider) + ;; Config still defaults to http://localhost:11434/v1 for model-listing and + ;; old compatibility. Native chat uses the same host without the /v1 suffix. + (let ((base (trim-trailing-slash (provider-base-url provider)))) + (if (string-suffix? "/v1" base) + (substring base 0 (- (string-length base) 3)) + base))) + +(def (ollama-openai-compat?) + (let ((wire (config-ref "providers" "ollama" "wire"))) + (and (string? wire) + (or (equal? wire "openai") + (equal? wire "openai-compatible") + (equal? wire "v1"))))) + +(def (ollama-keep-alive) + ;; Ollama unloads models after a short idle window by default. Keep the model + ;; resident across jcode/opencode-style test loops so the local server can + ;; reuse its own prefix/KV cache instead of paying full prefill every turn. + (let ((v (config-ref "providers" "ollama" "keep_alive"))) + (cond + ((or (string? v) (number? v)) v) + (else "30m")))) + +(def (copy-hash-key! dst src key) + (when (hash-key? src key) + (hash-put! dst key (hash-ref src key)))) + +(def (copy-hash-key-as! dst src src-key dst-key) + (when (hash-key? src src-key) + (hash-put! dst dst-key (hash-ref src src-key)))) + +(def (ollama-sampling-options provider) + (let ((openai-style (make-hash-table)) + (opts (make-hash-table))) + (apply-sampling-to-body! openai-style (provider-model provider) (provider-name provider)) + (copy-hash-key! opts openai-style "temperature") + (copy-hash-key! opts openai-style "top_p") + (copy-hash-key! opts openai-style "top_k") + (copy-hash-key! opts openai-style "min_p") + (copy-hash-key-as! opts openai-style "repetition_penalty" "repeat_penalty") + (let ((configured (config-ref "providers" "ollama" "options"))) + (when (hash-table? configured) + (hash-for-each (lambda (k v) (hash-put! opts k v)) configured))) + (if (pair? (hash-keys opts)) opts #f))) + +(def (ollama-native-body provider messages tools stream?) + (let ((body (make-hash-table))) + (hash-put! body "model" (provider-model provider)) + (hash-put! body "messages" (map message->json messages)) + (hash-put! body "stream" stream?) + (hash-put! body "keep_alive" (ollama-keep-alive)) + (let ((opts (ollama-sampling-options provider))) + (when opts (hash-put! body "options" opts))) + (when (and tools (not (null? tools)) + (not (model-rejects-tools? (provider-model provider)))) + (hash-put! body "tools" tools)) + body)) + +(def (ollama-native-headers) + '(("Content-Type" . "application/json"))) + +(def (ollama-json-arguments args) + (cond + ((string? args) args) + ((hash-table? args) (json-object->string args)) + (else "{}"))) + +(def (ollama-native-tool-call tc idx) + (let* ((fn (and (hash-table? tc) (hash-get tc "function"))) + (name (and (hash-table? fn) (hash-get fn "name"))) + (args (and (hash-table? fn) (hash-get fn "arguments"))) + (id (and (hash-table? tc) (hash-get tc "id")))) + (and name + (restore-tool-call + (or id (format "call_~a" idx)) + name + (ollama-json-arguments args))))) + +(def (ollama-native-tool-calls msg) + (let ((tcs (and (hash-table? msg) (hash-get msg "tool_calls")))) + (if (and tcs (list? tcs)) + (let loop ((xs tcs) (idx 0) (out '())) + (cond + ((null? xs) (reverse out)) + (else + (let ((tc (ollama-native-tool-call (car xs) idx))) + (loop (cdr xs) (+ idx 1) + (if tc (cons tc out) out)))))) + '()))) + +(def (ollama-usage->alist provider usage) + (let* ((u (if (and usage (hash-table? usage)) usage (make-hash-table))) + (prompt (json-number-or-zero (hash-get u "prompt_eval_count"))) + (out (json-number-or-zero (hash-get u "eval_count"))) + (prompt-dur (json-number-or-zero (hash-get u "prompt_eval_duration"))) + (eval-dur (json-number-or-zero (hash-get u "eval_duration"))) + (load-dur (json-number-or-zero (hash-get u "load_duration"))) + (total-dur (json-number-or-zero (hash-get u "total_duration")))) + (list (cons 'tokens-in prompt) + (cons 'tokens-out out) + (cons 'cache-read 0) + (cons 'cache-creation 0) + (cons 'cost 0) + (cons 'prompt-eval-duration prompt-dur) + (cons 'eval-duration eval-dur) + (cons 'load-duration load-dur) + (cons 'total-duration total-dur)))) + +(def (ollama-native-stream-chat provider messages tools token-cb) + ;; Native /api/chat streams newline-delimited JSON objects, not SSE. The + ;; HTTP streamer already hands us complete lines, so each line is parsed as + ;; one Ollama event. + (let* ((url (string-append (ollama-native-base-url provider) "/api/chat")) + (headers (ollama-native-headers)) + (body (ollama-native-body provider messages tools #t)) + (text-acc (open-output-string)) + (tool-calls-box (box '())) + (usage-acc (make-hash-table)) + (finish-reason-box (box #f))) + (let* ((body-json (json-object->string body)) + (dummy (begin + (log-info logger "ollama-native-stream-request" + `((url . ,url) (body-len . ,(string-length body-json)))) + (when (tracing?) + (log-trace logger "ollama-native-stream-request" + `((url . ,(redact-url url)) + (headers . ,(redact-headers headers)) + (body . ,body-json)))))) + (http-status + (jcode-http-post-stream url headers body-json + (lambda (line) + (when (and line (> (string-length (string-trim line)) 0)) + (let ((json (guard (e [(error? e) #f]) + (string->json-object line)))) + (when (and json (hash-table? json)) + (let ((err (hash-get json "error"))) + (when err + (error 'ollama-native-stream-chat "~a" err))) + (let ((msg (hash-get json "message"))) + (when (hash-table? msg) + (let ((content (hash-get msg "content"))) + (when (and content (string? content) + (> (string-length content) 0)) + (put-string text-acc content) + (token-cb content))) + (let ((tcs (ollama-native-tool-calls msg))) + (when (pair? tcs) + (set-box! tool-calls-box + (append (unbox tool-calls-box) tcs)))))) + (when (hash-get json "done") + (hash-for-each + (lambda (k v) (hash-put! usage-acc k v)) + json) + (let ((reason (hash-get json "done_reason"))) + (set-box! finish-reason-box + (if (and reason (string? reason)) reason "stop"))))))))))) + (unless (= http-status 200) + (log-error logger "ollama-native-stream-http-error" + `((status . ,http-status) (url . ,url))))) + (let* ((raw-content (get-output-string text-acc)) + (recovery (call-with-values + (lambda () (extract-text-tool-calls raw-content)) + cons)) + (content (car recovery)) + (text-extras (cdr recovery)) + (tool-calls (append (unbox tool-calls-box) text-extras)) + (usage (ollama-usage->alist provider usage-acc))) + (when (pair? text-extras) + (log-info logger "tool-call-text-recovery" + `((count . ,(length text-extras)) + (names . ,(map tool-call-name text-extras))))) + (log-info logger "ollama-native-stream-result" + `((content-len . ,(string-length content)) + (tool-calls . ,(length tool-calls)) + (tokens-in . ,(cdr (assoc 'tokens-in usage))) + (tokens-out . ,(cdr (assoc 'tokens-out usage))))) + (values content tool-calls usage + (build-stats (unbox finish-reason-box) '() '()))))) + +(def (ollama-native-chat provider messages tools) + (let-values (((content tcs usage stats) + (ollama-native-stream-chat provider messages tools + (lambda (token) #f)))) + (if (null? tcs) + (make-assistant-message content) + (make-assistant-message content tcs)))) + +(def (ollama-openai-chat provider messages tools) + (openai-chat provider messages tools)) + +(def (ollama-openai-stream-chat provider messages tools token-cb) + (openai-stream-chat provider messages tools token-cb)) + (def (ollama-chat provider messages tools) - ;; Ollama exposes an OpenAI-compatible /v1/chat/completions endpoint (guard (e [#t (if (and (ollama-tools-present? tools) (ollama-tool-template-error? e)) (begin (log-warn logger "ollama-tool-template-fallback" `((mode . "chat") (err . ,(short-error-string e)))) - (openai-chat provider messages '())) + (if (ollama-openai-compat?) + (ollama-openai-chat provider messages '()) + (ollama-native-chat provider messages '()))) (raise e))]) - (openai-chat provider messages tools))) + (if (ollama-openai-compat?) + (ollama-openai-chat provider messages tools) + (ollama-native-chat provider messages tools)))) (def (ollama-stream-chat provider messages tools token-cb) (guard (e [#t @@ -1423,9 +1664,13 @@ (begin (log-warn logger "ollama-tool-template-fallback" `((mode . "stream") (err . ,(short-error-string e)))) - (openai-stream-chat provider messages '() token-cb)) + (if (ollama-openai-compat?) + (ollama-openai-stream-chat provider messages '() token-cb) + (ollama-native-stream-chat provider messages '() token-cb))) (raise e))]) - (openai-stream-chat provider messages tools token-cb))) + (if (ollama-openai-compat?) + (ollama-openai-stream-chat provider messages tools token-cb) + (ollama-native-stream-chat provider messages tools token-cb)))) ;;; OpenAI Streaming ;;; @@ -1777,12 +2022,14 @@ (else (grok-responses-stream-chat provider messages tools token-cb)))) ((anthropic) (anthropic-stream-chat provider messages tools token-cb)) ((google) - ;; Google: non-streaming fallback — no stats available. - (let* ((response (provider-chat provider messages tools)) - (content (or (message-content response) "")) - (tcs (or (message-tool-calls response) '()))) - (when (> (string-length content) 0) (token-cb content)) - (values content tcs '() (build-stats #f '() '())))) + ;; Google: non-streaming fallback, but preserve usageMetadata so prompt + ;; cache hits are visible in the TUI/cost accounting. + (let-values (((response usage) + (google-chat-with-usage provider messages tools))) + (let* ((content (or (message-content response) "")) + (tcs (or (message-tool-calls response) '()))) + (when (> (string-length content) 0) (token-cb content)) + (values content tcs usage (build-stats #f '() '()))))) (else (error 'provider-stream-chat-with-stats (format "Unknown provider '~a'" (provider-name provider)))))) --- a/test/run.ss +++ b/test/run.ss @@ -177,6 +177,27 @@ (close-port out) (close-port in))))))) +(define (serve-one-captured-json! srv captured-body status body) + (fork-thread + (lambda () + (let-values ([(in out) (tcp-accept srv)]) + (dynamic-wind + (lambda () (void)) + (lambda () + (vector-set! captured-body 0 (read-test-http-request in)) + (put-string out + (string-append + "HTTP/1.1 " (number->string status) " OK\r\n" + "Content-Type: application/json\r\n" + "Content-Length: " (number->string (string-length body)) "\r\n" + "Connection: close\r\n" + "\r\n" + body)) + (flush-output-port out)) + (lambda () + (close-port out) + (close-port in))))))) + ;; ── Setup ───────────────────────────────────────────────────────── (current-log-level 'warn) @@ -1990,14 +2011,70 @@ (grok-list-models (make-provider "grok" "" "grok-build" "https://x/v1")) (lambda (ms) (assoc "grok-build" ms))) +;; ── provider: Google prompt cache accounting ─────────────────────── + +(section "=== provider: Google prompt cache accounting ===") + +(let* ([body (string-append + "{\"candidates\":[{\"content\":{\"parts\":[{\"text\":\"hello\"}]}}]," + "\"usageMetadata\":{\"promptTokenCount\":100," + "\"cachedContentTokenCount\":75," + "\"candidatesTokenCount\":4," + "\"thoughtsTokenCount\":6," + "\"totalTokenCount\":110}}")] + [srv (tcp-listen "127.0.0.1" 0)] + [base-url (format "http://127.0.0.1:~a/v1beta" (tcp-server-port srv))] + [captured (vector #f)] + [cfg (make-hashtable equal-hash equal?)] + [providers (make-hashtable equal-hash equal?)] + [google (make-hashtable equal-hash equal?)]) + (hashtable-set! google "cached_content" "cachedContents/unit") + (hashtable-set! providers "google" google) + (hashtable-set! cfg "providers" providers) + (dynamic-wind + (lambda () (void)) + (lambda () + (serve-one-captured-json! srv captured 200 body) + (let* ([p (make-provider "google" "unit-key" "gemini-2.5-flash" base-url)] + [result (call-with-values + (lambda () + (parameterize ([*config* cfg]) + (provider-stream-chat + p + (list (make-user-message "hi")) + '() + (lambda (token) #f)))) + (lambda (content tcs usage) (list content tcs usage)))] + [req (vector-ref captured 0)] + [usage (list-ref result 2)]) + (check! "google stream cache reply" (car result) "hello") + (check! "google explicit cachedContent forwarded" + (and req + (str-contains? req "\"cachedContent\"") + (str-contains? req "cachedContents/unit")) + #t) + (check! "google stream tokens-in" + (cdr (assoc 'tokens-in usage)) 100) + (check! "google stream tokens-out includes thoughts" + (cdr (assoc 'tokens-out usage)) 10) + (check! "google stream cache-read" + (cdr (assoc 'cache-read usage)) 75) + (check! "google stream cache-creation zero" + (cdr (assoc 'cache-creation usage)) 0) + (check-pred! "google stream cache cost positive" + (cdr (assoc 'cost usage)) + (lambda (x) (and (number? x) (> x 0)))))) + (lambda () (tcp-close srv)))) + ;; ── provider: streaming HTTP error bodies ───────────────────────── (section "=== provider: streaming HTTP error bodies ===") -(let* ([sse-body (string-append - "data: {\"id\":\"chatcmpl-test\",\"object\":\"chat.completion.chunk\",\"created\":0,\"model\":\"unit\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"ok\"},\"finish_reason\":null}]}\n\n" - "data: {\"id\":\"chatcmpl-test\",\"object\":\"chat.completion.chunk\",\"created\":0,\"model\":\"unit\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n" - "data: [DONE]\n\n")] +(let* ([ndjson-body (string-append + "{\"model\":\"unit\",\"message\":{\"role\":\"assistant\",\"content\":\"ok\"},\"done\":false}\n" + "{\"model\":\"unit\",\"done\":true,\"prompt_eval_count\":100,\"eval_count\":2," + "\"prompt_eval_duration\":123,\"eval_duration\":456,\"load_duration\":7," + "\"total_duration\":586}\n")] [srv (tcp-listen "127.0.0.1" 0)] [base-url (format "http://127.0.0.1:~a/v1" (tcp-server-port srv))] [captured (vector #f)] @@ -2019,27 +2096,39 @@ (dynamic-wind (lambda () (void)) (lambda () - (serve-one-captured-sse! srv captured sse-body) + (serve-one-captured-json! srv captured 200 ndjson-body) (let* ([p (make-provider "ollama" "" "unit-test-model" base-url)] - [reply (call-with-values - (lambda () - (parameterize ([*config* cfg]) - (provider-stream-chat - p - (list (make-user-message "hi")) - (list tool) - (lambda (token) #f)))) - (lambda (content tcs usage) content))] + [result (call-with-values + (lambda () + (parameterize ([*config* cfg]) + (provider-stream-chat + p + (list (make-user-message "hi")) + (list tool) + (lambda (token) #f)))) + (lambda (content tcs usage) (list content tcs usage)))] [req (vector-ref captured 0)]) - (check! "ollama stream tools reply" reply "ok") - (check! "ollama stream tools include tools" + (check! "ollama native stream tools reply" (car result) "ok") + (check! "ollama native stream uses keep_alive" + (and req (str-contains? req "\"keep_alive\"")) #t) + (check! "ollama native stream uses native stream flag" + (and req (str-contains? req "\"stream\":true")) #t) + (check! "ollama native stream tools include tools" (and req (str-contains? req "\"tools\"")) #t) - (check! "ollama stream tools omit logprobs" + (check! "ollama native stream tools omit logprobs" (and req (str-contains? req "\"logprobs\"")) #f) - (check! "ollama stream tools omit top_logprobs" + (check! "ollama native stream tools omit top_logprobs" (and req (str-contains? req "\"top_logprobs\"")) #f) - (check! "ollama stream omits unsupported prompt cache key" - (and req (str-contains? req "\"prompt_cache_key\"")) #f))) + (check! "ollama native stream omits OpenAI stream options" + (and req (str-contains? req "\"stream_options\"")) #f) + (check! "ollama native stream omits unsupported prompt cache key" + (and req (str-contains? req "\"prompt_cache_key\"")) #f) + (check! "ollama native stream tokens-in" + (cdr (assoc 'tokens-in (list-ref result 2))) 100) + (check! "ollama native stream tokens-out" + (cdr (assoc 'tokens-out (list-ref result 2))) 2) + (check! "ollama native stream cache-read unavailable" + (cdr (assoc 'cache-read (list-ref result 2))) 0))) (lambda () (tcp-close srv)))) (let* ([sse-body (string-append