updates
ober
7827f3c04dd96bd65c8ae78b3f76d3b9fbe27805
--- a/.jerbuild +++ b/.jerbuild @@ -2,10 +2,8 @@ ;; ;; jerbuild bundles Chez Scheme + the jerboa stdlib, so NO jerboa source ;; checkout or external Chez is required. The only external input is the native -;; Rust lib (libjerboa_native), built from vendor/jerboa-native-rs with just the -;; features jcode needs (tls + crypto) — see the `native-rs` Makefile target. -;; Platform link flags (frameworks, -lsqlite3, -lc++, ...) are passed by the -;; Makefile via `jerbuild build --os-libs "..."` (per-uname). +;; Rust lib (libjerboa_native), built from vendor/jerboa-native-rs with the +;; features jcode needs (tls + sqlite + crypto). ;; ;; Static FFI needs NO source patching: chez-sqlite.sls falls through to (void) ;; and tui-ffi.ss honors JERBOA_STATIC=1 (set by support/jcode-main.c), so the @@ -14,7 +12,7 @@ (entry "main-binary.ss") (output "jcode") (requires "cargo" "cc" "sqlite3 headers/libs" "vendor/jerboa-native-rs") -(notes "jerboa-native-rs is built with no default features and tls,crypto only. DuckDB is intentionally not required.") +(notes "jerboa-native-rs is built with no default features and tls,sqlite,crypto only. DuckDB is intentionally not required.") ;; do-binary-build auto-appends the bundled stdlib; list only project + vendor. (libdirs "lib" @@ -34,9 +32,9 @@ "vendor/chez-sqlite/chez_sqlite_shim.c" ("src/jcode/ui/jcode_tui_shim.c" cflags: "-DTB_OPT_ATTR_W=32 -Ivendor/termbox2")) -;; jerboa-native (Rust): tls + crypto only (no duckdb/pcap/postgres). jerbuild +;; jerboa-native (Rust): tls + sqlite + crypto only (no duckdb/pcap/postgres). jerbuild ;; runs cargo and links the resulting libjerboa_native.a. (rust-crates ("vendor/jerboa-native-rs/Cargo.toml" - features: "tls,crypto" + features: "tls,sqlite,crypto" no-default-features: #t)) new file mode 100644 --- /dev/null +++ b/docs/add-grok.md @@ -0,0 +1,322 @@ +# Plan: Add Grok CLI-token support + +Audience: Opus 4.7 implementing this in `jerboa-code`. + +Goal: add first-class Grok support that can reuse an existing Grok CLI login by +reading `~/.grok` state, especially `~/.grok/auth.json`, instead of requiring +the user to copy an API key into `jcode`. + +## Current State + +`jcode` already has an `xai` provider: + +- Provider id: `xai` +- Endpoint: `https://api.x.ai/v1` +- Auth: `XAI_API_KEY` or a stored `xai` key +- Wire format: OpenAI chat completions +- Default model: `grok-3-mini` + +That should remain unchanged. The requested support is different: the Grok CLI +can authenticate with browser/OIDC login and stores session credentials under +`~/.grok`. That token is used against the Grok CLI chat proxy, whose local model +cache currently describes the default model as: + +```json +{ + "model": "grok-build", + "base_url": "https://cli-chat-proxy.grok.com/v1", + "api_backend": "responses", + "auth_scheme": "bearer", + "context_window": 512000 +} +``` + +Add a separate provider id, `grok`, so users can choose between: + +- `xai`: console API-key access to `api.x.ai` +- `grok`: Grok CLI session/API-key access to the CLI chat proxy + +## Observed Grok Files + +Do not commit or print secret values. The local redacted auth shape is: + +```json +{ + "https://auth.x.ai::<uuid>": { + "key": "<access token>", + "auth_mode": "<mode>", + "create_time": "<timestamp>", + "user_id": "<id>", + "email": "<email>", + "refresh_token": "<refresh token>", + "expires_at": "<timestamp>", + "oidc_issuer": "<issuer>", + "oidc_client_id": "<client id>" + } +} +``` + +Relevant files: + +- `~/.grok/auth.json`: browser/OIDC session token. Use the `key` field as the + Bearer token. +- `~/.grok/models_cache.json`: model metadata, including `base_url`, + `api_backend`, `auth_scheme`, `context_window`, and display name. +- `~/.grok/config.toml`: user overrides. Initial support can skip TOML parsing + unless it is already easy to use a local parser; prefer the JSON model cache + first. +- `~/.grok/docs/user-guide/02-authentication.md`: says auth precedence is API + key first, then OIDC refresh, then external provider, then browser login. +- `~/.grok/docs/user-guide/11-custom-models.md`: documents Grok's + `chat_completions`, `responses`, and `messages` API backends. + +## Design Decisions + +1. Keep `xai` and `grok` separate. + `xai` is a conventional API-key provider. `grok` is a compatibility provider + that mirrors the Grok CLI's session-backed default model. + +2. Read `~/.grok/auth.json` at runtime. + Do not copy the session token into `~/.jcode/keys.enc` by default. Grok + tokens expire and the CLI hot-reloads its auth file. Runtime lookup keeps + `jcode` aligned with `grok login`. + +3. Use env vars first. + For provider `grok`, resolve credentials in this order: + `XAI_API_KEY` -> `GROK_CODE_XAI_API_KEY` -> stored `grok` key in + `~/.jcode/keys.enc` -> plaintext config -> `~/.grok/auth.json`. + This mirrors Grok's API-key-first behavior while preserving `jcode`'s current + env/store/config order. + +4. Start with token reuse, not refresh-token refresh. + If `expires_at` is present and expired, return a clear error telling the user + to run `grok login`. Do not implement OIDC refresh in the first pass unless + it is straightforward and fully tested. + +5. Use the Grok model cache for provider metadata. + If `~/.grok/models_cache.json` is present, prefer its `grok-build` metadata + for default base URL, display name, context window, and API backend. + Fall back to hardcoded `grok-build`, + `https://cli-chat-proxy.grok.com/v1`, `responses`, and 512k context. + +6. Implement the Responses API path deliberately. + The observed `grok-build` cache uses `api_backend: "responses"`, so simply + adding `grok` to the existing OpenAI chat-completions case is likely wrong. + Add a small Responses adapter or a backend dispatch based on model metadata. + +## Files To Touch + +### `src/jcode/core/grok-auth.ss` + +Create a new module for Grok-specific local state: + +- `grok-home`: `~/.grok` +- `grok-auth-path`: `~/.grok/auth.json` +- `grok-models-cache-path`: `~/.grok/models_cache.json` +- `grok-auth-token`: returns a string token or `#f` +- `grok-auth-expired?`: conservative check from `expires_at` +- `grok-default-model-info`: returns a hash/alist with `model`, `base_url`, + `api_backend`, `auth_scheme`, `context_window`, and `name` +- `grok-provider-available?`: true when env/API key or auth token exists + +Parsing rules for `auth.json`: + +- Read JSON with `read-json`. +- Iterate top-level hash values. +- Prefer entries whose key starts with `https://auth.x.ai::` and whose value + has a non-empty string `key`. +- If several entries exist, prefer a non-expired entry. If timestamps are hard + to parse, prefer the first usable token and let the provider return 401. +- Never log the token, refresh token, email, user id, or principal id. + +Jerboa reminders: + +- User-facing source is `.ss`, not `.sls`. +- Import `(jerboa prelude)` only if writing standalone scripts; repo modules + already use `:jerboa/core` and `:jerboa/runtime`. +- Use `hash-get`, `hash-ref`, `hash-keys`, `string-prefix?`, + `string-contains`, and `path-join` with the arities used elsewhere in this + repo. +- Before writing Scheme code, use `jerboa_howto`, then verify APIs with + `jerboa_module_exports` / `jerboa_function_signature`. + +### `src/jcode/core/config.ss` + +Add `grok` to provider key resolution without breaking `xai`: + +- Extend `*provider-env-vars*` or add a special-case lookup for `grok`. +- Support both `XAI_API_KEY` and `GROK_CODE_XAI_API_KEY` for `grok`. +- In `config-get-provider-key`, after existing env/store/config lookup, call + `grok-auth-token` when provider is `"grok"`. +- In `config-detect-provider`, include `"grok"` late in the list so existing + configured API-key providers keep precedence. + +Avoid injecting the Grok token into the config hash in `merge-env-config`; that +would make accidental config printing/logging riskier. + +### `src/jcode/core/models.ss` + +Add provider metadata: + +- `all-providers`: include `"grok"` near `"xai"` but keep `"groq"` distinct. +- `provider-display-name`: `"Grok CLI"` or `"Grok Build"`. +- `provider-default-model`: `"grok-build"`. +- `provider-default-models`: return cached Grok models when + `~/.grok/models_cache.json` exists; otherwise: + +```scheme +'(("grok-build" . "Grok Build")) +``` + +- `model-context-window`: return 512000 for `"grok-build"` or load the cached + `context_window` from Grok model metadata if that is kept in memory. + +### `src/jcode/provider/provider.ss` + +Provider endpoint and dispatch: + +- `provider-default-url`: for `"grok"`, use the cached + `base_url` if available, else `https://cli-chat-proxy.grok.com/v1`. +- `provider-chat`, `provider-chat-with-stats`, and + `provider-stream-chat-with-stats`: dispatch `"grok"` based on backend. +- If the chosen model info says `api_backend` is `"chat_completions"`, the + existing OpenAI path is acceptable. +- If `api_backend` is `"responses"`, implement: + - non-streaming `responses-chat` + - streaming `responses-stream-chat` + - response parsing into `make-assistant-message` + - tool-call extraction if the endpoint emits tool calls + - usage parsing into the existing `(tokens-in tokens-out cost)` shape + +Responses adapter requirements: + +- Headers: `Content-Type: application/json` and + `Authorization: Bearer <token>`. +- URL: `{base_url}/responses`. +- Body: include `model`, translated conversation input, and tool definitions + when supported. +- Preserve system/developer/user/assistant/tool message semantics as closely as + the Responses API allows. +- Streaming should reuse `jcode-http-post-stream` and parse SSE events. +- Do not log raw Authorization headers; existing `redact-headers` already + handles them. + +Open questions for implementation: + +- Confirm the exact Responses payload accepted by `cli-chat-proxy.grok.com`. + Prefer a minimal request first, then add tools. +- Confirm whether Grok Build accepts OpenAI-style function tools, Responses + tool specs, or only its own agent tool protocol. +- If tool calling is not available through Responses initially, make that + limitation explicit and route `grok` through text-only chat first. + +### `src/jcode/core/secrets-import.ss` + +Optional but useful: + +- Add `"grok"` to `*known-providers*`. +- Add `import-from-grok!` only if the team wants an explicit + `jcode keys import grok`. +- If implemented, import only the current `key`, never `refresh_token`. +- Label the import as a session token that may expire, not a durable API key. + +Runtime `~/.grok/auth.json` lookup is still the primary path. + +### `src/jcode/tool/external-llm.ss` + +If the scope includes second-opinion tabs and `/ask-grok`, add Grok CLI support +in the external runner too: + +- `external-llm-providers`: add `grok`. +- `provider-spec`: use `("grok" "-p" prompt "--output-format" "plain" + "--permission-mode" "bypassPermissions")` if this matches local CLI behavior. +- `provider-spec-session`: use `grok -p prompt -s <session-id> + --output-format json` for multi-turn tabs, or `--resume <id>` if local + testing proves that is the better contract. +- Parser: JSON output has `text`, `stopReason`, `sessionId`, and `requestId`. +- Auth paths: allow writes to `~/.grok`; deny other providers' auth dirs. + +Then update `src/jcode/ui/tui.ss`: + +- Help text: add `/ask-grok`. +- Slash dispatch: include `ask-grok`. +- Tab dispatch: include `/grok`. +- `*external-providers*`: add `grok`. + +This CLI integration is independent of native provider support. Native provider +support is the priority because it lets Grok run through `jcode`'s own tools and +guardrails. + +### Docs + +Update after implementation: + +- `docs/providers.md`: add `grok` as a separate provider from `xai`. +- `docs/getting-started.md`: mention that `grok` can reuse `grok login`. +- `docs/tools.md`: if `/ask-grok` or Grok tabs are added, include `~/.grok` in + the external-LLM sandbox write-path description. +- `docs/cli.md`: update key import help only if `keys import grok` is added. + +## Test Plan + +Static/unit tests: + +- Add tests for parsing a sample redacted `auth.json` shape with one token. +- Add tests for no `~/.grok`, missing `auth.json`, empty `key`, and multiple + auth entries. +- Add tests for `models_cache.json` parsing and fallback metadata. +- Verify `config-get-provider-key "grok"` returns env/store/config before the + `~/.grok` token. +- Verify `all-providers`, `provider-display-name`, and provider switch UI include + `grok`. + +Provider tests: + +- Add `("grok" . "grok-build")` to `test/run-providers.ss`. +- The test should skip when no `XAI_API_KEY`, `GROK_CODE_XAI_API_KEY`, stored + `grok` key, or usable `~/.grok/auth.json` token exists. +- For live smoke, ask for a short response with no tools first. +- Add a tool-call smoke only after Responses tool calling is confirmed. + +Manual tests: + +1. Run `grok login` if needed. +2. Run `jcode config` and confirm provider key status does not print the token. +3. Run `jcode --provider grok --model grok-build -p "Say hello in five words."` +4. Run `jcode --tui`, switch `/provider grok`, and send a short prompt. +5. Run with `--trace /tmp/jcode-grok.trace` and confirm Authorization is + redacted. +6. If external CLI support was added, test `/ask-grok` and `/grok` tab resume. + +Jerboa verification: + +- After any `.ss` edit, run `jerboa_verify` on touched files. +- Then run `jerboa_make` for the repo. +- If edits appear stale, run the stale-artifact cleanup from `AGENTS.md`. +- Run `jerboa_security_scan` for the auth-file and HTTP changes because they + touch file I/O and bearer-token handling. + +## Security Requirements + +- Never print or trace `key`, `refresh_token`, `email`, `user_id`, + `principal_id`, or `team_id`. +- Redact Authorization headers in all traces. Existing redaction should cover + this, but add a regression check. +- Do not write `~/.grok` tokens into project-local `jcode.json`. +- Do not import `refresh_token` into `~/.jcode/keys.enc`. +- Do not let external CLIs read each other's auth directories. If adding Grok to + `external-llm.ss`, update both allow and deny path lists. +- On expired or rejected token, fail with: `Run grok login to refresh + ~/.grok/auth.json`, not with a raw API error containing provider internals. + +## Acceptance Criteria + +- `jcode --provider grok --model grok-build -p "hello"` works after an existing + `grok login`, with no manual API-key copy. +- `jcode` still supports `xai` exactly as before. +- `/provider` shows Grok as a distinct provider. +- `/refresh-models` either handles Grok models or skips with a clear message. +- Traces and logs redact all Grok credentials. +- The test suite passes, and live provider smoke skips cleanly when no Grok auth + exists. +