Save Jerboa discovery knowledge
ober
31c256d8479dcf63a782d9e4c34a95d0263cf94d
--- a/data/cookbooks.sexp +++ b/data/cookbooks.sexp @@ -5158,4 +5158,100 @@ "command-protocol") ("title" . - "Expose a machine-readable JSON command surface for editor frontends"))) + "Expose a machine-readable JSON command surface for editor frontends")) + (("code" + . + "#!/usr/bin/env jerboa\n;;; `jerboa <script.ss>` runs a file directly; this shebang does the same.\n(import (chezscheme))\n\n;; (command-line-arguments) INCLUDES the script name as the first element:\n;; jerboa foo.ss a b -> (\"foo.ss\" \"a\" \"b\")\n;; so drop the head to get real args. ((command-line) also prepends the\n;; interpreter's own prog path.)\n(define argv (cdr (command-line-arguments)))\n(for-each (lambda (a) (display a) (newline)) argv)") ("id" . "jerboa-script-interpreter-bundled-subset") + ("imports") + ("notes" + . + "The STANDALONE `jerboa` interpreter bundles only a SUBSET of (std ...) — not the whole stdlib. Confirmed resolvable from a script: (jerboa prelude), (std misc string), (chezscheme). Confirmed NOT bundled: (std os aproc), (std net tcp) — importing them raises `library (std os aproc) not found`. So for subprocess/socket/OS work in a standalone script, fall back to (chezscheme) kernel primitives (open-process-ports, file ports) — see [[open-process-ports-full-duplex]]. Import form is the PAREN form `(std misc string)`, NOT the colon `:std/misc/string` form (the colon form is only for jerbuild-compiled project source). Probe availability fast with a one-liner: `printf '(import (std X Y))(display 1)' > p.ss && jerboa p.ss`.") + ("tags" "shebang" "command-line-arguments" "interpreter" + "script" "std-subset" "library-not-found") + ("title" + . + "Standalone jerboa scripts: shebang, args, and the bundled (std ...) subset")) + (("code" + . + "(import (chezscheme))\n\n;; Returns FOUR values: stdin-port, stdout-port, stderr-port, pid.\n;; A transcoder makes them textual (get-line / put-string); omit it for\n;; binary ports. The command runs via /bin/sh -c, so shell redirections work.\n(call-with-values\n (lambda ()\n (open-process-ports\n \"exec timeout 3 some-interactive-cmd 2>/dev/null\" ;; exec: pid IS the cmd\n (buffer-mode block)\n (make-transcoder (utf-8-codec))))\n (lambda (to-in from-out from-err pid)\n (put-string to-in \"a line of input\\n\")\n (flush-output-port to-in)\n (let ((out (get-string-all from-out))) ;; blocks until EOF\n (display out))\n (close-port to-in)\n (close-port from-out)\n (close-port from-err)))") ("id" . "open-process-ports-full-duplex") ("imports") + ("notes" + . + "Value ORDER is (stdin stdout stderr pid) — stdout is the 2nd value, not the 1st. Two gotchas this solves: (1) when the child keeps the pipe open and never EOFs (a server, `openssl s_client`, a REPL), `get-string-all`/blocking reads hang forever — wrap the command in `timeout N` so it self-terminates and the pipe hits a clean EOF, avoiding the need for non-blocking reads. (2) Use `exec` in the command so the spawned pid IS the program (not a wrapping /bin/sh), so `timeout`/kill act on it directly. This is the go-to for subprocess work when the standalone interpreter lacks (std os aproc) — see [[jerboa-script-interpreter-bundled-subset]]. For interactive duplex (send commands, read replies) without an EOF, either rely on the `timeout` bound or do non-blocking reads on the port's fd.") + ("tags" "open-process-ports" "subprocess" "pipe" + "full-duplex" "chezscheme" "timeout") + ("title" + . + "Full-duplex subprocess I/O with open-process-ports (4 values; bound reads with timeout)")) + (("code" + . + ";; Chez's call-with-output-file uses (error) file-options by default:\n;; writing to a path that already exists raises\n;; \"failed for <path>: file exists\"\n;; Delete the stale file first when rewriting caches/outputs:\n(def (write-text-file path text)\n (when (file-exists? path) (delete-file path))\n (call-with-output-file path\n (lambda (p) (display text p))))") ("id" . "overwrite-file-delete-first") ("imports") + ("notes" + . + "Bit jcode's repomap cache: first write succeeded, every rewrite failed with [WARN] cache-write-failed. The try/catch around it masked the bug for weeks — the cache silently never refreshed. Alternative is passing (file-options no-fail) in raw Chez, but delete-first is portable across the jerboa prelude's wrappers.") + ("tags" "call-with-output-file" "overwrite" "file-exists" + "delete-file" "output" "cache") + ("title" + . + "Overwrite an existing file (call-with-output-file fails with 'file exists')")) + (("code" + . + "(def (deep-merge! base overlay)\n ;; Merge OVERLAY into BASE per-key. When both sides hold a hash the\n ;; merge recurses; otherwise OVERLAY's value replaces BASE's.\n (hash-for-each\n (lambda (k v)\n (let ((bv (hash-get base k)))\n (if (and (hash-table? bv) (hash-table? v))\n (deep-merge! bv v)\n (hash-put! base k v))))\n overlay)\n base)\n\n;; Layered config files, lowest precedence first:\n(let loop ((paths (reverse (config-paths))) (acc (make-hash-table)))\n (cond\n ((null? paths) acc)\n ((file-exists? (car paths))\n (loop (cdr paths)\n (deep-merge! acc (call-with-input-file (car paths) read-json))))\n (else (loop (cdr paths) acc))))") ("id" . "hash-deep-merge") ("imports") + ("notes" + . + "hash-for-each takes (proc hash) in that order. Mutates base in place — pass a fresh (make-hash-table) as the accumulator when layering. Used to fix jcode config: picking the FIRST existing config file meant a project file silently dropped every global key (mcpServers etc); per-key merge keeps global values unless the project file overrides them.") + ("tags" "hash-table" "deep-merge" "merge" "config" "nested" + "hash-for-each") + ("title" + . + "Deep-merge two hash tables per-key (nested hashes merge recursively)")) + (("code" + . + "(import (jerboa prelude))\n\n(def *esc-char* (integer->char #x1b))\n(def *bel-char* (integer->char #x07))\n\n(def (ansi-final-byte? ch)\n (let ((n (char->integer ch)))\n (and (>= n #x40) (<= n #x7e))))\n\n(def (skip-csi text pos len)\n (let loop ((i pos))\n (cond\n ((>= i len) len)\n ((ansi-final-byte? (string-ref text i)) (+ i 1))\n (else (loop (+ i 1))))))\n\n(def (skip-osc text pos len)\n (let loop ((i pos))\n (cond\n ((>= i len) len)\n ((char=? (string-ref text i) *bel-char*) (+ i 1))\n ((and (char=? (string-ref text i) *esc-char*)\n (< (+ i 1) len)\n (char=? (string-ref text (+ i 1)) #\\\\))\n (+ i 2))\n (else (loop (+ i 1))))))\n\n(def (strip-terminal-escapes text)\n (let ((out (open-output-string))\n (len (string-length text)))\n (let loop ((i 0))\n (when (< i len)\n (let ((ch (string-ref text i)))\n (cond\n ((char=? ch *esc-char*)\n (let ((next (+ i 1)))\n (cond\n ((>= next len) (loop next))\n ((char=? (string-ref text next) #\\[)\n (loop (skip-csi text (+ next 1) len)))\n ((char=? (string-ref text next) #\\])\n (loop (skip-osc text (+ next 1) len)))\n (else (loop (min len (+ i 2)))))))\n ((or (< (char->integer ch) #x20)\n (= (char->integer ch) #x7f))\n (if (or (char=? ch #\\newline) (char=? ch #\\tab))\n (begin (write-char ch out) (loop (+ i 1)))\n (loop (+ i 1))))\n (else\n (write-char ch out)\n (loop (+ i 1)))))))\n (get-output-string out)))\n\n(def sample\n (string-append \"Error: \"\n (string *esc-char*) \"[91m\"\n (string *esc-char*) \"[1mbad\"\n (string *esc-char*) \"[0m ok\"))\n\n(displayln (strip-terminal-escapes sample))\n;; => Error: bad ok\n") ("id" . "strip-terminal-escapes-before-tui-render") + ("imports" "(jerboa prelude)") + ("notes" + . + "Useful at a TUI rendering boundary when captured subprocess output or external CLI errors may contain SGR, CSI, OSC, or other terminal control bytes. Preserve newline and tab, drop other C0/DEL controls, and decode/render the cleaned text instead of feeding raw terminal escape sequences to a renderer.") + ("tags" "tui" "ansi" "terminal" "control-sequences" + "open-output-string" "string-loop") + ("title" + . + "Strip ANSI and terminal control sequences before TUI rendering")) + (("code" + . + "(import (jerboa prelude))\n\n(def (condition->string e)\n (with-output-to-string (lambda () (display-condition e))))\n\n(def (tool-template-error? e)\n (let ((msg (condition->string e)))\n (and (string-contains msg \"API error 500\")\n (or (string-contains msg \"unexpected EOF\")\n (string-contains msg \"tool_call\")\n (and (string-contains msg \"closed by\")\n (or (string-contains msg \"function\")\n (string-contains msg \"parameter\")))))))\n\n(def (call-with-tool-fallback call-with-tools call-without-tools tools)\n (guard (e [#t\n (if (and tools (not (null? tools)) (tool-template-error? e))\n (call-without-tools)\n (raise e))])\n (call-with-tools)))") ("id" . "ollama-tool-template-fallback") + ("imports" "(jerboa prelude)") + ("notes" + . + "Ollama's OpenAI-compatible endpoint may parse model-generated tool calls through an XML-like template before client-side recovery can run. For specific HTTP 500 parse failures, retrying without the structured tools field lets prompt-level/text tool-call recovery handle the response. Keep the predicate narrow and re-raise unrelated errors.") + ("tags" "ollama" "openai-compatible" "tool-calls" "guard" + "fallback" "provider") + ("title" + . + "Retry Ollama tool-template XML 500s without structured tools")) + (("code" + . + "(import (jerboa prelude)\n :std/text/json\n :jcode/core/message)\n\n(def tc (make-tool-call \"read\" \"{\\\"path\\\":\\\"README.md\\\"}\"))\n(def msg (make-assistant-message #f (list tc)))\n(def json (message->json msg))\n\n(assert! (string=? (hash-ref json \"role\") \"assistant\"))\n(assert! (string=? (hash-ref json \"content\") \"\"))\n(assert! (pair? (hash-ref json \"tool_calls\")))\n(displayln (json-object->string json))") ("id" . "openai-assistant-tool-call-empty-content") + ("imports" + "(jerboa prelude)" + ":std/text/json" + ":jcode/core/message") + ("notes" + . + "jcode may store assistant tool-call turns with internal content #f. Some OpenAI-compatible servers, including Ollama, reject a request message whose content is omitted/null with errors such as \"invalid message content type: <nil>\". Preserve the internal #f representation, but serialize assistant/tool messages without content as an explicit empty string.") + ("tags" "jcode/core/message" "message->json" "ollama" + "openai-compatible" "tool-calls" "content") + ("title" + . + "Serialize Assistant Tool Calls With Empty Content for OpenAI-Compatible Providers")) + (("code" + . + "(import (jerboa prelude)\n (std text json))\n\n(def h (make-hash-table))\n(hash-put! h \"ok\" #t)\n(displayln (json-object->string h))") ("id" . "plain-chez-repl-module-paths") + ("imports" "(jerboa prelude)" "(std text json)") + ("notes" + . + "Jerboa .ss files can use reader-extension module paths like :std/text/json. A plain Chez new-cafe REPL launched through jerbuild exec does not read those extensions at the prompt, so :std/text/json is treated as an unknown module. Use normal parenthesized library names such as (std text json), (jerboa prelude), or (jcode core message) in that REPL.") + ("tags" "chezscheme" "new-cafe" "jerbuild-exec" + "reader-extension" "module-import" "colon-path") + ("title" + . + "Use Parenthesized Module Paths in Plain Chez REPLs"))) --- a/data/features.sexp +++ b/data/features.sexp @@ -1311,4 +1311,164 @@ ("use_case" . "Building standalone Jerboa applications that use jsqlite, jerboa-native crypto/tls, or other FFI-backed modules.") + ("votes" . 0)) + (("description" + . + "jerboa_verify and jerboa_compile_check crashed internally with `Exception in string-ref: <idx> is not a valid index for \"<file contents>\"` while checking edited large files including test/run.ss, src/jcode/ui/tui.ss, and src/jcode/ui/tui-input.ss. The files compiled and tests passed via make, so the verifier should return a normal diagnostic or success result instead of crashing and dumping large file contents.") + ("estimated_token_reduction" + . + "500-1500 tokens per failure by avoiding fallback commands and huge file-content exception output.") + ("example_scenario" + . + "During the jcode stream-error and /copy command work, MCP verification repeatedly crashed on edited files, requiring fallback to direct make test and make binary despite the code being valid.") + ("id" . "verify-large-file-string-ref-crash") + ("impact" . "medium") + ("tags" + "verify" + "compile-check" + "large-files" + "diagnostics") + ("title" + . + "Fix jerboa_verify string-ref crash on large files") + ("use_case" + . + "Validating edited Jerboa files before running the full build/test cycle.") + ("votes" . 0)) + (("description" + . + "jerboa-code's CLAUDE.md mandates jerboa_balanced_replace for all .ss edits (validate paren balance before/after, dry-run preview), but the current jmcp build doesn't expose it — ToolSearch finds neither balanced_replace nor check_balance, in hybrid surface or behind the dispatcher. Agents must fall back to plain Edit and hand-count parens, then jerboa_check_syntax on extracted snippets.") + ("estimated_token_reduction" + . + "Eliminates 1-2 verification round-trips per .ss edit (~800 tokens each); prevents broken-balance build failures that cost a full make cycle to surface.") + ("example_scenario" + . + "This session rewrote load-config in jcode's config.ss (15 lines, 4 nesting levels). Project instructions require balanced_replace dry-run → apply → check_balance; none of the three calls were available, so the edit went through generic Edit with manual paren counting and a separate check_syntax on a hand-extracted form.") + ("id" . "balanced-replace-tool") ("impact" . "high") + ("tags" "edit" "balance" "parens" "mcp" "tooling") + ("title" + . + "Expose jerboa_balanced_replace + jerboa_check_balance via jmcp") + ("use_case" + . + "Every multi-line edit to .ss source in any jerboa repo — the mandated safe-edit workflow is currently impossible to follow.") + ("votes" . 0)) + (("description" + . + "jerbuild source files start with a bare (export ...) + (import :pkg/path ...) that jerbuild wraps into a library at build time. jerboa_compile_check feeds the raw file to the expander and dies with 'export form outside of a module or library', so per-file compile validation is impossible in jerbuild projects like jcode — the only check is a full project build. compile_check could detect the leading export form and synthesize the library wrapper the way jerbuild does.") + ("estimated_token_reduction" + . + "Replaces a full build cycle (or blind commit) per edit with one tool call; ~2-5k tokens saved per validation round.") + ("example_scenario" + . + "After editing src/jcode/core/config.ss, jerboa_compile_check file_path=... failed on the file's own (export load-config ...) header. Fallback was jerboa_check_syntax on a hand-pasted snippet, which validates syntax but not bindings/arity.") + ("id" . "compile-check-jerbuild-modules") + ("impact" . "medium") + ("tags" "compile-check" "jerbuild" "export" "module" + "validation") + ("title" + . + "jerboa_compile_check should handle jerbuild-style modules (bare top-level export)") + ("use_case" + . + "Validating a single edited src file in any jerbuild-managed project without paying a multi-minute full build (or in jcode's case `make binary`, since `make build` deadlocks).") + ("votes" . 0)) + (("description" + . + "Allow jerboa_verify and jerboa_compile_check to handle repository source files that use the project/transpiler top-level .ss style with import/export forms, instead of reporting \"export form outside of a module or library\". The tool should resolve project_path/local libdirs and run the same wrapper/transpile context as make build where practical.") + ("estimated_token_reduction" + . + "~800-1500 tokens per affected edit by avoiding failed verifier debugging and reducing fallback build iterations.") + ("example_scenario" + . + "While editing src/jcode/ui/tui-message.ss, jerboa_verify failed with \"export form outside of a module or library\" even though make build transpiled and compiled the file successfully.") + ("id" . "verify-top-level-project-ss-files") + ("impact" . "medium") + ("tags" "verify" "compile-check" "project-path" + "top-level-ss" "exports") + ("title" + . + "Verify top-level project .ss files with exports") + ("use_case" + . + "Agents editing Jerboa project source files need a fast per-file verifier before falling back to a full project build.") + ("votes" . 0)) + (("description" + . + "Provide a tool or test helper that renders TUI components into a deterministic terminal buffer or screenshot-like text/bitmap artifact and checks borders, separators, reserved rows, and overlap. It should support targeted component fixtures such as sidebars/status bars and whole-screen smoke captures.") + ("estimated_token_reduction" + . + "~1000 tokens per TUI iteration by replacing repeated screenshot inspection and manual layout reasoning with targeted visual assertions.") + ("example_scenario" + . + "A manually inspected screenshot showed the sidebar section dividers and bottom edge were uneven. Build/tests passed, but only visual review exposed the missing bottom border and inconsistent separator rows.") + ("id" . "tui-screenshot-layout-regression-harness") + ("impact" . "medium") + ("tags" "tui" "screenshot" "layout" "regression" "termbox") + ("title" . "TUI screenshot and layout regression harness") + ("use_case" + . + "Agents working on terminal UI need a repeatable way to catch visual defects like uneven borders, stale cells, and status/sidebar overlap that normal Scheme tests miss.") + ("votes" . 0)) + (("description" + . + "jerboa_verify and jerboa_compile_check can fail internally with `Exception in string-ref: <n> is not a valid index` on large source files, dumping nearly the entire file instead of reporting syntax or compile diagnostics. The tool should guard index math in its pre-scan and return a concise internal-tool-error with file size, failing index, and a fallback recommendation.") + ("estimated_token_reduction" + . + "~10k-50k tokens per failed invocation on large files") + ("example_scenario" + . + "Running jerboa_verify on src/jcode/provider/provider.ss after a small provider fallback patch raised `Exception in string-ref: 97675 is not a valid index` and printed most of the file, forcing fallback to make build.") + ("id" . "verify-large-file-string-ref-boundary") + ("impact" . "medium") + ("tags" "verify" "compile_check" "large-file" "diagnostics") + ("title" + . + "Handle large-file boundary errors in jerboa_verify") + ("use_case" + . + "When checking a large Jerboa source file after a small edit, users need actionable syntax/compile diagnostics rather than a tool-internal string-ref exception.") + ("votes" . 0)) + (("closed_reason" . "") + ("description" + . + "jerboa_verify and jerboa_compile_check currently report \"export form outside of a module or library\" for project .ss files that start with top-level export forms, even though jerbuild transpile/build accepts the same files and compiles them into libraries. The tools should use the same project-aware wrapping/transpile path as the build for user-facing .ss files.") + ("estimated_token_reduction" + . + "~300-800 tokens per single-file verification attempt") + ("example_scenario" + . + "After editing src/jcode/core/message.ss, jerboa_verify and jerboa_compile_check failed immediately on the leading export form, while jerboa_make build successfully transpiled and compiled the file. The session had to fall back to make build/test for compile validation.") + ("id" . "verify-top-level-export-ss-files") + ("impact" . "medium") ("implemented_in" . "") + ("implemented_tool" . "") ("status" . "open") + ("tags" "verify" "compile-check" "export" "project-ss" + "jerbuild") + ("title" + . + "Allow verify and compile_check on top-level .ss export files") + ("use_case" + . + "Validate a single changed Jerboa source file before running a full project build.") + ("votes" . 0)) + (("closed_reason" . "") + ("description" + . + "jerboa_howto returned no results for queries containing terms that appear in existing recipe titles and tags, while jerboa_howto_get by exact ID retrieved the recipes. Cookbook search should strongly weight exact matches in recipe IDs, titles, and tags, and ideally expose an exact-id lookup fallback in search results.") + ("estimated_token_reduction" + . + "~200-600 tokens per save-discoveries pass") + ("example_scenario" + . + "Queries such as \"Ollama OpenAI compatible assistant tool_calls empty content nil\" and \"Ollama tool template malformed XML fallback without tools\" returned no results, even though recipes openai-assistant-tool-call-empty-content and ollama-tool-template-fallback existed with overlapping title/tag terms.") + ("id" . "howto-search-exact-tag-title-matches") + ("impact" . "medium") ("implemented_in" . "") + ("implemented_tool" . "") ("status" . "open") + ("tags" "howto" "cookbook" "search" "tags" "ranking") + ("title" + . + "Improve howto search exact tag and title matching") + ("use_case" + . + "Avoid duplicate cookbook entries and quickly confirm whether a known pattern has already been saved.") ("votes" . 0)))