Implement Jerboa AI-authorship scanner MVP
ober
e6743352f77c7365d4cf5b01b82999dda1a75d49
new file mode 100644 --- /dev/null +++ b/.builds/ci.yml @@ -0,0 +1,12 @@ +image: alpine/latest +packages: + - git +sources: + - https://git.sr.ht/~lisp/jerboa-aigit +tasks: + - test: | + cd jerboa-aigit + echo "Install jerboa before running CI tests on this worker." + command -v jerboa + make verify + new file mode 100644 --- /dev/null +++ b/.gitignore @@ -0,0 +1,8 @@ +/.jcode/ +/.jerboa/ +/dist/ +/tmp/ +/jerboa-aigit +*.so +*.wpo + new file mode 100644 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,560 @@ +## STOP: Editing `.ss`/`.sls` Files — Mandatory Rules + +These rules exist because local-model sessions have lost **hours** fighting +parenthesis imbalances that these rules would have prevented in seconds. +They override every habit from other editors and languages. + +1. **NEVER use `edit`, `write`, `sed`, `python`, `perl`, or `awk` to modify + `*.ss` or `*.sls` files.** Use the jerboa-mcp tools instead: + - Add a top-level form → `jerboa_balanced_insert` (anchor = one unique + complete form, e.g. the `def` above the insertion point). + - Replace exact text → `jerboa_balanced_replace`. It is **dry-run by + default** — pass `dry_run: false` to actually write. + - Create a whole new file → `jerboa_write_file` (use `verify: true` to + reject unbalanced content before it lands). +2. **After EVERY `.ss` change, run `jerboa_check_balance` before building.** + The balanced tools check automatically; if you bypassed them, check now. +3. **If a file ever becomes unbalanced, STOP.** Do NOT count parens by hand, + do NOT write paren-counting scripts, do NOT poke one character at a time. + Recovery is exactly one of: + - `git checkout -- <file>` and redo the edit with `jerboa_balanced_insert` + (preferred — one command, seconds), or + - `jerboa_repair_balance` (dry-run shows the plan; `apply: true` writes). +4. **Keep closer-runs short.** Never hand-write code that ends in more than + ~4 consecutive closers (`))))]` runs). Flatten deep nesting with helper + `def`s, `let*`, or cond `=>` clauses so no edit ever depends on counting + a long `)` run. +5. **`(def ...)` after an expression in a body is invalid.** Internal defines + must come first in a body, or use `let`/`let*`. The build error + "invalid context for definition" means you violated this — or a missing + paren above glued two top-level forms together (check balance first). + +## The Jerboa Language — Quick Reference + +Jerboa is a Scheme dialect built on Chez Scheme. It is Gerbil-inspired but its own language. **All user-facing code is `.ss` files. Never write `.sls` files for the user** — those are internal implementation files. + +### File Structure + +Every Jerboa file looks like this: + +```scheme +(import (jerboa prelude)) ;; ONE import gives you the ENTIRE language +;; Optional extra imports for modules NOT in the prelude: +;; (import (std net request)) + +(def (my-function x y) + (+ x y)) + +(displayln (my-function 1 2)) +``` + +Run with: `~/.local/bin/jerboa run file.ss` (or `jerboa run file.ss` if `~/.local/bin` is on your PATH). + +**NEVER** write `(library ...)` forms — that's `.sls` internal syntax. + +### Reader Syntax Extensions + +``` +[...] → plain parentheses — same as Gerbil and Chez Scheme +{method obj args} → (~ obj 'method args) — method dispatch +name: → keyword #:name +:std/sort → (std sort) — Gerbil-style module path +#<<END ... END → heredoc string +``` + +Square brackets `[...]` are interchangeable with `(...)`, exactly like Gerbil and stock Chez Scheme. You can freely use them in bindings, match clauses, and anywhere you'd use parentheses: +```scheme +;; All of these are correct: +(let ([x 1] [y 2]) (+ x y)) +(for/collect ([x (in-range 5)]) (* x x)) +(match val ([list a b] (+ a b))) +(cond [(> x 0) "positive"] [else "non-positive"]) +``` + +### CRITICAL: Things That DO NOT EXIST in Jerboa/Chez + +Claude frequently hallucinates these from Gerbil, Gambit, Racket, or R7RS training data. +**NONE of them are real in Jerboa/Chez. STOP and use the correct form.** + +#### AI Compatibility Aliases (these now work in the prelude) +The following names from other Scheme dialects are aliased in `(jerboa prelude)`: +- `hash-has-key?` → `hash-key?` (Racket) +- `hash-table-set!` → `hash-put!` (Racket) +- `directory-exists?` → `file-directory?` (Gambit) +- `eql?` → `eqv?` (Common Lisp) +- `random-integer` → `random` (Gambit) +- `read-line` → `get-line` wrapper (Gambit) — works with or without port arg +- `force-output` → `flush-output-port` wrapper (Gambit) — works with or without port arg +- `string-map` → char-level map (Racket/R7RS) — `(string-map f str)` + +#### Hallucinated Functions (still do NOT exist) +- `symbol<?` — use `(lambda (a b) (string<? (symbol->string a) (symbol->string b)))` +- `string-contains?` — use `(string-contains str sub)` (returns index or #f, NOT boolean) +- `define-struct` — use `(defstruct name (fields ...))` +- `raise` with a string — use `(error 'who "message" irritants ...)` +- `environment-bound?` — Gerbil-only. No direct Chez equivalent + +#### Gerbil/Gambit-isms (from training data — wrong in Jerboa) +- `time->seconds` — use `(time-second (current-time))` for epoch seconds +- `thread-sleep!` — Gambit. Use `(sleep (make-time 'time-duration 0 seconds))` +- `thread-yield` — no Chez equivalent. Use `(sleep (make-time 'time-duration 0 0))` as workaround +- `path-expand` with 2 args — Gerbil takes `(path-expand rel base)`. Jerboa takes 1 arg. Use `(path-join base rel)` for 2-arg version +- `process-status` — Gerbil. Use `(std misc process)` API in Jerboa +- `user-info-home` — Gerbil. Use `(getenv "HOME")` +- `the-environment` — Gerbil. Use `(interaction-environment)` in Chez +- `condition/report-string` — Gerbil. Use `(with-output-to-string (lambda () (display-condition c)))` +- `make-class-type` — Gerbil. Use `(defstruct ...)` or `(defclass ...)` in Jerboa +- `string-subst` — Gerbil. Not in prelude. Use `(string-replace str old new)` or implement manually +- `open-fd-pair` — Gambit. Does not exist in Chez; requires different API + +#### R6RS/Racket-isms (wrong variant) +- `make-equal-hashtable` — R6RS. Use `(make-hash-table)` from Jerboa prelude +- `arithmetic-shift` — Racket. Use `(bitwise-arithmetic-shift n k)` or `(ash n k)` in Chez +- `pregexp-match` — Racket. Use `(std text regex)` or `(std pregexp)` API in Jerboa + +### CRITICAL: Common Arity Mistakes + +- `(list-of? pred)` → returns a PREDICATE. It takes 1 arg. Use: `((list-of? number?) lst)` +- `(maybe pred)` → returns a PREDICATE. It takes 1 arg. Use: `((maybe string?) val)` +- `(in-range end)` or `(in-range start end)` or `(in-range start end step)` — NOT `(in-range start step end)` +- `(hash-ref ht key)` or `(hash-ref ht key default)` — NOT `(hash-ref key ht)` +- `(string-split str delimiter)` where delimiter is a CHAR: `(string-split "a,b" #\,)` +- `(make-rwlock)` — takes **0 args**, NOT `(make-rwlock 'name)` (Gerbil takes a name; Jerboa does not) +- `(path-expand path)` — takes **1 arg**, NOT `(path-expand rel base)` (Gerbil takes 2; use `path-join` for 2-arg) +- `(sort list predicate)` — Jerboa `(std sort)`/prelude order. Raw Chez `sort` is predicate-first, but Jerboa-facing code should use list first. + +### Core Forms (all from `(import (jerboa prelude))`) + +#### Definitions +```scheme +(def x 42) ;; variable +(def (f x y) (+ x y)) ;; function +(def (f x (y 10)) body) ;; optional param with default +(def (f x . rest) body) ;; rest args +(def* f ((x) ...) ((x y) ...)) ;; multi-arity +(defrule (name pat) template) ;; macro +``` + +#### Data Structures +```scheme +(defstruct point (x y)) ;; → make-point, point?, point-x, point-y, point-x-set! +(defstruct (circle shape) (radius)) ;; inheritance (single only) +(defmethod (area (self circle)) body) ;; method on type +(~ obj 'method arg ...) ;; dispatch (or {method obj arg ...}) +(defrecord person (name age)) ;; struct + pretty-print + ->alist +(define-enum color (red green blue)) ;; → color-red, color?, color->name +``` + +#### Pattern Matching +```scheme +(match value + (42 "exact") ;; literal + ((list a b c) (+ a b c)) ;; list destructure + ((cons h t) h) ;; pair + ((? number?) "num") ;; predicate + ((? string? s) (string-upcase s)) ;; predicate + bind + ((and (? number?) (? positive?)) "positive number") + ((or "yes" "y") #t) + ((=> string->number n) n) ;; view pattern + (n (where (> n 0)) "positive") ;; guard + (_ "default")) ;; wildcard +``` + +#### Error Handling +```scheme +(try expr (catch (e) handler) (finally cleanup)) +(try expr (catch (error? e) handler)) +(unwind-protect body cleanup) +(with-resource (var init cleanup) body) +``` + +#### Result Type (Rust-inspired ok/err) +```scheme +(ok 42) (err "bad") (ok? r) (err? r) +(unwrap (ok 42)) ;; → 42 (raises on err) +(unwrap-or (err "x") 0) ;; → 0 +(map-ok f result) (map-err f result) +(and-then result f) ;; monadic bind +(try-result expr) ;; exceptions → (err condition) +(try-result* expr) ;; exceptions → (err "message string") +(sequence-results list-of-results) ;; → (ok list) or first (err) +(->? (ok 10) (+ 5) (* 2)) ;; → (ok 30), short-circuits on err +``` + +#### Iterators +```scheme +(for ((x (in-range 5))) (displayln x)) +(for/collect ((x (in-range 5))) (* x x)) ;; → (0 1 4 9 16) +(for/fold ((sum 0)) ((x (in-range 10))) (+ sum x)) ;; → 45 +(for/or ((x lst)) (and (pred? x) x)) ;; first truthy +(for/and ((x lst)) (pred? x)) ;; all truthy + +;; Iterators: in-list, in-vector, in-string, in-range, in-hash-keys, +;; in-hash-values, in-hash-pairs, in-naturals, in-indexed, +;; in-port, in-lines, in-chars, in-bytes, in-producer +``` + +#### Threading Macros +```scheme +(-> x (f a) (g b)) ;; thread first: (g (f x a) b) +(->> x (f a) (g b)) ;; thread last: (g b (f a x)) +(as-> x v (f v) (g v)) ;; named +(some-> x (f) (g)) ;; short-circuit on #f +(cond-> x test (f) t2 (g)) ;; conditional steps +(->? (ok x) (f) (g)) ;; result-aware thread first +``` + +#### Ergo Typing +```scheme +(: expr pred?) ;; checked cast +(using (p (make-point 1 2) : point?) + (+ p.x p.y)) ;; dot-access → (point-x p) etc. +((list-of? number?) '(1 2 3)) ;; predicate factory → #t +((maybe string?) #f) ;; accepts #f or string → #t +``` + +#### Hash Tables +```scheme +(def ht (make-hash-table)) +(hash-put! ht "key" "val") +(hash-ref ht "key") ;; error if missing +(hash-ref ht "key" "default") ;; with default +(hash-get ht "key") ;; → val or #f +(hash-key? ht "key") ;; → #t/#f +(hash-remove! ht "key") +(hash->list ht) (hash-keys ht) (hash-values ht) +(hash-for-each (lambda (k v) ...) ht) +(list->hash-table '(("a" . 1) ("b" . 2))) +``` + +#### Strings +```scheme +(string-split "a,b,c" #\,) ;; → ("a" "b" "c") NOTE: char delimiter +(string-join '("a" "b") ",") ;; → "a,b" +(string-trim " hi ") ;; → "hi" +(string-prefix? "he" "hello") ;; → #t +(string-suffix? "lo" "hello") ;; → #t +(string-contains "hello" "ell") ;; → 1 (index, not boolean!) +(string-empty? "") ;; → #t +(str "age: " 42 "!") ;; → "age: 42!" (auto-coerce) +``` + +#### Lists +```scheme +(flatten '(1 (2 (3)))) ;; → (1 2 3) +(unique '(1 2 2 3)) ;; → (1 2 3) +(take lst n) (drop lst n) (take-last lst n) (drop-last lst n) +(every pred lst) (any pred lst) (filter-map f lst) +(group-by f lst) (zip lst1 lst2) (frequencies lst) +(partition pred lst) (interleave l1 l2) (mapcat f lst) +(distinct lst) (keep f lst) (split-at lst n) +(append-map f lst) (snoc lst elem) +``` + +#### Functional Combinators +```scheme +(compose f g) (comp f g) ;; (f (g x)) +(partial f arg ...) ;; partial application +(complement pred) (negate pred) ;; logical not +(identity x) (constantly v) ;; basic combinators +(curry f arg) (flip f) ;; currying, arg swap +(conjoin p1 p2) (disjoin p1 p2) ;; predicate AND/OR +(juxt f g) ;; → (lambda (x) (list (f x) (g x))) +(cut f <> y) ;; SRFI-26 partial: (lambda (x) (f x y)) +``` + +#### JSON +```scheme +(string->json-object "{\"key\":\"val\"}") ;; → hash table +(json-object->string ht) ;; → JSON string +(read-json port) (write-json obj port) +``` + +#### CSV +```scheme +(csv->alists "name,age\nAlice,30") ;; → (((name . "Alice") (age . "30"))) +(read-csv-file "data.csv") ;; → list of row lists +(write-csv-file "out.csv" rows) +``` + +#### DateTime +```scheme +(datetime-now) (datetime-utc-now) +(make-datetime 2026 3 27 12 0 0) +(parse-datetime "2026-03-27T12:00:00Z") +(datetime->iso8601 dt) (datetime->epoch dt) +(datetime-add dt duration) (datetime-diff dt1 dt2) +(datetime<? dt1 dt2) (day-of-week dt) (leap-year? 2024) +``` + +#### Paths +```scheme +(path-join "/home" "user" "f.txt") ;; → "/home/user/f.txt" +(path-directory "/a/b/f.txt") ;; → "/a/b" +(path-extension "file.txt") ;; → "txt" +(path-absolute? "/home") ;; → #t +``` + +#### File I/O +```scheme +(read-file-string "f.txt") ;; → entire file +(read-file-lines "f.txt") ;; → list of lines +(write-file-string "f.txt" "data") +``` + +#### Pretty Printing +```scheme +(pp expr) (pp-to-string expr) (pprint expr) +``` + +#### Formatting +```scheme +(format "~a is ~a" "Alice" 30) ;; → "Alice is 30" +(printf "x = ~a\n" 42) +(displayln "hello" " " "world") +``` + +#### Anaphoric / Conditional Binding +```scheme +(awhen (find x) (use it)) ;; binds result to `it` +(aif (find x) (use it) (default)) +(when-let (x (find y)) (use x)) +(if-let (x (find y)) (use x) (default)) +``` + +#### Loops +```scheme +(while test body ...) +(until test body ...) +(dotimes (i 10) body ...) ;; 0..9 +``` + +#### Misc Sugar +```scheme +(assert! (> x 0)) (assert! (> x 0) "message") +(alist (name "Alice") (age 30)) ;; → ((name . "Alice") (age . 30)) +(let-alist data (name age) body) +``` + +### What's NOT in the Prelude (requires separate import) + +```scheme +(import (std net request)) ;; HTTP client +(import (std net httpd)) ;; HTTP server +(import (std db sqlite)) ;; SQLite +(import (std actor)) ;; Actor system +(import (std async)) ;; Async/await +(import (std crypto digest)) ;; SHA, MD5 +(import (std text regex)) ;; Regex +(import (std text xml)) ;; XML +(import (std text yaml)) ;; YAML +(import (std os env)) ;; Environment variables +(import (std os signal)) ;; Signal handling +(import (std security sandbox)) ;; Sandboxing +``` + +### Chez Scheme Conflicts (handled by prelude) + +The prelude shadows these Chez builtins with Jerboa versions: +`make-hash-table`, `hash-table?`, `sort`, `sort!`, `printf`, `fprintf`, +`path-extension`, `path-absolute?`, `with-input-from-string`, +`with-output-to-string`, `iota`, `1+`, `1-`, `partition`, +`make-date`, `make-time` + +All standard Chez Scheme is still available — the prelude just re-exports +improved versions of the above. + +--- + +## Repository Boundaries + +When working in a Jerboa project, **ONLY modify files in the current repo** unless the user explicitly names another path. + +Common sibling repos that exist but must NOT be touched without explicit instruction: +- `~/mine/jerboa-emacs` — **NEVER touch**. Another model owns it. +- `~/mine/jerboa-mcp` — Legacy node MCP, superseded. The active MCP server now lives in THIS repo at `mcp/` + `data/`. Don't modify the legacy repo unless told. +- `~/mine/jerboa-shell` — Only modify when user explicitly says to work there. +- `~/mine/gerbil-mcp` — **NEVER touch**. Deprecated. +- `~/mine/gerbil-orig` — Read-only reference for upstream Gerbil. Never modify. + +If a user instruction mentions a file path, use EXACTLY that path. Do not substitute a similar-looking path from another repo. + +### Never Reference Sibling Checkouts in Build Files + +Build files (Makefile, shell scripts, CI config) must **never** resolve a +dependency via a relative sibling path (`../jerboa-foo`) or an absolute +`~/mine/jerboa-foo` path. That layout is specific to this one machine — +other users and CI do not have it. Always vendor instead: fetch/clone the +dependency into `vendor/` (or this repo's equivalent) at build time, or use +a pinned-release fetch script, so the build is reproducible without +assuming any sibling checkout exists. + +A sibling-path fallback is not just a portability bug: it can silently +substitute a full alternate source tree (build config, embedded data, +secrets) for the vendored one, with no equivalent safety default, changing +what actually gets built without any indication. If you find one +(`grep -rn '\.\./jerboa\|~/mine/jerboa'` over Makefiles/scripts), remove it +and vendor properly instead. + +--- + +## Build & Verification + +After modifying any `.ss` or `.sls` (Jerboa Scheme) source files, always run the build command and fix any errors before moving on. Common issues include: missing imports, wrong function names, and duplicate definitions. + +### Stale Artifacts + +If edits seem to have no effect after `make build`, delete stale compiled files: +```bash +find lib -name "*.so" -delete && find lib -name "*.wpo" -delete && make build +``` +Run `jerboa_stale_static` to detect stale `.so` files before debugging "why doesn't my edit work?". + +## Pre-commit Requirements + +**ALWAYS** run a clean build **before** committing any code to this repository. Pick the right target for the *current* platform: + +- **Linux**: run `make docker-build` — the Docker image must build cleanly against the full musl-static release pipeline. +- **macOS / FreeBSD / other**: run `make binary` — the native local build must succeed. Do **not** run `make docker-build` here; Docker on non-Linux hosts is slow and not the canonical pipeline for those platforms. + +Do not commit if the build fails. + +## Act First, Read Less + +When making changes, read only what you need to make the edit, then make it. +Do not read more than 3 files before acting. Do not re-read files you already +read. Do not verify things you already know. If you have enough context to make +a change, make it. The user will interrupt you if you are wrong. + +## Jerboa MCP Tools — MANDATORY Usage + +Jerboa is a niche Scheme dialect with limited training data. **Never guess — always verify** with MCP tools. Tool descriptions are available at runtime via the MCP server; this section covers **when** and **why** to use each tool. + +### MANDATORY Workflow Order (for writing new code) + +1. **`jerboa_howto`** — BEFORE writing code, search cookbook for verified patterns +2. **`jerboa_module_exports`** / **`jerboa_function_signature`** — confirm APIs exist and check arities +3. Write the code +4. **`jerboa_verify`** — combined syntax + compile + lint + arity + duplicate check (use instead of individual tools) +5. **`jerboa_security_scan`** — for code involving FFI, shell commands, or file I/O + +Security fixes must update `data/security-rules.sexp` in the same commit when +the vulnerable pattern is mechanically detectable. If a useful scanner rule +would be too noisy, document the reason in the fix review or changelog instead +of silently skipping scanner coverage. + +### Essential Tools (use proactively) + +| When | Tool | +|---|---| +| Before writing ANY Jerboa code | `jerboa_howto` — cookbook has verified patterns with correct imports | +| Check what a module exports | `jerboa_module_exports` — never guess function names | +| Check function arity/args | `jerboa_function_signature` — prevents wrong-arg-count errors | +| Validate your code | `jerboa_verify` — one-stop syntax+compile+lint+arity check | +| Test expressions interactively | `jerboa_eval` — use `imports` param for modules, `env` for FFI paths | +| Persistent interactive testing | `jerboa_repl_session` — maintains state across evaluations | +| Debug an error message | `jerboa_explain_error` + `jerboa_error_fix_lookup` | +| Understand unfamiliar code | `jerboa_file_summary` + `jerboa_document_symbols` | +| Find where something is defined | `jerboa_find_definition` — source file, module, kind, arity | +| Search for symbol by substring | `jerboa_apropos` or `jerboa_smart_complete` | +| Build the project | `jerboa_build_and_report` or `jerboa_make` — prefer over bash `make` | +| Run tests | `jerboa_run_tests` — prefer over ad hoc shell test invocations | +| Check for stale .so artifacts | `jerboa_stale_static` — common cause of "edit has no effect" | +| Macro expansion | `jerboa_expand_macro` / `jerboa_trace_macro` | +| Inspect struct/class types | `jerboa_class_info` — fields, inheritance, constructor signature | +| FFI work | `jerboa_ffi_scaffold` / `jerboa_ffi_type_check` / `jerboa_ffi_null_safety` | +| Port Gerbil code | `jerboa_migration_check` + `jerboa_translate_scheme` | +| Detect paren imbalance | `jerboa_check_balance` — use BEFORE `make build` after deep edits | +| Edit a `.ss` file | `jerboa_balanced_insert` / `jerboa_balanced_replace` — NEVER raw `edit`/`sed`/`python` (see top of file) | +| Create a `.ss` file | `jerboa_write_file` — whole-file atomic write, `verify: true` rejects unbalanced content | +| File already unbalanced | `jerboa_repair_balance` — dry-run repair plan; or `git checkout -- <file>` and redo with `balanced_insert` | +| Full project audit | `jerboa_project_health_check` — balance, exports, cycles, duplicates | +| Security audit | `jerboa_security_audit` + `jerboa_import_policy_check` | +| Static build audit | `jerboa_static_symbol_audit` + `jerboa_boot_library_audit` | +| Explore a module | `jerboa_module_catalog` (replaces multiple `jerboa_doc` calls) | +| Look up any symbol | `jerboa_doc` — type, arity, qualified name, related symbols | +| Read stdlib source | `jerboa_stdlib_source` — see internal implementations | + +### Cookbook & Knowledge Management + +- **`jerboa_howto`** / **`jerboa_howto_get`**: Search and retrieve verified recipes +- **`jerboa_howto_add`**: Save new patterns to cookbook (MANDATORY when you discover something non-trivial) +- **`jerboa_howto_run`** / **`jerboa_howto_verify`**: Validate recipes still work +- **`jerboa_error_fix_add`**: Save error→fix mappings for common mistakes +- **`jerboa_anti_pattern_lookup`**: Search reusable local-model mistakes and failed strategies + +**The knowledge base is `data/*.sexp` in THIS repo**, embedded into `jmcp` at build time. The write tools above edit it live — the server reads `data/` from disk first, with the embedded copy as fallback (`JERBOA_MCP_REPO` points every client at this repo). When you add a stdlib/language feature, also update `data/` (a cookbook recipe + `api-signatures.sexp` + `changelog.sexp`) and **commit it**. Run `make jmcp` (or `make jmcp-portable`) only to refresh the embedded copy shipped in portable binaries. + +### Code Generation & Refactoring + +`jerboa_rename_symbol`, `jerboa_balanced_replace`, `jerboa_balanced_insert`, `jerboa_write_file`, `jerboa_repair_balance`, `jerboa_wrap_form`, `jerboa_splice_form`, `jerboa_scaffold_test`, `jerboa_generate_module`, `jerboa_translate_scheme`, `jerboa_project_template`, `jerboa_httpd_handler_scaffold`, `jerboa_db_pattern_scaffold`, `jerboa_actor_ensemble_scaffold` + +### Feature Suggestions + +- **`jerboa_list_features`** / **`jerboa_suggest_feature`** / **`jerboa_vote_feature`**: Track and submit tooling improvement ideas + +--- + +## MANDATORY: Save What You Learn + +Jerboa is niche — every non-trivial pattern you discover prevents future sessions from re-discovering it. + +### Save to cookbook (`jerboa_howto_add`) whenever you: +- Discover a working pattern through `jerboa_eval` or trial-and-error +- Figure out correct imports, arities, or calling conventions that weren't obvious +- Find a workaround for a Jerboa quirk or undocumented behavior + +**Before saving**: check `jerboa_howto` to avoid duplicates. **Do NOT save**: trivial one-liners, project-specific logic, or existing recipes. + +**Recipe format**: `id` (kebab-case), `tags` (4-6 search keywords incl. module name), `imports` (all required), `code` (complete working example), `notes` (gotchas/alternatives). + +### Save anti-patterns (`data/anti-patterns.sexp`) whenever you: +- See a plausible local-model strategy that failed verification +- Find a weak verifier pattern that allowed false success +- See a repeated repair loop, such as broad-reading after a concrete error +- Find a generic runtime mistake, such as missing lower-bound checks before vector access + +**Before saving**: check `jerboa_anti_pattern_lookup` to avoid duplicates. If none exists, call `jerboa_anti_pattern_add`; only edit `data/anti-patterns.sexp` directly if the writer tool is unavailable. Save the normalized reusable mistake, not the whole trace or benchmark name. + +**Anti-pattern format**: `id`, `title`, `kinds`, `severity`, `tags`, `pattern`, `avoid`, `advice`, `tools`. + +### Save error fixes (`jerboa_error_fix_add`) whenever you: +- See exact compiler/runtime/verifier text with a repeatable repair +- Hit an error that `jerboa_failure_advisor` should classify better next time +- Debug a local-model generated-code failure where a short diagnosis prevents another failed iteration + +**Before saving**: check `jerboa_error_fix_lookup` with the exact error text. **Do NOT save**: one-off project business-logic mistakes. + +**Error-fix format**: `id`, `pattern`, `fix`; optional `type`, `explanation`, `code_example`. + +### Suggest tooling improvements (`jerboa_suggest_feature`) whenever you: +- Make multiple sequential tool calls that could be one tool +- Fall back to bash because an MCP tool is missing or insufficient + +**Before suggesting**: check `jerboa_list_features`; vote with `jerboa_vote_feature` if it already exists. + +### Save Discoveries Mechanisms +- **`/save-discoveries` skill**: invoke anytime to review session and save recipes, anti-patterns, error fixes, feature suggestions, and security patterns +- **PreCompact hook**: add `PreCompact` hook with `type: "prompt"` in `.claude/settings.json` to auto-save before context compaction + +--- + +## Common Workflows + +- **Write new code**: `jerboa_howto` -> `jerboa_module_exports` -> write code -> `jerboa_verify` -> `jerboa_security_scan` +- **Debug an error**: `jerboa_explain_error` -> follow suggested tools -> `jerboa_howto` for fix patterns +- **Understand unfamiliar code**: `jerboa_file_summary` -> `jerboa_document_symbols` -> `jerboa_module_deps` +- **Refactor a module**: `jerboa_check_exports` -> `jerboa_find_callers` -> `jerboa_rename_symbol` -> `jerboa_check_import_conflicts` +- **Build project**: `jerboa_build_conflict_check` -> `jerboa_make` -> `jerboa_build_and_report` +- **Port from Gerbil**: `jerboa_migration_check` -> `jerboa_translate_scheme` -> `jerboa_verify` -> `jerboa_check_syntax` +- **Audit project quality**: `jerboa_verify` -> `jerboa_lint` -> `jerboa_dead_code` -> `jerboa_dependency_cycles` +- **Debug a crash**: `jerboa_stale_static` -> `jerboa_bisect_crash` -> `jerboa_ffi_type_check` +- **Learn a module**: `jerboa_stdlib_source` -> `jerboa_module_catalog` -> `jerboa_module_quickstart` +- **Security audit**: `jerboa_security_audit` -> `jerboa_import_policy_check` -> `jerboa_unsafe_import_lint` +- **Static build audit**: `jerboa_static_symbol_audit` -> `jerboa_boot_library_audit` -> `jerboa_rust_musl_build` +- **Safe-by-default check**: `jerboa_safe_prelude_check` -> `jerboa_resource_leak_check` -> `jerboa_safe_prelude_generate` +- **Debug editor command**: `jerboa_command_trace` with `project_path` and `buffer_type` + +## Workflow Conventions + +When implementing new features, always complete the documentation update in the same session. Document non-trivial solutions as howto recipes in the cookbook system. new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,17 @@ +Apache License +Version 2.0, January 2004 + +Copyright 2026 + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + new file mode 100644 --- /dev/null +++ b/Makefile @@ -0,0 +1,33 @@ +JERBOA ?= jerboa +BIN := bin/jerboa-aigit + +.PHONY: all help test verify install clean + +all: help + +help: + @echo "jerboa-aigit" + @echo "" + @echo " make test Run fixture smoke tests" + @echo " make verify Run syntax smoke plus fixture tests" + @echo " make install Install wrapper to ~/.local/bin/sniff-ai" + @echo " make clean Remove local build artifacts" + +$(BIN): bin/jerboa-aigit + +test: + @tests/fixture-smoke.sh + +verify: + @$(JERBOA) main-binary.ss --help >/dev/null + @tests/fixture-smoke.sh + +install: + @mkdir -p "$(HOME)/.local/bin" + @cp bin/jerboa-aigit "$(HOME)/.local/bin/jerboa-aigit" + @chmod +x "$(HOME)/.local/bin/jerboa-aigit" + @echo "installed $(HOME)/.local/bin/jerboa-aigit" + +clean: + @rm -rf dist tmp + new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,78 @@ +# jerboa-aigit + +`jerboa-aigit` scans Git history for evidence that commits or changed lines may +have been AI-assisted. It separates recorded provenance from heuristics: + +- Git AI notes in `refs/notes/ai` are reported as recorded authorship evidence. +- Agent/tool metadata in author identity, commit messages, and notes is reported + separately. +- Message, code, structure, history, baseline, and SimHash heuristics are scored + as probabilistic evidence, not proof. + +The current implementation is offline-only. It never calls an LLM or sends +repository data over the network. + +## Usage + +```bash +./bin/jerboa-aigit scan /path/to/repo --count 25 +./bin/jerboa-aigit scan /path/to/repo --format json +./bin/jerboa-aigit stats /path/to/repo --count 100 +./bin/jerboa-aigit verify-authorship /path/to/repo --count 25 +``` + +During development, the same entrypoint can be run directly: + +```bash +jerboa main-binary.ss scan /path/to/repo --format json +``` + +Supported options are `--count N`, `--commit REV`, `--format table|json|jsonl`, +`--metadata-only`, and `--heuristics-only`. + +## What It Reads + +The scanner runs Git commands against the requested repository using fixed +argument lists, not shell-interpolated command strings. It reads commit +metadata, first-parent diffs, `--numstat`, added patch lines, and AI authorship +notes from `refs/notes/ai`. It does not execute repository code and does not +write to the scanned worktree, refs, notes, hooks, or config. + +## Output Contract + +JSON output includes: + +- `detector_version` +- repository path +- commit and parent IDs +- author/committer-facing metadata +- changed files, additions, deletions, and added line count +- recorded AI note presence and excerpt +- metadata hits +- raw signal scores, weights, evidence, and limitations +- aggregate score and verdict +- warnings for missing or unavailable evidence + +Verdicts are intentionally conservative: + +- `recorded-ai-authorship` +- `metadata-indicated-agent` +- `likely-ai-assisted` +- `mixed-uncertain` +- `likely-human-style` + +Only recorded authorship metadata is high-confidence provenance. Heuristic +verdicts can be wrong for generated scaffolds, formatter-only changes, +disciplined commit conventions, bot commits, and large refactors. + +## Development + +```bash +make test +make verify +``` + +All Jerboa source is in `main-binary.ss`. Per `AGENTS.md`, edit `.ss` files only +with Jerboa MCP balanced tools and run balance plus verification after every +change. + new file mode 100755 --- /dev/null +++ b/bin/jerboa-aigit @@ -0,0 +1,8 @@ +#!/usr/bin/env sh +set -eu + +script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +repo_dir=$(CDPATH= cd -- "$script_dir/.." && pwd) + +exec jerboa "$repo_dir/main-binary.ss" "$@" + new file mode 100644 --- /dev/null +++ b/main-binary.ss @@ -0,0 +1,442 @@ +(import (jerboa prelude)) +(import (std misc process)) + +(def detector-version "0.1.0") +(def us (integer->char 31)) +(def tab (integer->char 9)) + +(defstruct options (command path count format commit metadata-only? heuristics-only?)) +(defstruct signal (name category score weight confidence reason evidence limitations)) +(defstruct finding + (commit parent author-name author-email time subject files additions deletions + added-lines note metadata signals score verdict warnings)) + +(def raw-command-line (command-line)) +(def cli-args + (let ([xs (if (and (pair? raw-command-line) (string? (car raw-command-line))) + (cdr raw-command-line) + raw-command-line)]) + (if (and (pair? xs) (string? (car xs)) (string-suffix? ".ss" (car xs))) + (cdr xs) + xs))) + +(def (usage) + (displayln "usage: jerboa main-binary.ss scan [PATH] [--count N] [--commit REV] [--format table|json|jsonl] [--metadata-only|--heuristics-only]") + (displayln " jerboa main-binary.ss stats [PATH] [--count N]") + (displayln " jerboa main-binary.ss verify-authorship [PATH] [--count N]")) + +(def (down s) (string-map char-downcase s)) +(def (contains? s sub) (and (string-contains s sub) #t)) +(def (blank? s) (string-empty? (string-trim s))) +(def (split-lines s) (if (string-empty? s) '() (string-split s #\newline))) +(def (split-tabs s) (string-split s tab)) +(def (safe-ref xs n fallback) (if (< n (length xs)) (list-ref xs n) fallback)) +(def (decimal-digits? s) + (and (> (string-length s) 0) + (for/and ([ch (in-string s)]) + (and (char>=? ch #\0) (char<=? ch #\9))))) + +(def (decimal-int s fallback) + (if (decimal-digits? s) + (for/fold ([n 0]) ([ch (in-string s)]) + (+ (* n 10) (- (char->integer ch) (char->integer #\0)))) + fallback)) + +(def (option-flag? s) + (and (>= (string-length s) 2) + (char=? (string-ref s 0) #\-) + (char=? (string-ref s 1) #\-))) +(def (parse-int s fallback) (decimal-int s fallback)) +(def (clamp01 x) (cond [(< x 0.0) 0.0] [(> x 1.0) 1.0] [else x])) +(def (sum xs) (for/fold ([acc 0]) ([x xs]) (+ acc x))) +(def (mean xs) (if (null? xs) 0.0 (/ (exact->inexact (sum xs)) (length xs)))) +(def (square x) (* x x)) +(def (stddev xs) + (if (< (length xs) 2) + 0.0 + (let* ([m (mean xs)] + [v (/ (for/fold ([acc 0.0]) ([x xs]) (+ acc (square (- x m)))) + (length xs))]) + (sqrt v)))) +(def (count-where pred xs) (for/fold ([n 0]) ([x xs]) (if (pred x) (+ n 1) n))) +(def (any-contains? text needles) (for/or ([needle needles]) (contains? text needle))) +(def (starts-with-any? text prefixes) (for/or ([prefix prefixes]) (string-prefix? prefix text))) + +(def (git repo args) + (try + (run-process (append (list "git" "-C" repo "--literal-pathspecs") args)) + (catch (e) ""))) + +(def (repo-root path) + (let ([root (string-trim (git path '("rev-parse" "--show-toplevel")))]) + (if (string-empty? root) #f root))) + +(def (commit-list repo count commit) + (if commit + (list commit) + (filter (lambda (line) (not (blank? line))) + (split-lines (git repo (list "rev-list" (str "--max-count=" count) "HEAD")))))) + +(def (commit-fields repo rev) + (let* ([fmt "%H%x1f%P%x1f%an%x1f%ae%x1f%ct%x1f%s"] + [out (string-trim (git repo (list "show" "-s" (str "--format=" fmt) rev)))]) + (string-split out us))) + +(def (commit-message repo rev) (git repo (list "show" "-s" "--format=%B" rev))) +(def (note-text repo rev) (string-trim (git repo (list "notes" "--ref=ai" "show" rev)))) +(def (parent-time repo parent) + (if (string-empty? parent) 0 (parse-int (string-trim (git repo (list "show" "-s" "--format=%ct" parent))) 0))) + +(def (first-parent parents) + (let ([parts (filter (lambda (x) (not (blank? x))) (string-split parents #\space))]) + (if (pair? parts) (car parts) ""))) + +(def (parse-numstat line) + (let* ([parts (split-tabs line)] + [adds (parse-int (safe-ref parts 0 "0") 0)] + [dels (parse-int (safe-ref parts 1 "0") 0)] + [path (safe-ref parts 2 "")]) + (list path adds dels))) + +(def (changed-files repo rev) + (map parse-numstat + (filter (lambda (line) (not (blank? line))) + (split-lines (git repo (list "show" "--format=" "--numstat" "--first-parent" rev)))))) + +(def (numstat-adds files) (sum (map cadr files))) +(def (numstat-dels files) (sum (map caddr files))) +(def (numstat-paths files) (map car files)) + +(def (added-lines repo rev) + (map (lambda (line) (substring line 1 (string-length line))) + (filter (lambda (line) + (and (string-prefix? "+" line) (not (string-prefix? "+++" line)))) + (split-lines (git repo (list "show" "--format=" "--first-parent" "--unified=0" "--no-ext-diff" rev)))))) + +(def known-agents + '("codex" "copilot" "claude" "cursor" "openai" "anthropic" "aider" "windsurf" "cody" "tabnine" "ai-agent")) + +(def (metadata-hits author-name author-email subject body note) + (let ([text (down (string-join (list author-name author-email subject body note) "\n"))]) + (filter (lambda (marker) (contains? text marker)) known-agents))) + +(def (sig name category score weight confidence reason evidence limitations) + (make-signal name category (clamp01 score) weight confidence reason evidence limitations)) + +(def conventional-prefixes '("feat:" "fix:" "docs:" "style:" "refactor:" "test:" "chore:" "perf:" "ci:" "build:")) +(def ai-message-phrases '("this commit introduces" "this change implements" "comprehensive" "robust" "ensure that" "designed to" "allows users to" "provides a" "leverages")) + +(def (word-count s) (length (filter (lambda (x) (not (blank? x))) (string-split s #\space)))) +(def (capitalized? s) (and (> (string-length s) 0) (char-upper-case? (string-ref s 0)))) +(def (ends-sentence? s) + (and (> (string-length s) 0) + (let ([ch (string-ref s (- (string-length s) 1))]) + (or (char=? ch #\.) (char=? ch #\!) (char=? ch #\?))))) + +(def (message-signal subject body additions) + (let* ([text (down (string-join (list subject body) "\n"))] + [conv? (starts-with-any? text conventional-prefixes)] + [phrases (filter (lambda (p) (contains? text p)) ai-message-phrases)] + [terse? (<= (word-count subject) 2)] + [polished? (and (capitalized? subject) (ends-sentence? subject))] + [bullets? (or (contains? body "\n- ") (contains? body "\n* "))] + [mismatch? (and (> additions 120) (< (word-count subject) 5))] + [score (+ (if conv? 0.12 0.0) (if polished? 0.10 0.0) + (if terse? 0.15 0.0) (if bullets? 0.15 0.0) + (if mismatch? 0.20 0.0) (min 0.30 (* 0.12 (length phrases))))] + [evidence (append (if conv? '("conventional prefix") '()) + (if polished? '("polished sentence style") '()) + (if terse? '("very terse subject") '()) + (if bullets? '("markdown bullets") '()) + (if mismatch? '("large diff with short message") '()) + phrases)]) + (sig "message-style" "text" score 0.10 "low" "commit message matches weak generated-text patterns" evidence + "message style alone is weak evidence"))) + +(def comment-prefixes '("//" "#" ";" "/*" "*" "--")) +(def human-noise '("todo" "fixme" "debug" "console.log" "print(" "printf(" "hack" "xxx")) +(def explanatory-phrases '("this function" "this method" "ensure" "handles" "responsible for" "in order to" "used to")) +(def generic-names '("data" "result" "item" "items" "value" "values" "temp" "helper" "manager" "processor" "handler")) + +(def (comment-line? line) (starts-with-any? (string-trim line) comment-prefixes)) +(def (identifier-char? ch) (or (char-alphabetic? ch) (char-numeric? ch) (= (char->integer ch) 95))) + +(def (lexemes-from-line line) + (let loop ([chars (string->list line)] [cur '()] [out '()]) + (cond [(null? chars) + (reverse (if (null? cur) out (cons (list->string (reverse cur)) out)))] + [(identifier-char? (car chars)) + (loop (cdr chars) (cons (char-downcase (car chars)) cur) out)] + [(null? cur) (loop (cdr chars) '() out)] + [else (loop (cdr chars) '() (cons (list->string (reverse cur)) out))]))) + +(def (source-lexemes lines) + (filter (lambda (lexeme) (> (string-length lexeme) 1)) (append-map lexemes-from-line lines))) + +(def (code-signal lines) + (let* ([nonblank (filter (lambda (line) (not (blank? line))) lines)] + [line-count (length nonblank)] + [comments (count-where comment-line? nonblank)] + [comment-ratio (if (= line-count 0) 0.0 (/ (exact->inexact comments) line-count))] + [text (down (string-join nonblank "\n"))] + [noise? (any-contains? text human-noise)] + [phrases (filter (lambda (p) (contains? text p)) explanatory-phrases)] + [lexemes (source-lexemes nonblank)] + [generic-count (count-where (lambda (lexeme) (member lexeme generic-names)) lexemes)] + [generic-ratio (if (null? lexemes) 0.0 (/ (exact->inexact generic-count) (length lexemes)))] + [large? (> line-count 80)] + [score (+ (if large? 0.20 0.0) + (if (> comment-ratio 0.30) 0.25 0.0) + (if (and large? (not noise?)) 0.15 0.0) + (if (> generic-ratio 0.08) 0.15 0.0) + (min 0.20 (* 0.08 (length phrases))))] + [evidence (append (if large? '("large added block") '()) + (if (> comment-ratio 0.30) (list (str "comment ratio " comment-ratio)) '()) + (if (and large? (not noise?)) '("large block lacks TODO/FIXME/debug markers") '()) + (if (> generic-ratio 0.08) (list (str "generic identifier ratio " generic-ratio)) '()) + phrases)]) + (sig "lexical-code-style" "code" score 0.35 "medium" "added code has lexical/comment patterns associated with generated scaffolding" evidence + "language-neutral lexical analysis is noisy"))) + +(def (line-lengths lines) (map string-length (filter (lambda (line) (not (blank? line))) lines))) +(def (blank-gaps lines) + (let loop ([xs lines] [idx 0] [last #f] [out '()]) + (cond [(null? xs) (reverse out)] + [(blank? (car xs)) + (if last (loop (cdr xs) (+ idx 1) idx (cons (- idx last) out)) (loop (cdr xs) (+ idx 1) idx out))] + [else (loop (cdr xs) (+ idx 1) last out)]))) + +(def (structure-signal paths additions deletions lines) + (let* ([lengths (line-lengths lines)] + [avg (mean lengths)] + [cv (if (= avg 0.0) 0.0 (/ (stddev lengths) avg))] + [regular? (and (>= (length (blank-gaps lines)) 3) (< (stddev (blank-gaps lines)) 1.0))] + [many-files? (> (length paths) 12)] + [ratio (if (= deletions 0) additions (/ (exact->inexact additions) deletions))] + [balanced? (and (> additions 40) (> deletions 40) (< (abs (- ratio 1.0)) 0.20))] + [low-cv? (and (> (length lengths) 20) (< cv 0.25))] + [score (+ (if low-cv? 0.25 0.0) (if regular? 0.20 0.0) + (if many-files? 0.18 0.0) (if balanced? 0.18 0.0))] + [evidence (append (if low-cv? (list (str "low line-length coefficient of variation " cv)) '()) + (if regular? '("regular blank-line spacing") '()) + (if many-files? (list (str "many files changed: " (length paths))) '()) + (if balanced? '("balanced large addition/deletion ratio") '()))]) + (sig "diff-structure" "structure" score 0.15 "medium" "diff shape is unusually regular or broad" evidence + "formatters, generated files, and refactors can look similar"))) + +(def (fnv-step h ch) (modulo (* (bitwise-xor h (char->integer ch)) 16777619) 4294967296)) +(def (stable-hash32 s) (for/fold ([h 2166136261]) ([ch (in-string s)]) (fnv-step h ch))) +(def (bit-set? n bit) (not (= 0 (bitwise-and n (ash 1 bit))))) +(def (simhash32 lexemes) + (let ([weights (make-vector 32 0)]) + (for ([lexeme lexemes]) + (let ([h (stable-hash32 lexeme)]) + (for ([i (in-range 32)]) + (vector-set! weights i (+ (vector-ref weights i) (if (bit-set? h i) 1 -1)))))) + (for/fold ([acc 0]) ([i (in-range 32)]) + (if (> (vector-ref weights i) 0) (bitwise-ior acc (ash 1 i)) acc)))) +(def (popcount32 n) (let loop ([x n] [acc 0]) (if (= x 0) acc (loop (ash x -1) (+ acc (bitwise-and x 1)))))) +(def (hamming32 a b) (popcount32 (bitwise-xor a b))) +(def (best-sim hash hashes) + (if (null? hashes) 0.0 (- 1.0 (/ (apply min (map (lambda (h) (hamming32 hash h)) hashes)) 32.0)))) +(def (similarity-signal lines hashes) + (let* ([hash (simhash32 (source-lexemes lines))] + [sim (best-sim hash hashes)] + [score (cond [(>= sim 0.92) 0.80] [(>= sim 0.85) 0.50] [(>= sim 0.80) 0.25] [else 0.0])] + [evidence (if (> score 0.0) (list (str "best simhash similarity " sim)) '())]) + (list (sig "simhash-similarity" "similarity" score 0.15 "medium" "added code is near another scanned addition" evidence + "similarity in one scan window cannot prove original authorship") + hash))) + +(def (history-signal additions commit-time parent-time author-times) + (let* ([delta (- commit-time parent-time)] + [minutes (if (> delta 0) (/ delta 60.0) 0.0)] + [velocity (if (> minutes 0.0) (/ additions minutes) 0.0)] + [nearby (count-where (lambda (t) (<= (abs (- commit-time t)) 600)) author-times)] + [burst? (>= nearby 5)] + [score (+ (cond [(> velocity 50.0) 1.0] [(> velocity 20.0) 0.60] [else 0.0]) (if burst? 0.20 0.0))] + [evidence (append (if (> velocity 20.0) (list (str "line velocity " velocity " additions/minute")) '()) + (if burst? (list (str nearby " commits by author inside ten minutes")) '()))]) + (sig "history-timing" "history" score 0.15 "low" "commit timestamps suggest high throughput or bursts" evidence + "Git commit timestamps are not typing timestamps"))) + +(def (baseline-signal additions author-additions) + (if (< (length author-additions) 5) + (sig "author-baseline" "baseline" 0.0 0.00 "low" "author baseline unavailable" '("fewer than five comparison commits") + "missing baseline is not evidence of human authorship") + (let* ([m (mean author-additions)]