Track parity gaps and add semantic alignment
ober
fb8c17f9da50d9175c3519357fd5dc18836890d0
new file mode 100644 --- /dev/null +++ b/GAPS.md @@ -0,0 +1,404 @@ +# Parity gaps + +This document tracks what prevents `jerboa-aigit` from being on par with the +reference tools investigated in `plan.md`: + +- `/Users/user/.local/bin/sniff` +- `/Users/user/aisniff/Cadence` +- `/Users/user/aisniff/git-ai` + +The goal is parity with all useful behavior from those tools while preserving +the stronger safety and provenance model already present here. “Parity” does +not mean copying unsafe defaults or overstating heuristic certainty. + +## Current baseline + +Implemented today: + +- safe Git invocation with fixed argv and repo-relative pathspec validation; +- scan, stats, explain, and verify-authorship commands; +- table, JSON, JSONL, and Markdown output; +- Git AI note parsing from `refs/notes/ai`; +- file-level note attribution and unmatched attribution reporting; +- metadata clues from author/message/note text; +- deterministic text, lexical code, diff-structure, SimHash, timing, author + baseline, and repository baseline signals; +- configurable thresholds, weights, excludes, and resource limits; +- first-parent and all-parent merge modes; +- warnings for root commits, shallow/missing parents, malformed notes, binary + files, rename/copy diffs, resource truncation, and missing evidence; +- UTF-8 path and path-with-spaces handling; +- explicit offline-by-default behavior; +- safe local provider adapter for optional model scoring. + +The remaining gaps below are ordered by impact on parity and correctness. + +## P0: Evaluation and calibration + +### G-001: No labeled corpus + +Reference parity cannot be claimed without labeled evidence. The current test +fixture covers edge cases, not accuracy. + +Acceptance criteria: + +- Add an `eval/` fixture corpus format with labels for human, AI-assisted, + metadata-attested, bot, generated/vendor, formatter-only, refactor, merge, + and ambiguous commits. +- Include at least one fixture repo generated by scripts and at least one + documented external/manual corpus path. +- Store labels separately from scanner output. +- Document how labels were assigned and how ambiguity is represented. + +### G-002: No precision/recall report + +The scanner has no measured false-positive or false-negative rates. + +Acceptance criteria: + +- Add an evaluation command or script that scans the corpus and emits precision, + recall, F1, false-positive rate, false-negative rate, and confusion matrix by + verdict band. +- Report metrics separately for recorded provenance, metadata clues, and + heuristic-only findings. +- Include a committed sample report. + +### G-003: Thresholds and weights are not tuned from data + +Current weights are plausible and reference-inspired, but not calibrated. + +Acceptance criteria: + +- Add a repeatable threshold/weight sensitivity runner. +- Compare default weights against at least two candidate configurations. +- Document why the default thresholds were selected. +- Keep config hash/version stable and visible in reports. + +### G-004: No differential comparison against reference tools + +Parity with `sniff`, Cadence, and git-ai is not measured. + +Acceptance criteria: + +- Add a compatibility/differential report that compares overlapping cases with + each reference tool. +- Record intentional differences, especially where this project is stricter + about provenance or safety. +- Preserve raw reference outputs or concise normalized summaries. + +## P1: High-value sniff parity + +### G-010: Missing semantic alignment signal + +`sniff` uses optional embeddings to compare commit message meaning with changed +code. This project has no semantic/message-code alignment signal. + +Status: implemented for offline lexical alignment. Remaining work for full +embedding parity is tracked in G-011. + +Acceptance criteria: + +- Add an offline lexical alignment signal first: overlap between normalized + commit-message tokens and added-code tokens, excluding stopwords and generic + code words. +- Report it under category `semantic` or `alignment` with limitations. +- Add it to stats coverage and explanation output. +- Keep embeddings optional; do not add a network dependency. + +### G-011: No optional embedding adapter + +`sniff` can use `all-MiniLM-L6-v2` embeddings. This project has no embedding +provider interface. + +Acceptance criteria: + +- Define a local embedding-provider contract equivalent in safety to the local + LLM provider contract. +- Only run it with explicit opt-in. +- Treat output as secondary evidence and record `network_used:false`. + +### G-012: Missing perplexity-style message signal + +`sniff` has optional GPT-2 perplexity. This project lacks any equivalent. + +Acceptance criteria: + +- Add a provider interface or offline approximation for message naturalness. +- Keep it optional or clearly mark the offline approximation as weak. +- Include tests proving malformed provider output is ignored safely. + +### G-013: Python-specific AST/docstring analysis missing + +`sniff` has Python AST docstring/function analysis. This scanner only performs +language-neutral lexical checks. + +Acceptance criteria: + +- Add Python-specific heuristics only for Python changed files. +- Detect generated-style docstrings, repetitive function structure, function + length regularity, and low variable-name diversity. +- Fall back to existing language-neutral heuristics for other languages. + +### G-014: Stable SimHash is only 32-bit + +`sniff` uses 64-bit SimHash. This project uses a stable 32-bit FNV-derived +SimHash, which is less discriminating. + +Acceptance criteria: + +- Upgrade to a stable 64-bit SimHash or document why 32-bit is retained. +- Include tests for deterministic output across processes. +- Report cross-author vs same-author near matches separately. + +### G-015: Baseline model is too narrow + +`sniff` baselines more than commit size: comment ratio, average line length, +variance, function length, and naming style. This project baselines addition +counts only. + +Acceptance criteria: + +- Extend author and repository baselines to include comment ratio, average line + length, line-length variance, naming-style/generic-token ratio, and function + or block-size proxy. +- Keep insufficient-baseline warnings when fewer than five comparison commits + exist. +- Document whether baseline remains zero-weight or becomes weighted. + +## P1: Cadence parity + +### G-020: Strategy registry is implicit + +Cadence models strategies as named units with category, confidence, +description, reason, and evidence. This project has signal structs but no +registry/report inventory. + +Acceptance criteria: + +- Add a detector/strategy catalog in JSON stats or a new command. +- Include enabled/disabled status, category, default weight, limitations, and + config key for each strategy. +- Test that every emitted signal has a catalog entry. + +### G-021: Diff-shape heuristics are incomplete + +Cadence includes too many files, skewed addition/deletion ratio, suspiciously +balanced large changes, consistent per-file sizes, template/file-extension +patterns, and merge filtering. This project covers some but not all. + +Acceptance criteria: + +- Add explicit signals for: + - file dispersion / too many files; + - skewed addition/deletion ratio; + - balanced large changes; + - consistent per-file sizes; + - template/generated extension patterns. +- Add fixture commits for each signal and false-positive counterexamples. + +### G-022: Timing heuristics are incomplete + +Current timing handles velocity and simple nearby commit count, but not full +Cadence-style rapid-fire burst definitions. + +Acceptance criteria: + +- Implement burst windows requiring at least five commits in ten minutes and at + least three substantial nonblank commits. +- Report zero/negative time gaps explicitly. +- Add fixtures for high velocity, burst, normal batch commits, and amended + historical imports. + +### G-023: Code-pattern heuristics are incomplete + +Cadence detects generic variables/functions, overly regular camelCase, dense + error-handling scaffolding, template patterns, and TODO/FIXME patterns. + +Acceptance criteria: + +- Add separate evidence items for generic names, naming-style regularity, dense + error handling, template patterns, and TODO/FIXME/debug absence/presence. +- Ensure TODO/FIXME/debug presence can act as counter-evidence rather than + only lack-of-evidence. + +### G-024: Repository metrics are too limited + +Cadence reports strategy coverage and hit counts. This project reports signal +category coverage, but not strategy hit rates or category confidence summaries. + +Acceptance criteria: + +- Add stats for strategy hit count, average score by strategy, category + coverage, and warning counts. +- Include JSON and table forms. + +## P1: git-ai parity + +### G-030: Git AI note formats are only partially parsed + +`git-ai` records sessions, checkpoints, attestations, tool/model identity, +line ranges, commit/blob notes, and possibly alternate shapes. This scanner +supports a simple `lines` array plus top-level tool/model/session. + +Acceptance criteria: + +- Inventory supported git-ai note formats from the local reference. +- Parse sessions, checkpoints, attestations, external session IDs, and blob + references where present. +- Preserve unsupported fields in a bounded raw/extra object or warning. +- Add malformed and alternate-format fixtures. + +### G-031: Batch note resolution missing + +`git-ai` can resolve notes in batches for commit/blob objects. This project +looks up one commit note at a time. + +Acceptance criteria: + +- Add batched note lookup for selected commits. +- Preserve read-only behavior. +- Benchmark against per-commit lookup on a medium history. + +### G-032: Recovered attribution is too shallow + +`git-ai` can recover attribution from known agent identities, file timestamps, +local session events, and metadata. This scanner only reports metadata hits. + +Acceptance criteria: + +- Add a recovered-attribution section distinct from recorded attribution. +- Include source and confidence for each recovered item. +- Support known agent identity catalog from config. +- Never label recovered attribution as recorded provenance. + +### G-033: Provider/tool catalog is hard-coded + +Known AI tools are detected through hard-coded strings. + +Acceptance criteria: + +- Move known tool/provider identity patterns into config/data. +- Support Cursor, GitHub Copilot variants, Claude, Codex, and user-defined + tool identities. +- Include tests for false positives such as human names or ordinary words that + contain provider substrings. + +## P2: Robustness and security parity + +### G-040: No subprocess timeout + +The plan requires Git/provider timeouts. Current subprocess calls are fixed +argv but not explicitly timed out. + +Acceptance criteria: + +- Add timeout handling for Git and provider subprocesses. +- Report timeout warnings without crashing. +- Add fixture/fake-provider timeout tests. + +### G-041: NUL-delimited parsing not implemented + +The plan requires NUL-delimited output where applicable. Current parsing uses +line-oriented Git output. + +Acceptance criteria: + +- Use NUL-delimited Git output for name/status/numstat paths where Git supports + it. +- Preserve spaces, tabs, quotes, newlines, and non-ASCII paths. +- Add fixture paths containing tabs and newlines if the platform allows them. + +### G-042: Non-UTF-8 behavior is not directly tested + +The scanner warns when no UTF-8 patch lines are available, but there is no +explicit non-UTF-8 fixture. + +Acceptance criteria: + +- Add fixture commits with invalid UTF-8 file contents and paths where Git + permits. +- Ensure scans warn rather than crash. +- Ensure metadata-only scanning still works. + +### G-043: Prompt-injection/adversarial model tests missing + +The local provider adapter is bounded and explicit, but prompt injection in +commit messages/source comments is not tested. + +Acceptance criteria: + +- Add adversarial fixture messages and source comments. +- Ensure untrusted text is only payload data and never part of an executed + command. +- If a prompt template is added later, delimit untrusted data clearly. + +### G-044: Provider command safety is limited + +The local provider command is refused when it is inside the scanned repository, +but there is no ownership/mode/symlink validation. + +Acceptance criteria: + +- Validate provider path is canonical and not a symlink. +- Warn or refuse group/world-writable provider executables. +- Keep repo-local executable refusal. + +## P2: Product and architecture parity + +### G-050: Monolithic implementation + +`plan.md` proposed separate modules. The project currently keeps all Jerboa +source in `main-binary.ss`. + +Acceptance criteria: + +- Split into modules only when doing so does not regress static binary build. +- Preserve the CLI wrapper. +- Keep module boundaries aligned to Git, signals, aggregation, reports, config, + and providers. + +### G-051: No golden JSON snapshots + +Current shell tests grep output. They do not preserve full stable report +snapshots. + +Acceptance criteria: + +- Add normalized golden JSON fixtures. +- Avoid timestamp/path nondeterminism in snapshots. +- Review intentional output contract changes via snapshot diffs. + +### G-052: No property/fuzz tests + +The plan calls for arbitrary text, score bounds, deterministic output, bounded +memory, and command-injection tests. + +Acceptance criteria: + +- Add property/fuzz-style tests for JSON escaping, path validation, score + bounds, signal aggregation, and parser robustness. +- Include failing seeds as regression fixtures. + +### G-053: README does not include measured limitations + +The README explains heuristic uncertainty, but not measured accuracy limits. + +Acceptance criteria: + +- Add a limitations section tied to evaluation results. +- Include known false-positive classes and false-negative classes. +- Link to the latest evaluation report. + +## Completion definition for parity + +Parity is complete only when: + +1. Every P0 and P1 gap above is implemented or explicitly rejected with a + documented reason. +2. Every implemented parity feature has fixture coverage. +3. Evaluation reports show this project compared against all three reference + tools on the same corpus. +4. README and JSON output clearly distinguish recorded, recovered, metadata, + heuristic, and model evidence. +5. `make verify`, Jerboa compile/security verification, and platform-required + `make binary` pass. --- a/main-binary.ss +++ b/main-binary.ss @@ -405,6 +405,32 @@ (def (source-lexemes lines) (filter (lambda (lexeme) (> (string-length lexeme) 1)) (append-map lexemes-from-line lines))) +(def semantic-stopwords + '("the" "and" "for" "with" "that" "this" "from" "into" "onto" "are" "was" "were" "has" "have" "had" "will" "can" "could" "should" "would" "a" "an" "to" "of" "in" "on" "by" "as" "is" "it" "be" "or" "if" "else" "return" "def" "class" "let" "const" "var" "function" "value" "values" "data" "result" "item" "items" "temp")) +(def (semantic-term? term) + (and (> (string-length term) 2) + (not (member term semantic-stopwords)))) +(def (semantic-terms lines) + (unique (filter semantic-term? (source-lexemes lines)))) +(def (term-overlap xs ys) + (filter (lambda (x) (member x ys)) xs)) +(def (semantic-alignment-signal subject body lines) + (let* ([message-terms (semantic-terms (list subject body))] + [code-terms (semantic-terms lines)] + [overlap (term-overlap message-terms code-terms)] + [denom (min (length message-terms) (length code-terms))] + [ratio (if (= denom 0) 0.0 (/ (exact->inexact (length overlap)) denom))] + [score (cond [(>= ratio 0.50) 0.45] + [(>= ratio 0.30) 0.28] + [(>= ratio 0.18) 0.14] + [else 0.0])] + [evidence (if (> score 0.0) + (list (str "message/code term overlap " ratio ": " (string-join overlap ","))) + '())]) + (sig "semantic-alignment" "semantic" score 0.10 "low" + "commit message vocabulary overlaps unusually strongly with added code" + evidence + "lexical overlap is a weak offline proxy for semantic self-summary and can match intentional human summaries"))) (def (code-signal lines) (let* ([nonblank (filter (lambda (line) (not (blank? line))) lines)] @@ -794,6 +820,7 @@ [sim-pair (similarity-signal lines hashes)] [raw-signals (if eligible? (list (message-signal subject body adds) (code-signal lines) (structure-signal paths adds dels lines) + (semantic-alignment-signal subject body lines) (car sim-pair) (history-signal adds time (parent-time repo parent) (author-times repo revs author-email)) (baseline-signal adds (prior-additions repo revs author-email rev file)) (repository-baseline-signal adds (prior-repository-additions repo revs rev file))) @@ -954,7 +981,7 @@ (displayln "warnings:") (for ([w (finding-warnings f)]) (displayln (str "- " w)))))) (def stats-verdicts '("recorded-ai-authorship" "metadata-indicated-agent" "likely-ai-assisted" "mixed-uncertain" "likely-human-style" "insufficient-evidence")) -(def stats-signal-categories '("text" "code" "structure" "similarity" "history" "baseline")) +(def stats-signal-categories '("text" "code" "structure" "semantic" "similarity" "history" "baseline")) (def (alist-ref/default pairs name fallback) (let ([hit (assoc name pairs)]) --- a/tests/fixture-smoke.sh +++ b/tests/fixture-smoke.sh @@ -55,6 +55,8 @@ printf '%s\n' "$json" | grep -q '"unmatched_recorded_attribution":\[\]' printf '%s\n' "$json" | grep -q '"metadata_hits":\["codex","openai"\]' printf '%s\n' "$json" | grep -q '"category":"code"' printf '%s\n' "$json" | grep -q '"category":"structure"' +printf '%s\n' "$json" | grep -q '"name":"semantic-alignment"' +printf '%s\n' "$json" | grep -q '"category":"semantic"' printf '%s\n' "$json" | grep -q '"category":"similarity"' printf '%s\n' "$json" | grep -q '"category":"history"' printf '%s\n' "$json" | grep -q '"category":"baseline"' @@ -130,6 +132,7 @@ printf '%s\n' "$stats" | grep -q 'score bands:' printf '%s\n' "$stats" | grep -q ' recorded-ai-authorship: 1' printf '%s\n' "$stats" | grep -q 'signal coverage:' printf '%s\n' "$stats" | grep -q ' code: 2' +printf '%s\n' "$stats" | grep -q ' semantic: 2' stats_json=$("$root/bin/jerboa-aigit" stats "$fixture" --count 2 --format json) printf '%s\n' "$stats_json" | grep -q '"count":2' @@ -138,6 +141,7 @@ printf '%s\n' "$stats_json" | grep -q '"tools":\["codex"\]' printf '%s\n' "$stats_json" | grep -q '"recorded_ai_lines":360' printf '%s\n' "$stats_json" | grep -q '"score_bands":\[{"name":"recorded-ai-authorship","count":1}' printf '%s\n' "$stats_json" | grep -q '"signal_coverage":\[{"category":"text","count":2}' +printf '%s\n' "$stats_json" | grep -q '{"category":"semantic","count":2}' jsonl=$("$root/bin/jerboa-aigit" scan "$fixture" --format jsonl --count 2) jsonl_lines=$(printf '%s\n' "$jsonl" | wc -l | tr -d ' ')