Document measured limitations
ober
447f794b184eff5f878a8cd0f890145b621e445a
--- a/GAPS.md +++ b/GAPS.md @@ -523,6 +523,11 @@ Acceptance criteria: - Include known false-positive classes and false-negative classes. - Link to the latest evaluation report. +Status: implemented in `README.md` with an evaluation/limitations section +linked to `eval/sample-report.md`, `eval/sensitivity-report.md`, and +`eval/differential-report.md`. It records current fixture-sized metrics and +lists known false-positive and false-negative classes. + ## Completion definition for parity Parity is complete only when: --- a/README.md +++ b/README.md @@ -44,8 +44,8 @@ Supported options are `--config FILE`, `--count N`, `--all`, `--from REV`, `heuristics_only`. Supported scoring keys are `human_threshold`, `ai_threshold`, `weight_text`, `weight_code`, `weight_structure`, `weight_similarity`, `weight_history`, `weight_baseline`, `max_files`, -`max_added_lines`, and `max_note_bytes`. Later CLI flags override earlier -config values. +`max_added_lines`, `max_note_bytes`, `git_timeout_seconds`, and +`provider_timeout_seconds`. Later CLI flags override earlier config values. Threshold CLI values override the active config for the current invocation. The scanner has two operative cutoffs: scores below `HUMAN` are @@ -59,9 +59,10 @@ provider adapter when either `local_provider_command` is set in the JSON config as an argv array or `JERBOA_AIGIT_LOCAL_PROVIDER` points at an executable. The scanner sends one bounded JSON payload as the final argv item and expects strict JSON on stdout with a numeric `score` plus optional `reason` and `evidence`. -The provider executable must be an absolute path outside the scanned repository; -repo-local provider commands are refused so scanning never executes repository -code. +The provider executable must be an absolute canonical path outside the scanned +repository. Repo-local, symlinked, missing, and group/world-writable provider +commands are refused so scanning never executes repository code or unsafe local +adapters. Local provider output is reported as a secondary `model` signal; it never overwrites recorded authorship metadata. Remote/network providers are not implemented in this release, and `network_used` remains `false`. @@ -83,12 +84,17 @@ By default, scans skip `vendor/`, `generated/`, `dist/`, `node_modules/`, and `.git/` paths. Use `--include PATH` or `--file PATH` to inspect one of those paths explicitly. `--include` and `--file` accept repository-relative pathspecs; absolute paths and `..` components are refused before invoking Git. UTF-8 paths -and paths containing spaces are covered by the fixture tests. +and paths containing spaces, tabs, newlines, and non-ASCII characters are +covered by fixture tests using NUL-delimited Git parsing where supported. Resource limits default to 500 changed files per commit, 20,000 added lines analyzed per commit, and 50,000 bytes per AI note. When a limit is hit, output keeps the bounded data and includes a warning. +Git subprocesses and local provider subprocesses are timeout-bound. Git timeout +warnings mean partial scan data may be missing; provider timeout warnings mean +the scanner fell back to offline heuristics. + Baseline signals are reported as contextual evidence with zero default weight. The author baseline uses same-author addition history; the repository baseline uses median/MAD-style addition-count deviation across the selected scan window. @@ -121,6 +127,8 @@ JSON output includes: - recorded AI note presence and excerpt - structured `recorded_attribution` entries for supported Git AI note line ranges (`source`, `tool`, `model`, `session`, `path`, `start`, `end`) +- structured `recovered_attribution` entries for inferred identity evidence + (`source`, `agent`, `confidence`, `recorded_provenance:false`, `evidence`) - `file_findings` entries that attach matching recorded note ranges to changed files without duplicating commit-level heuristic scores - `unmatched_recorded_attribution` for recorded ranges whose path was not part @@ -143,6 +151,50 @@ Only recorded authorship metadata is high-confidence provenance. Heuristic verdicts can be wrong for generated scaffolds, formatter-only changes, disciplined commit conventions, bot commits, and large refactors. +## Evaluation and Measured Limitations + +The committed evaluation corpus is in [`eval/labels.tsv`](eval/labels.tsv), with +generated fixtures from [`eval/generate-fixtures.sh`](eval/generate-fixtures.sh). +Regenerate the current report with: + +```bash +eval/report.py --output eval/sample-report.md +eval/sensitivity.py --output eval/sensitivity-report.md +eval/differential.py --output eval/differential-report.md +``` + +Current sample metrics are committed in +[`eval/sample-report.md`](eval/sample-report.md). On the small generated corpus, +the latest report shows overall precision/recall/F1 of 1.000, recorded +provenance precision/recall/F1 of 1.000, metadata F1 of 0.667, and +heuristic-only F1 of 1.000. These numbers are guardrails, not a broad accuracy +claim: the corpus has 10 labeled cases, includes one ambiguous case excluded +from rate denominators, and is intentionally fixture-sized. + +Known false-positive classes: + +- generated/vendor or lockfile updates if explicitly included; +- formatter-only or mechanical refactor commits with broad regular diffs; +- bot commits whose metadata resembles AI-agent identity; +- disciplined commit-message templates that match generated prose patterns; +- repetitive framework scaffolds written by humans. + +Known false-negative classes: + +- AI-assisted changes with no recorded notes, no agent metadata, and ordinary + human-style diffs; +- small edits below meaningful heuristic thresholds; +- copied AI-generated code edited enough to avoid similarity/style signals; +- unsupported provenance formats outside the documented Git AI note shapes; +- remote/model evidence when no explicit local provider is configured. + +The differential report in +[`eval/differential-report.md`](eval/differential-report.md) compares overlapping +fixture cases with installed `sniff` and treats git-ai `refs/notes/ai` as the +recorded-provenance reference. Cadence comparison is recorded when `CADENCE_BIN` +points at a prebuilt executable; otherwise it is reported as unavailable rather +than building or modifying the sibling checkout. + ## Development ```bash