Add labeled evaluation corpus format
ober
d4deb6e29004af4de6eece51fd07a10711fa2796
--- a/GAPS.md +++ b/GAPS.md @@ -50,6 +50,11 @@ Acceptance criteria: - Store labels separately from scanner output. - Document how labels were assigned and how ambiguity is represented. +Status: implemented with `eval/labels.tsv`, `eval/manual-corpus.example.tsv`, +`eval/README.md`, and `eval/generate-fixtures.sh`. The generated fixture repo +is reproducible and ignored rather than committed; labels use stable tags and +remain separate from scanner output. + ### G-002: No precision/recall report The scanner has no measured false-positive or false-negative rates. new file mode 100644 --- /dev/null +++ b/eval/README.md @@ -0,0 +1,46 @@ +# Evaluation corpus + +This directory defines the parity/accuracy evaluation corpus used by +`jerboa-aigit`. Labels are intentionally stored separately from scanner output +so detector changes cannot rewrite ground truth. + +## Label format + +Labels are tab-separated: + +```text +case_id repo rev label mode ambiguous notes +``` + +- `case_id`: stable case name. +- `repo`: path to the fixture or external repository, relative to this project + when possible. +- `rev`: tag, commit, or ref to scan. +- `label`: one of `human`, `ai_assisted`, `metadata_attested`, `bot`, + `generated_vendor`, `formatter_only`, `refactor`, `merge`, `ambiguous`. +- `mode`: expected evidence family: `recorded`, `metadata`, `heuristic`, + `non_ai`, or `mixed`. +- `ambiguous`: `true` when label assignment is intentionally uncertain. +- `notes`: short rationale. This is ground-truth context, not scanner output. + +## Generated fixture repo + +Run: + +```sh +eval/generate-fixtures.sh +``` + +The script creates `eval/fixtures/labeled-repo` with tagged commits covering +human, AI-assisted, metadata-attested, bot, generated/vendor, formatter-only, +refactor, merge, and ambiguous cases. It is safe to delete and regenerate that +directory. + +## Manual/external corpus + +`manual-corpus.example.tsv` documents the format for manually curated or +external repositories. Do not commit private or proprietary source snapshots. +Commit only labels and concise provenance notes that are safe to share. + +Ambiguity is explicit: use `label=ambiguous`, `mode=mixed`, and +`ambiguous=true` when reviewers cannot defensibly assign a single source. new file mode 100644 --- /dev/null +++ b/eval/fixtures/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore new file mode 100755 --- /dev/null +++ b/eval/generate-fixtures.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env sh +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd) +repo="$root/eval/fixtures/labeled-repo" + +rm -rf "$repo" +mkdir -p "$repo" +git -C "$repo" init -q +git -C "$repo" config user.name "Human Dev" +git -C "$repo" config user.email "human@example.test" + +commit_with_identity() { + name="$1" + email="$2" + tag="$3" + message="$4" + date="$5" + shift 5 + git -C "$repo" add "$@" + GIT_AUTHOR_NAME="$name" \ + GIT_AUTHOR_EMAIL="$email" \ + GIT_COMMITTER_NAME="$name" \ + GIT_COMMITTER_EMAIL="$email" \ + GIT_AUTHOR_DATE="$date" \ + GIT_COMMITTER_DATE="$date" \ + git -C "$repo" commit -q -m "$message" + git -C "$repo" tag "$tag" +} + +printf 'human baseline\n' > "$repo/README.md" +commit_with_identity "Human Dev" "human@example.test" "human-basic" "document human baseline" "2026-07-29T08:00:00-06:00" README.md + +mkdir -p "$repo/src" +i=1 +while [ "$i" -le 40 ]; do + printf 'def generated_helper_%s(value):\n """This function provides robust handling for the provided value."""\n result = value + %s\n return result\n\n' "$i" "$i" >> "$repo/src/generated.py" + i=$((i + 1)) +done +commit_with_identity "Human Dev" "human@example.test" "ai-recorded-note" "add generated helper implementation" "2026-07-29T08:01:00-06:00" src/generated.py +git -C "$repo" notes --ref=ai add -m '{"tool":"codex","model":"gpt-5","lines":[{"path":"src/generated.py","start":1,"end":160}]}' HEAD + +printf 'metadata attested\n' >> "$repo/README.md" +commit_with_identity "Codex" "codex@openai.example" "metadata-attested-author" "codex assisted update" "2026-07-29T08:02:00-06:00" README.md + +mkdir -p "$repo/vendor" +printf '{"lockfileVersion":3,"packages":{"example":{"version":"1.2.3"}}}\n' > "$repo/vendor/package-lock.json" +commit_with_identity "Dependabot" "dependabot@example.test" "bot-dependency-update" "update dependency lockfile" "2026-07-29T08:03:00-06:00" vendor/package-lock.json + +mkdir -p "$repo/vendor/generated" +printf 'generated artifact\n' > "$repo/vendor/generated/client.js" +commit_with_identity "Human Dev" "human@example.test" "generated-vendor" "update generated vendor client" "2026-07-29T08:04:00-06:00" vendor/generated/client.js + +printf 'human baseline\n\n' > "$repo/README.md" +commit_with_identity "Human Dev" "human@example.test" "formatter-only" "format README spacing" "2026-07-29T08:05:00-06:00" README.md + +printf 'def renamed_helper(value):\n return value + 1\n' > "$repo/src/refactor.py" +commit_with_identity "Human Dev" "human@example.test" "mechanical-refactor" "rename helper during refactor" "2026-07-29T08:06:00-06:00" src/refactor.py + +base=$(git -C "$repo" rev-parse HEAD) +main_branch=$(git -C "$repo" rev-parse --abbrev-ref HEAD) +git -C "$repo" checkout -q -b side-branch "$base" +printf 'side branch\n' > "$repo/side.txt" +commit_with_identity "Human Dev" "human@example.test" "side-branch-change" "side branch change" "2026-07-29T08:07:00-06:00" side.txt +git -C "$repo" checkout -q "$main_branch" +printf 'main branch\n' > "$repo/main.txt" +commit_with_identity "Human Dev" "human@example.test" "main-branch-change" "main branch change" "2026-07-29T08:08:00-06:00" main.txt +GIT_AUTHOR_NAME="Human Dev" \ +GIT_AUTHOR_EMAIL="human@example.test" \ +GIT_COMMITTER_NAME="Human Dev" \ +GIT_COMMITTER_EMAIL="human@example.test" \ +GIT_AUTHOR_DATE="2026-07-29T08:09:00-06:00" \ +GIT_COMMITTER_DATE="2026-07-29T08:09:00-06:00" \ + git -C "$repo" merge -q --no-ff side-branch -m "merge side branch" +git -C "$repo" tag "merge-commit" + +printf 'possible assisted text\n' >> "$repo/README.md" +commit_with_identity "Human Dev" "human@example.test" "ambiguous-assistance" "polish implementation with assistant suggestions" "2026-07-29T08:10:00-06:00" README.md + +printf 'generated %s\n' "$repo" new file mode 100644 --- /dev/null +++ b/eval/labels.tsv @@ -0,0 +1,10 @@ +case_id repo rev label mode ambiguous notes +human_basic eval/fixtures/labeled-repo human-basic human non_ai false Small ordinary human-authored README change with human author identity. +ai_recorded_note eval/fixtures/labeled-repo ai-recorded-note ai_assisted recorded false Commit has refs/notes/ai recorded line attribution. +metadata_attested_author eval/fixtures/labeled-repo metadata-attested-author metadata_attested metadata false Author identity uses a known configured/default AI agent marker. +bot_dependency_update eval/fixtures/labeled-repo bot-dependency-update bot metadata false Bot author updates a dependency lock-style file. +generated_vendor eval/fixtures/labeled-repo generated-vendor generated_vendor non_ai false Generated/vendor path should be excluded or treated separately from authored code. +formatter_only eval/fixtures/labeled-repo formatter-only formatter_only non_ai false Whitespace/format-only style change should not be treated as AI authorship by itself. +mechanical_refactor eval/fixtures/labeled-repo mechanical-refactor refactor heuristic false Mechanical rename/refactor without AI provenance. +merge_commit eval/fixtures/labeled-repo merge-commit merge non_ai false Merge commit exercises parent/shape handling. +ambiguous_assistance eval/fixtures/labeled-repo ambiguous-assistance ambiguous mixed true Message suggests assistance but no recorded provenance is available. new file mode 100644 --- /dev/null +++ b/eval/manual-corpus.example.tsv @@ -0,0 +1,4 @@ +case_id repo rev label mode ambiguous notes +external_human_example /path/to/manual/repo <commit> human non_ai false Reviewer-confirmed human-authored change; include reviewer/date in local private copy. +external_ai_example /path/to/manual/repo <commit> ai_assisted recorded false Reviewer-confirmed AI-assisted change or recorded refs/notes/ai provenance. +external_ambiguous_example /path/to/manual/repo <commit> ambiguous mixed true Signals conflict or provenance is incomplete; do not force into human/AI.