λgitsite

~ober/jerboa-aigit

Imported from ~/mine/jerboa-aigit

download snapshot

about

# jerboa-aigit

`jerboa-aigit` scans Git history for evidence that commits or changed lines may
have been AI-assisted. It separates recorded provenance from heuristics:

- Git AI notes in `refs/notes/ai` are reported as recorded authorship evidence.
- Agent/tool metadata in author identity, commit messages, and notes is reported
  separately.
- Message, code, structure, history, baseline, and SimHash heuristics are scored
  as probabilistic evidence, not proof.

The current implementation is offline-only. It never calls an LLM or sends
repository data over the network.

## Usage

```bash
./bin/jerboa-aigit scan /path/to/repo --count 25
./bin/jerboa-aigit scan /path/to/repo --format json
./bin/jerboa-aigit scan /path/to/repo --config aigit.json
./bin/jerboa-aigit scan /path/to/repo --from main~20 --to HEAD --file src/app.ss
./bin/jerboa-aigit explain HEAD /path/to/repo --format markdown
./bin/jerboa-aigit stats /path/to/repo --count 100
./bin/jerboa-aigit verify-authorship /path/to/repo --count 25
```

During development, the same entrypoint can be run directly:

```bash
jerboa main-binary.ss scan /path/to/repo --format json
```

Supported options are `--config FILE`, `--count N`, `--all`, `--from REV`,
`--to REV`, `--commit REV`, `--file PATH`, `--include PATH`,
`--exclude PREFIX`, `--first-parent`, `--all-parents`, `--min-lines N`, `--max-files N`,
`--max-added-lines N`, `--max-note-bytes N`,
`--threshold HUMAN,AI`, `--threshold HUMAN,UNCERTAIN,AI`,
`--no-llm`, `--llm`, `--provider NAME`,
`--format table|json|jsonl|markdown`, `--metadata-only`, and
`--heuristics-only`.

`--config FILE` reads guarded JSON. Supported scan keys are `path`, `count`,
`format`, `from`, `to`, `file`, `exclude`, `min_lines`, `metadata_only`, and
`heuristics_only`. Supported scoring keys are `human_threshold`,
`ai_threshold`, `weight_text`, `weight_code`, `weight_structure`,
`weight_similarity`, `weight_history`, `weight_baseline`, `max_files`,
`max_added_lines`, `max_note_bytes`, `git_timeout_seconds`, and
`provider_timeout_seconds`. Later CLI flags override earlier config values.

Threshold CLI values override the active config for the current invocation.
The scanner has two operative cutoffs: scores below `HUMAN` are
`likely-human-style`, scores from `HUMAN` to below `AI` are `mixed-uncertain`,
and scores at or above `AI` are `likely-ai-assisted`. The three-value form is
accepted for compatibility with `human,uncertain,ai` wording; the middle value
is retained as a label boundary only and does not create a separate verdict.

`--no-llm` is the default. `--llm --provider local` enables an optional local
provider adapter when either `local_provider_command` is set in the JSON config
as an argv array or `JERBOA_AIGIT_LOCAL_PROVIDER` points at an executable. The
scanner sends one bounded JSON payload as the final argv item and expects strict
JSON on stdout with a numeric `score` plus optional `reason` and `evidence`.
The provider executable must be an absolute canonical path outside the scanned
repository. Repo-local, symlinked, missing, and group/world-writable provider
commands are refused so scanning never executes repository code or unsafe local
adapters.
Local provider output is reported as a secondary `model` signal; it never
overwrites recorded authorship metadata. Remote/network providers are not
implemented in this release, and `network_used` remains `false`.

`stats` reports commit count, distinct authors, recorded tools, recorded AI line
counts, metadata/heuristic counts, score-band counts, and signal-category
coverage for the selected revision range. Use `stats --format json` for the
same summary as a stable machine-readable object.

`explain` prints the selected commit's thresholds, raw signals, weighted signal
contributions, corroboration bonus, counter-evidence from zero-score signals,
and warnings.

Merge commits default to first-parent diffs. Use `--all-parents` to inspect
per-parent merge diffs via Git's `-m` mode; merge warnings state which mode was
used.

By default, scans skip `vendor/`, `generated/`, `dist/`, `node_modules/`, and
`.git/` paths. Use `--include PATH` or `--file PATH` to inspect one of those
paths explicitly. `--include` and `--file` accept repository-relative pathspecs;
absolute paths and `..` components are refused before invoking Git. UTF-8 paths
and paths containing spaces, tabs, newlines, and non-ASCII characters are
covered by fixture tests using NUL-delimited Git parsing where supported.

Resource limits default to 500 changed files per commit, 20,000 added lines
analyzed per commit, and 50,000 bytes per AI note. When a limit is hit, output
keeps the bounded data and includes a warning.

Git subprocesses and local provider subprocesses are timeout-bound. Git timeout
warnings mean partial scan data may be missing; provider timeout warnings mean
the scanner fell back to offline heuristics.

Baseline signals are reported as contextual evidence with zero default weight.
The author baseline uses same-author addition history; the repository baseline
uses median/MAD-style addition-count deviation across the selected scan window.

Warnings also call out root commits with no parent baseline, merge commits where
the scanner intentionally uses the first-parent diff, missing parent objects in
shallow history, rename/copy changes, and binary file changes that are skipped
by text-line heuristics.

## What It Reads

The scanner runs Git commands against the requested repository using fixed
argument lists, not shell-interpolated command strings. It reads commit
metadata, first-parent diffs, `--numstat`, added patch lines, and AI authorship
notes from `refs/notes/ai`. It does not execute repository code and does not
write to the scanned worktree, refs, notes, hooks, or config.

## Output Contract

JSON output includes:

- `detector_version`
- `config_hash`
- analysis provenance fields: `analysis_provider`, `provider`, `llm_used`,
  and `network_used`
- repository path
- commit and parent IDs
- author/committer-facing metadata
- changed files, additions, deletions, and added line count
- recorded AI note presence and excerpt
- structured `recorded_attribution` entries for supported Git AI note line
  ranges (`source`, `tool`, `model`, `session`, `path`, `start`, `end`)
- structured `recovered_attribution` entries for inferred identity evidence
  (`source`, `agent`, `confidence`, `recorded_provenance:false`, `evidence`)
- `file_findings` entries that attach matching recorded note ranges to changed
  files without duplicating commit-level heuristic scores
- `unmatched_recorded_attribution` for recorded ranges whose path was not part
  of the changed-file set
- metadata hits
- raw signal scores, weights, evidence, and limitations
- aggregate score and verdict
- warnings for missing or unavailable evidence

Verdicts are intentionally conservative:

- `recorded-ai-authorship`
- `metadata-indicated-agent`
- `likely-ai-assisted`
- `mixed-uncertain`
- `likely-human-style`
- `insufficient-evidence`

Only recorded authorship metadata is high-confidence provenance. Heuristic
verdicts can be wrong for generated scaffolds, formatter-only changes,
disciplined commit conventions, bot commits, and large refactors.

## Evaluation and Measured Limitations

The committed evaluation corpus is in [`eval/labels.tsv`](eval/labels.tsv), with
generated fixtures from [`eval/generate-fixtures.sh`](eval/generate-fixtures.sh).
Regenerate the current report with:

```bash
eval/report.py --output eval/sample-report.md
eval/sensitivity.py --output eval/sensitivity-report.md
eval/differential.py --output eval/differential-report.md
```

Current sample metrics are committed in
[`eval/sample-report.md`](eval/sample-report.md). On the small generated corpus,
the latest report shows overall precision/recall/F1 of 1.000, recorded
provenance precision/recall/F1 of 1.000, metadata F1 of 0.667, and
heuristic-only F1 of 1.000. These numbers are guardrails, not a broad accuracy
claim: the corpus has 10 labeled cases, includes one ambiguous case excluded
from rate denominators, and is intentionally fixture-sized.

Known false-positive classes:

- generated/vendor or lockfile updates if explicitly included;
- formatter-only or mechanical refactor commits with broad regular diffs;
- bot commits whose metadata resembles AI-agent identity;
- disciplined commit-message templates that match generated prose patterns;
- repetitive framework scaffolds written by humans.

Known false-negative classes:

- AI-assisted changes with no recorded notes, no agent metadata, and ordinary
  human-style diffs;
- small edits below meaningful heuristic thresholds;
- copied AI-generated code edited enough to avoid similarity/style signals;
- unsupported provenance formats outside the documented Git AI note shapes;
- remote/model evidence when no explicit local provider is configured.

The differential report in
[`eval/differential-report.md`](eval/differential-report.md) compares overlapping
fixture cases with installed `sniff` and treats git-ai `refs/notes/ai` as the
recorded-provenance reference. Cadence comparison is recorded when `CADENCE_BIN`
points at a prebuilt executable; otherwise it is reported as unavailable rather
than building or modifying the sibling checkout.

## Development

```bash
make test
make verify
make binary
```

All Jerboa source is in `main-binary.ss`. Per `AGENTS.md`, edit `.ss` files only
with Jerboa MCP balanced tools and run balance plus verification after every
change.

recent commits