Include Cadence in differential report
ober
c07399ce63c753d8344d96a5ed989b4e6dda756a
--- a/GAPS.md +++ b/GAPS.md @@ -105,9 +105,9 @@ Acceptance criteria: Status: implemented with `eval/differential.py` and committed `eval/differential-report.md`. The report compares overlapping generated-corpus cases with installed `sniff`, checks git-ai-compatible `refs/notes/ai` -presence, records Cadence availability via `CADENCE_BIN`, and documents -intentional differences around provenance separation, offline defaults, and -vendor/generated exclusions. +presence, includes Cadence summaries from a temp-built `CADENCE_BIN`, and +documents intentional differences around provenance separation, offline +defaults, and vendor/generated exclusions. ## P1: High-value sniff parity --- a/eval/differential-report.md +++ b/eval/differential-report.md @@ -11,13 +11,13 @@ Intentional differences: | case | label | jerboa band | jerboa verdict | sniff band | cadence | git-ai notes | agreement notes | |---|---|---|---|---|---|---|---| -| human_basic | human | likely-human-style | likely-human-style | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| ai_recorded_note | ai_assisted | recorded | recorded-ai-authorship | Likely AI-assisted | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | refs/notes/ai present | recorded-note agreement; sniff returned overlapping commit | -| ai_heuristic_style | ai_assisted | likely-ai-assisted | likely-ai-assisted | Likely AI-assisted | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| metadata_attested_author | metadata_attested | metadata | metadata-indicated-agent | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| bot_dependency_update | bot | insufficient-evidence | insufficient-evidence | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| generated_vendor | generated_vendor | insufficient-evidence | insufficient-evidence | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| formatter_only | formatter_only | likely-human-style | likely-human-style | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| mechanical_refactor | refactor | likely-human-style | likely-human-style | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| merge_commit | merge | likely-human-style | likely-human-style | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | -| ambiguous_assistance | ambiguous | likely-human-style | likely-human-style | Likely Human | CADENCE_BIN not set to an executable; skipped to avoid building/writing in sibling checkout | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| human_basic | human | likely-human-style | likely-human-style | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| ai_recorded_note | ai_assisted | recorded | recorded-ai-authorship | Likely AI-assisted | detections=9; flagged=9; analyzed=12 | refs/notes/ai present | recorded-note agreement; sniff returned overlapping commit | +| ai_heuristic_style | ai_assisted | likely-ai-assisted | likely-ai-assisted | Likely AI-assisted | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| metadata_attested_author | metadata_attested | metadata | metadata-indicated-agent | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| bot_dependency_update | bot | insufficient-evidence | insufficient-evidence | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| generated_vendor | generated_vendor | insufficient-evidence | insufficient-evidence | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| formatter_only | formatter_only | likely-human-style | likely-human-style | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| mechanical_refactor | refactor | likely-human-style | likely-human-style | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| merge_commit | merge | likely-human-style | likely-human-style | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | +| ambiguous_assistance | ambiguous | likely-human-style | likely-human-style | Likely Human | detections=9; flagged=9; analyzed=12 | no refs/notes/ai note | recorded-note agreement; sniff returned overlapping commit | --- a/eval/differential.py +++ b/eval/differential.py @@ -86,7 +86,12 @@ def cadence_case(root, row): run([cadence_bin, "analyze", str(repo), "--output", name], cwd=Path(tmp)) output = Path(tmp) / "reports" / name data = json.loads(output.read_text()) if output.exists() else {} - return {"available": True, "summary": f"detections={data.get('detection_count', data.get('DetectionCount', 'unknown'))}"} + detections = data.get("detections", []) + detected = sum(1 for item in detections if item.get("detected")) + metrics = data.get("sourceMetrics", {}) + flagged = metrics.get("itemsFlagged", detected) + analyzed = metrics.get("itemsAnalyzed", "unknown") + return {"available": True, "summary": f"detections={detected}; flagged={flagged}; analyzed={analyzed}"} except Exception as exc: return {"available": False, "summary": str(exc)}