Report upstream current oracle errors
ober
55798c9ac4468a1206a267abf64a0136aa92cf01
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,6 +1,6 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 03:50 MDT +Date: 2026-05-29 04:00 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` @@ -98,6 +98,18 @@ The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner errors. The full sweep is slow and can run for several minutes because it invokes upstream Semgrep for each sorted case. +The current-error cases are now visible when `LIST_MISMATCHES=1`: + +```text +ERROR current anywhere_global +ERROR current anywhere_metavar +``` + +Both package-current failures come from empty `metadata:` keys in the upstream +fixture YAML. Packaged Semgrep 1.164.0 exits with +`InvalidRuleSchemaError: null values prohibited` before scanning. Jerboa +currently accepts these fixtures and finds the expected `ruleid` comments. + ## What Changed In This Checkpoint This checkpoint clears the last five known Jerboa mismatches in the current @@ -230,10 +242,9 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Recommended Next Target There is no known Jerboa mismatch in the current 241-case sorted upstream -sweep. The next useful work is to identify the two packaged-Semgrep -`current_error` cases and decide whether they should be excluded from this -oracle, pinned to a different Semgrep binary, or used to expand the Jerboa -compatibility corpus once upstream current can produce JSON for them. +sweep. The two packaged-Semgrep `current_error` cases are +`anywhere_global` and `anywhere_metavar`, both rejected by current CLI rule +validation because their upstream test YAML contains an empty `metadata:`. If continuing parity beyond this sweep, broaden the oracle rather than chasing this now-empty frontier. Candidate next expansions: @@ -242,8 +253,9 @@ this now-empty frontier. Candidate next expansions: - Add parser-specific cases for TypeScript/TSX beyond the current JavaScript parser fallback. - Add CLI/target-selection snapshot parity from the Semgrep checkout. -- Identify the two `current_error` cases by temporarily instrumenting - `tests/oracle/upstream-sweep.sh` to print current Semgrep failures. +- Decide whether the two null-metadata upstream fixtures should remain + accepted by Jerboa for fixture compatibility or be rejected under a stricter + current-CLI validation mode. ## Useful Commands --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -94,6 +94,14 @@ while IFS= read -r rule; do if ! SEMGREP_SEND_METRICS=off run_json "$case_name.current" "$current_json" \ "$SEMGREP_CURRENT" scan --quiet --json --config "$rule" "$target"; then current_error=$((current_error + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "ERROR current $case_name" + fi + if [[ "$diffs_shown" -lt "$MAX_DIFFS" ]]; then + echo "ERROR current $case_name" >&2 + sed 's/^/ /' "$TMPDIR/$case_name.current.stderr" >&2 + diffs_shown=$((diffs_shown + 1)) + fi continue fi