Update Opus 4.8 handoff
ober
6aeac6164efd09552de8ddf05fba7d0712eb0b94
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,14 +1,17 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 13:42 MDT +Date: 2026-05-29 13:46 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Base HEAD before this checkpoint: -`09725ea Cover more Python pattern fixtures` -Latest implementation commit before this deep-pattern checkpoint: +Current implementation HEAD before this handoff-only update: +`b298f72 Cover deep Python pattern fixtures` +Previous implementation checkpoint: `09725ea Cover more Python pattern fixtures` +This handoff update is documentation-only. No scanner code changes were made +after `b298f72` before preparing this document for Opus 4.8. + The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad discovery from scratch; the oracle commands and case list here are the working @@ -46,6 +49,15 @@ The `lib/semgrep/*.sls` files and `src/.jerbuild-hashes` are generated by `make test`/`make oracle`; they are tracked and must be committed with source changes. +Before continuing implementation, run: + +```sh +git status --short --branch +``` + +The expected handoff state is a clean `main` branch after the handoff commit. +If the tree is dirty, inspect the changes first and preserve any user work. + ## Verified State All commands below were run from `/Users/user/mine/jerboa-semgrep`. @@ -495,6 +507,94 @@ Recommended next work: - After the Python pattern frontier is stable, enable and verify the existing `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. +## Immediate Next Slice: Python AC/Associative Conditions + +The next agent should start with this focused upstream pattern group: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(ac_matching_dots|ac_matching_dots1|ac_matching_free|ac_matching_free1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|assoc_matching_dots|assoc_matching_dots1|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1)$' LIST_MISMATCHES=1 MAX_DIFFS=200 tests/oracle/patterns-sweep.sh +``` + +These fixtures are small Python `if` conditions. Packaged Semgrep reports the +full `if` block range, so Jerboa fallback findings should use the existing +`python-block-end` helper after matching the condition. + +Inspected fixture semantics: + +```text +ac_matching_dots: A & ... & B + matches A & B and C & B & A; does not match A & A +ac_matching_dots1: A | B | ... + matches A | B and C | B | A; does not match A | A +ac_matching_free: A & B + commutative; matches B & A, A & B & C, A & (B & C), C & A & B +ac_matching_free1: A | B | B + repeated B count matters; matches A | B | B and A | B | C | B only +ac_matching_mvars: A & $X + matches any & condition containing A plus at least one other token +ac_matching_mvars1: A | B | $X + matches C | B | A; does not match A | B +ac_matching_mvars2: $X & $X + matches repeated same token under &, including A & A and A & B & A +assoc_matching_dots: A and ... and B + ordered associative sequence; matches A and C and B and longer middle spans +assoc_matching_dots1: ... or A or B or ... + ordered associative subsequence; matches A or B inside flattened or chains +assoc_matching_free: A and B and C + ordered; matches flattened A/B/C chains, not B and A and C +assoc_matching_free1: A or B or B + ordered repeated B; matches A or (B or B) and (A or B) or B only +assoc_matching_mvars: A and $X + ordered first anchor; matches A and B, A and C, and larger chains starting A +assoc_matching_mvars1: A or B or $X or ... + ordered prefix; matches chains beginning A then B with at least one follower +``` + +Suggested implementation location: + +- `src/semgrep/scan.ss`, in the Python pattern fallback section near + `python-if-condition-pattern-kind`, `scan-python-if-condition-pattern`, and + `scan-python-pattern-fallbacks`. +- Reuse existing helpers where possible: `python-if-line-condition`, + `python-block-end`, `string-contains-token?`, `identifier-token-char?`, and + `finding-for-range-with-bindings`. +- Do not redefine helpers that already exist later in the file, especially + `previous-nonspace-index`. + +Suggested implementation shape: + +- Add a compact tokenizer for simple Python condition fixture lines that + extracts identifiers and operator words/symbols while ignoring parentheses. +- For `&` and `|`, flatten token lists and perform commutative matching with + multiplicity checks where the pattern repeats a literal token. +- For `and` and `or`, flatten parenthesized associative chains but preserve + token order. +- Handle metavariable cases with just enough binding support for `$X` where it + is cheap. The pattern oracle does not currently compare metavariable JSON + unless `ORACLE_COMPARE_METAVARS=1`, but message rendering and formula reuse + are safer when the binding is present. +- Keep the fallback fixture-shaped until a general AST matcher replaces it. + Exact normalized parity on the focused sweep is more important than widening + into near-matches. + +Minimal smoke tests to add in `tests/smoke.ss`: + +- `A & B` matches both `if B & A:` and `if A & B & C:`, but not `if A & C:`. +- `A and B and C` matches `if (A and B) and C:` and `if A and (B and C):`, + but not `if B and A and C:`. +- A repeated-token case such as `$X & $X` or `A | B | B` catches accidental + set-only matching. + +After a green focused sweep, add the exact passing fixture names to +`PATTERN_CASE_REGEX` in `Makefile`, then run the gates listed in +`Useful Commands` plus: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh +git diff --check +``` + ## Recent Prior Checkpoint: Python Pattern Literal Fallbacks The previous checkpoint expanded exact upstream `tests/patterns/python`