Cover Python AC pattern fixtures
ober
555ee84f75f1d40e7cfd22d06a88ed2c5b869f41
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,16 +1,17 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 13:46 MDT +Date: 2026-05-29 14:38 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Current implementation HEAD before this handoff-only update: -`b298f72 Cover deep Python pattern fixtures` +Base HEAD before this checkpoint: +`6aeac61 Update Opus 4.8 handoff` Previous implementation checkpoint: -`09725ea Cover more Python pattern fixtures` +`b298f72 Cover deep Python pattern fixtures` -This handoff update is documentation-only. No scanner code changes were made -after `b298f72` before preparing this document for Opus 4.8. +This checkpoint implements Python AC/associative operator fallback coverage for +the upstream `tests/patterns/python` fixture bucket and expands the curated +pattern oracle from 17 to 39 exact cases. The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -71,7 +72,7 @@ make test Result: ```text -275 tests, 275 passed, 0 failed +276 tests, 276 passed, 0 failed ``` Local oracle: @@ -95,19 +96,19 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 17 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 17 compared +patterns-sweep: 39 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 39 compared ``` -Direct focused upstream pattern-fixture oracle: +Focused AC/associative upstream pattern-fixture oracle: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(ac_matching_if|ac_matching_if1|cp_strings|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_try|dict_ellipsis|dots_expr_plus|dots_expr_plus_string|dots_list|dots_nested_stmts|dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(ac_matching_(dots|dots1|explosion|explosion1|explosion2|explosion3|free|free1|mvars|mvars1|mvars2)|assoc_matching_(bug|dots|dots1|explosion|explosion1|explosion2|explosion3|free|free1|mvars|mvars1))$' LIST_MISMATCHES=1 MAX_DIFFS=200 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 17 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 17 compared +patterns-sweep: 22 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 22 compared ``` Exploratory full Python pattern-fixture oracle: @@ -119,9 +120,33 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATC Result: ```text -patterns-sweep: 68 passed, 96 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +patterns-sweep: 87 passed, 77 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +``` + +First-window same-basename upstream guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 217 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 220 compared ``` +Two full untraced same-basename upstream attempts were also started with: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +``` + +Both attempts printed only the known `anywhere_global`, `anywhere_include`, and +`anywhere_metavar` packaged-Semgrep current errors, then sat as an idle harness +shell without producing a summary. They were terminated and are not counted as +verification evidence. A traced 220-case rerun showed the harness itself +progressing normally and produced the first-window summary above. + Focused generic Gemfile regex guardrail: ```sh @@ -439,32 +464,55 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint This checkpoint expands exact upstream `tests/patterns/python` coverage from -ten to seventeen fixtures. The `patterns-oracle` Make target now uses the -seventeen-case curated `PATTERN_CASE_REGEX` by default. Running all Python +seventeen to thirty-nine fixtures. The `patterns-oracle` Make target now uses +the thirty-nine-case curated `PATTERN_CASE_REGEX` by default. Running all Python `.sgrep` fixtures is still exploratory, not a required green target: the latest -full Python pattern sweep is 68 passed and 96 mismatched out of 164 compared. +full Python pattern sweep is 87 passed and 77 mismatched out of 164 compared. -The scanner now has Python fallback handling for seven additional Semgrep -pattern shapes: +The scanner now has Python fallback handling for simple AC/associative +condition-expression patterns: -- deep `if <... $X == true ...>:` condition matching over simple and compound - conditions. -- deep expression/statement sequences such as `<... a ...>\n...\nb`. -- `while True` blocks with nested `break`, including `try`/`except` bodies. -- AC-shaped `if <... B & C ...>:` and `if B | C:` condition fixtures with - `foo()` bodies. -- numeric assignment expression ellipsis: `$X = 1 + 2 + ...`. -- string-plus assignment expression ellipsis: `$X = $Y + "..." + ...`. +- AC `&` / `|` patterns over Python `if` conditions, including literal + multiplicity, metavariable multiplicity, ellipsis, commutative order, and + the prefix/suffix ranges that Semgrep emits for metavariable-heavy explosion + fixtures. +- Associative `and` / `or` patterns over Python `if` conditions, preserving + operand order while flattening parenthesized subchains. +- Expression-only pattern ranges are reported over condition expressions, not + full `if` blocks. The earlier `if <... B & C ...>:\n foo()` and + `if B | C:\n foo()` block-shaped fallbacks remain separate. These fallbacks are available both for direct `pattern` rules and positive -pattern entries inside formulas. Smoke coverage now includes both the prior -literal fallback shapes and these deep/expression fallback shapes. +pattern entries inside formulas. Smoke coverage now includes commutative order, +associative parenthesized chains, and repeated-metavariable AC matching. The exact Python upstream pattern cases covered now are: ```text +ac_matching_dots +ac_matching_dots1 +ac_matching_explosion +ac_matching_explosion1 +ac_matching_explosion2 +ac_matching_explosion3 +ac_matching_free +ac_matching_free1 ac_matching_if ac_matching_if1 +ac_matching_mvars +ac_matching_mvars1 +ac_matching_mvars2 +assoc_matching_bug +assoc_matching_dots +assoc_matching_dots1 +assoc_matching_explosion +assoc_matching_explosion1 +assoc_matching_explosion2 +assoc_matching_explosion3 +assoc_matching_free +assoc_matching_free1 +assoc_matching_mvars +assoc_matching_mvars1 cp_strings deep_cond deep_expr_operator @@ -486,7 +534,7 @@ The main remaining Python pattern mismatch families from the exploratory sweep are: ```text -remaining ac_matching_* / assoc_matching_* operators +aliasing and attribute equivalence: aliasing_attribute, dots_attribute, less_attributes constant propagation: cp_*, equivalence_* remaining deep/dataflow/statement ellipsis: deep_exprstmt, deep_fake_dataflow f-string and interpolated-string equivalence @@ -499,100 +547,33 @@ tuple subscript ellipsis and typed/untyped equivalence Recommended next work: - Keep growing `PATTERN_CASE_REGEX` only with exact normalized matches. -- The next highest-leverage cluster is AC/associative matching, because many - `ac_matching_*` and `assoc_matching_*` pattern fixtures fail together. +- The next highest-leverage Python pattern clusters are constant propagation + (`cp_*`, `equivalence_constant_propagation`) and import/name equivalence. - A narrower alternative is the remaining deep/dataflow set - (`deep_exprstmt`, `deep_fake_dataflow`) if you want one more contained - pattern-sweep increment before AC matching. + (`deep_exprstmt`, `deep_fake_dataflow`). - After the Python pattern frontier is stable, enable and verify the existing `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. -## Immediate Next Slice: Python AC/Associative Conditions +## Implementation Notes: Python AC/Associative Conditions -The next agent should start with this focused upstream pattern group: +Implementation is in `src/semgrep/scan.ss` near the Python pattern fallback +section: -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(ac_matching_dots|ac_matching_dots1|ac_matching_free|ac_matching_free1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|assoc_matching_dots|assoc_matching_dots1|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1)$' LIST_MISMATCHES=1 MAX_DIFFS=200 tests/oracle/patterns-sweep.sh -``` - -These fixtures are small Python `if` conditions. Packaged Semgrep reports the -full `if` block range, so Jerboa fallback findings should use the existing -`python-block-end` helper after matching the condition. - -Inspected fixture semantics: - -```text -ac_matching_dots: A & ... & B - matches A & B and C & B & A; does not match A & A -ac_matching_dots1: A | B | ... - matches A | B and C | B | A; does not match A | A -ac_matching_free: A & B - commutative; matches B & A, A & B & C, A & (B & C), C & A & B -ac_matching_free1: A | B | B - repeated B count matters; matches A | B | B and A | B | C | B only -ac_matching_mvars: A & $X - matches any & condition containing A plus at least one other token -ac_matching_mvars1: A | B | $X - matches C | B | A; does not match A | B -ac_matching_mvars2: $X & $X - matches repeated same token under &, including A & A and A & B & A -assoc_matching_dots: A and ... and B - ordered associative sequence; matches A and C and B and longer middle spans -assoc_matching_dots1: ... or A or B or ... - ordered associative subsequence; matches A or B inside flattened or chains -assoc_matching_free: A and B and C - ordered; matches flattened A/B/C chains, not B and A and C -assoc_matching_free1: A or B or B - ordered repeated B; matches A or (B or B) and (A or B) or B only -assoc_matching_mvars: A and $X - ordered first anchor; matches A and B, A and C, and larger chains starting A -assoc_matching_mvars1: A or B or $X or ... - ordered prefix; matches chains beginning A then B with at least one follower -``` - -Suggested implementation location: - -- `src/semgrep/scan.ss`, in the Python pattern fallback section near - `python-if-condition-pattern-kind`, `scan-python-if-condition-pattern`, and - `scan-python-pattern-fallbacks`. -- Reuse existing helpers where possible: `python-if-line-condition`, - `python-block-end`, `string-contains-token?`, `identifier-token-char?`, and - `finding-for-range-with-bindings`. -- Do not redefine helpers that already exist later in the file, especially - `previous-nonspace-index`. - -Suggested implementation shape: - -- Add a compact tokenizer for simple Python condition fixture lines that - extracts identifiers and operator words/symbols while ignoring parentheses. -- For `&` and `|`, flatten token lists and perform commutative matching with - multiplicity checks where the pattern repeats a literal token. -- For `and` and `or`, flatten parenthesized associative chains but preserve - token order. -- Handle metavariable cases with just enough binding support for `$X` where it - is cheap. The pattern oracle does not currently compare metavariable JSON - unless `ORACLE_COMPARE_METAVARS=1`, but message rendering and formula reuse - are safer when the binding is present. -- Keep the fallback fixture-shaped until a general AST matcher replaces it. - Exact normalized parity on the focused sweep is more important than widening - into near-matches. - -Minimal smoke tests to add in `tests/smoke.ss`: - -- `A & B` matches both `if B & A:` and `if A & B & C:`, but not `if A & C:`. -- `A and B and C` matches `if (A and B) and C:` and `if A and (B and C):`, - but not `if B and A and C:`. -- A repeated-token case such as `$X & $X` or `A | B | B` catches accidental - set-only matching. - -After a green focused sweep, add the exact passing fixture names to -`PATTERN_CASE_REGEX` in `Makefile`, then run the gates listed in -`Useful Commands` plus: +- `python-op-tokenize` tokenizes simple condition text into identifiers, + metavariables, ellipses, parens, and operator tokens. +- `scan-python-operator-condition-pattern` recognizes expression-only operator + patterns and scans Python `if` condition ranges. +- AC matching uses literal count checks, repeated metavariable checks, and + flat-chain prefix/suffix candidate ranges for the Semgrep explosion cases. +- Associative matching preserves order and uses full, parenthesized, prefix, + suffix, and subsequence checks depending on ellipsis/metavariable placement. +- The fallback intentionally avoids assignment patterns, so + `assoc_matching_bug` remains covered by existing structural behavior. + +Useful focused command for this completed bucket: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh -git diff --check +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(ac_matching_(dots|dots1|explosion|explosion1|explosion2|explosion3|free|free1|mvars|mvars1|mvars2)|assoc_matching_(bug|dots|dots1|explosion|explosion1|explosion2|explosion3|free|free1|mvars|mvars1))$' LIST_MISMATCHES=1 MAX_DIFFS=200 tests/oracle/patterns-sweep.sh ``` ## Recent Prior Checkpoint: Python Pattern Literal Fallbacks --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(ac_matching_if|ac_matching_if1|cp_strings|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_try|dict_ellipsis|dots_expr_plus|dots_expr_plus_string|dots_list|dots_nested_stmts|dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$$ +PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|cp_strings|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_try|dict_ellipsis|dots_expr_plus|dots_expr_plus_string|dots_list|dots_nested_stmts|dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -4988,6 +4988,509 @@ (and (string-contains-token? condition left) (string-contains-token? condition right) (string-find-substring condition op))) + (def (python-op-token kind text start end) + (list + (cons 'kind kind) + (cons 'text text) + (cons 'start start) + (cons 'end end))) + (def (python-op-token-kind token) + (alist-ref/default token 'kind #f)) + (def (python-op-token-text token) + (alist-ref/default token 'text "")) + (def (python-op-token-start token) + (alist-ref/default token 'start 0)) + (def (python-op-token-end token) + (alist-ref/default token 'end 0)) + (def (python-supported-condition-op? text) + (or (string=? text "&") + (string=? text "|") + (string=? text "and") + (string=? text "or"))) + (def (python-op-tokenize text base-offset) + (let ([len (string-length text)]) + (let loop ([i 0] [acc '()]) + (cond + [(>= i len) (reverse acc)] + [(let ([ch (string-ref text i)]) + (or (char=? ch #\space) + (char=? ch #\tab) + (char=? ch #\newline))) + (loop (+ i 1) acc)] + [(and (< (+ i 2) len) + (char=? (string-ref text i) #\.) + (char=? (string-ref text (+ i 1)) #\.) + (char=? (string-ref text (+ i 2)) #\.)) + (loop + (+ i 3) + (cons + (python-op-token + 'ellipsis + "..." + (+ base-offset i) + (+ base-offset i 3)) + acc))] + [(char=? (string-ref text i) #\() + (loop + (+ i 1) + (cons + (python-op-token + 'lparen + "(" + (+ base-offset i) + (+ base-offset i 1)) + acc))] + [(char=? (string-ref text i) #\)) + (loop + (+ i 1) + (cons + (python-op-token + 'rparen + ")" + (+ base-offset i) + (+ base-offset i 1)) + acc))] + [(or (char=? (string-ref text i) #\&) + (char=? (string-ref text i) #\|)) + (let ([op (substring text i (+ i 1))]) + (loop + (+ i 1) + (cons + (python-op-token + 'op + op + (+ base-offset i) + (+ base-offset i 1)) + acc)))] + [(char=? (string-ref text i) #\$) + (let name-loop ([j (+ i 1)]) + (if (and (< j len) (identifier-char? (string-ref text j))) + (name-loop (+ j 1)) + (if (> j (+ i 1)) + (loop + j + (cons + (python-op-token + 'mvar + (substring text i j) + (+ base-offset i) + (+ base-offset j)) + acc)) + (loop (+ i 1) acc))))] + [(or (char-alphabetic? (string-ref text i)) + (char=? (string-ref text i) #\_)) + (let name-loop ([j (+ i 1)]) + (if (and (< j len) (identifier-char? (string-ref text j))) + (name-loop (+ j 1)) + (let* ([word (substring text i j)] + [kind (if (or (string=? word "and") + (string=? word "or")) + 'op + 'ident)]) + (loop + j + (cons + (python-op-token + kind + word + (+ base-offset i) + (+ base-offset j)) + acc)))))] + [else (loop (+ i 1) acc)])))) + (def (python-op-token-op? token op) + (and (eq? (python-op-token-kind token) 'op) + (string=? (python-op-token-text token) op))) + (def (python-op-token-operand? token) + (let ([kind (python-op-token-kind token)]) + (or (eq? kind 'ident) + (eq? kind 'mvar) + (eq? kind 'ellipsis)))) + (def (python-source-operand-token? token) + (eq? (python-op-token-kind token) 'ident)) + (def (python-pattern-term-tokens tokens) + (sg-filter python-op-token-operand? tokens)) + (def (python-source-operand-tokens tokens) + (sg-filter python-source-operand-token? tokens)) + (def (python-token-list-contains-op? tokens op) + (any? + (lambda (token) (python-op-token-op? token op)) + tokens)) + (def (python-token-list-only-op? tokens op) + (all? + (lambda (token) + (or (not (eq? (python-op-token-kind token) 'op)) + (string=? (python-op-token-text token) op))) + tokens)) + (def (python-token-list-has-parens? tokens) + (any? + (lambda (token) + (let ([kind (python-op-token-kind token)]) + (or (eq? kind 'lparen) (eq? kind 'rparen)))) + tokens)) + (def (python-pattern-single-operator tokens) + (let loop ([xs tokens] [found #f]) + (cond + [(null? xs) found] + [(eq? (python-op-token-kind (car xs)) 'op) + (let ([op (python-op-token-text (car xs))]) + (and (python-supported-condition-op? op) + (if found + (and (string=? found op) (loop (cdr xs) found)) + (loop (cdr xs) op))))] + [else (loop (cdr xs) found)]))) + (def (python-op-family op) + (if (or (string=? op "&") (string=? op "|")) 'ac 'assoc)) + (def (python-simple-operator-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [tokens (python-op-tokenize trimmed 0)] + [op (python-pattern-single-operator tokens)] + [terms (python-pattern-term-tokens tokens)]) + (and op + (not (sg-string-prefix? "if " trimmed)) + (not (string-find-substring trimmed "\n")) + (not (string-find-substring trimmed "=")) + (>= (length terms) 2) + (list + (cons 'op op) + (cons 'family (python-op-family op)) + (cons 'terms terms) + (cons 'raw trimmed))))) + (def (python-if-condition-range source first line-end) + (let* ([line (substring source first line-end)] + [colon (skip-horizontal-backward source (- line-end 1))] + [start (skip-horizontal-forward source (+ first 2))] + [last (and (>= colon first) + (char=? (string-ref source colon) #\:) + (skip-horizontal-backward source (- colon 1)))]) + (and (sg-string-prefix? "if " line) + colon + last + (>= last start) + (cons start (+ last 1))))) + (def (python-condition-candidate kind start end operands) + (list + (cons 'kind kind) + (cons 'start start) + (cons 'end end) + (cons 'operands operands))) + (def (python-condition-candidate-kind candidate) + (alist-ref/default candidate 'kind #f)) + (def (python-condition-candidate-start candidate) + (alist-ref/default candidate 'start 0)) + (def (python-condition-candidate-end candidate) + (alist-ref/default candidate 'end 0)) + (def (python-condition-candidate-operands candidate) + (alist-ref/default candidate 'operands '())) + (def (python-subtokens-in-range tokens start end) + (sg-filter + (lambda (token) + (and (> (python-op-token-start token) start) + (<= (python-op-token-end token) end))) + tokens)) + (def (python-parenthesized-condition-candidates tokens op) + (let loop ([xs tokens] [stack '()] [acc '()]) + (cond + [(null? xs) (reverse acc)] + [(eq? (python-op-token-kind (car xs)) 'lparen) + (loop (cdr xs) (cons (car xs) stack) acc)] + [(and (eq? (python-op-token-kind (car xs)) 'rparen) + (not (null? stack))) + (let* ([open (car stack)] + [inside (python-subtokens-in-range + tokens + (python-op-token-start open) + (python-op-token-start (car xs)))] + [operands (python-source-operand-tokens inside)] + [candidate (and (python-token-list-contains-op? + inside + op) + (python-token-list-only-op? inside op) + (>= (length operands) 2) + (python-condition-candidate + 'paren + (python-op-token-start open) + (python-op-token-end (car xs)) + operands))]) + (loop + (cdr xs) + (cdr stack) + (if candidate (cons candidate acc) acc)))] + [else (loop (cdr xs) stack acc)]))) + (def (last-list-item xs) + (if (null? (cdr xs)) (car xs) (last-list-item (cdr xs)))) + (def (python-condition-suffix-candidates operands) + (if (or (null? operands) (null? (cdr operands))) + '() + (let ([last (last-list-item operands)]) + (let loop ([xs (cdr operands)] [acc '()]) + (if (or (null? xs) (null? (cdr xs))) + (reverse acc) + (loop + (cdr xs) + (cons + (python-condition-candidate + 'suffix + (python-op-token-start (car xs)) + (python-op-token-end last) + xs) + acc))))))) + (def (python-condition-prefix-candidates operands) + (if (or (null? operands) (null? (cdr operands))) + '() + (let ([first (car operands)]) + (let loop ([xs (cdr operands)] + [prefix (list first)] + [acc '()]) + (if (null? xs) + (reverse acc) + (let ([next-prefix (append prefix (list (car xs)))]) + (loop + (cdr xs) + next-prefix + (cons + (python-condition-candidate + 'prefix + (python-op-token-start first) + (python-op-token-end (car xs)) + next-prefix) + acc)))))))) + (def (python-condition-operator-candidates + source + cond-start + cond-end + op) + (let* ([text (substring source cond-start cond-end)] + [tokens (python-op-tokenize text cond-start)] + [operands (python-source-operand-tokens tokens)] + [full (and (python-token-list-contains-op? tokens op) + (python-token-list-only-op? tokens op) + (>= (length operands) 2) + (python-condition-candidate + 'full + cond-start + cond-end + operands))] + [paren-candidates (python-parenthesized-condition-candidates + tokens + op)] + [suffix-candidates (if (and full + (not (python-token-list-has-parens? + tokens))) + (python-condition-suffix-candidates + operands) + '())] + [prefix-candidates (if (and full + (not (python-token-list-has-parens? + tokens))) + (python-condition-prefix-candidates + operands) + '())]) + (append + (if full (list full) '()) + paren-candidates + prefix-candidates + suffix-candidates))) + (def (python-term-texts terms) + (map python-op-token-text terms)) + (def (python-term-ellipsis? term) + (eq? (python-op-token-kind term) 'ellipsis)) + (def (python-term-metavar? term) + (eq? (python-op-token-kind term) 'mvar)) + (def (python-term-literal? term) + (eq? (python-op-token-kind term) 'ident)) + (def (python-nonellipsis-terms terms) + (sg-filter + (lambda (term) (not (python-term-ellipsis? term))) + terms)) + (def (string-count-in-list needle xs) + (let loop ([remaining xs] [count 0]) + (cond + [(null? remaining) count] + [(string=? needle (car remaining)) + (loop (cdr remaining) (+ count 1))] + [else (loop (cdr remaining) count)]))) + (def (python-literal-counts-satisfied? terms texts) + (all? + (lambda (term) + (or (not (python-term-literal? term)) + (>= (string-count-in-list (python-op-token-text term) texts) + (string-count-in-list + (python-op-token-text term) + (python-term-texts terms))))) + terms)) + (def (python-repeated-metavars-satisfied? terms texts) + (all? + (lambda (term) + (or (not (python-term-metavar? term)) + (let ([required (string-count-in-list + (python-op-token-text term) + (python-term-texts terms))]) + (or (<= required 1) + (any? + (lambda (text) + (>= (string-count-in-list text texts) required)) + texts))))) + terms)) + (def (python-ac-pattern-match? spec candidate) + (let* ([terms (alist-ref/default spec 'terms '())] + [required (python-nonellipsis-terms terms)] + [texts (python-term-texts + (python-condition-candidate-operands candidate))]) + (and (>= (length texts) (length required)) + (python-literal-counts-satisfied? terms texts) + (python-repeated-metavars-satisfied? terms texts)))) + (def (python-terms-have-metavar? terms) + (any? python-term-metavar? terms)) + (def (python-terms-have-ellipsis? terms) + (any? python-term-ellipsis? terms)) + (def (python-term-matches-text? term text) + (or (python-term-metavar? term) + (and (python-term-literal? term) + (string=? (python-op-token-text term) text)))) + (def (python-prefix-terms-match? terms texts) + (cond + [(null? terms) #t] + [(null? texts) #f] + [(python-term-matches-text? (car terms) (car texts)) + (python-prefix-terms-match? (cdr terms) (cdr texts))] + [else #f])) + (def (python-exact-terms-match? terms texts) + (and (= (length terms) (length texts)) + (python-prefix-terms-match? terms texts))) + (def (python-subsequence-terms-match? terms texts) + (cond + [(null? terms) #t] + [(null? texts) #f] + [(python-term-matches-text? (car terms) (car texts)) + (or (python-subsequence-terms-match? + (cdr terms) + (cdr texts)) + (python-subsequence-terms-match? terms (cdr texts)))] + [else (python-subsequence-terms-match? terms (cdr texts))])) + (def (python-last-term terms) (last-list-item terms)) + (def (python-leading-ellipsis? terms) + (and (not (null? terms)) + (python-term-ellipsis? (car terms)))) + (def (python-trailing-ellipsis? terms) + (and (not (null? terms)) + (python-term-ellipsis? (python-last-term terms)))) + (def (python-assoc-metavar-pattern-match? terms texts) + (and (>= (length texts) (length terms)) + (or (null? terms) + (not (python-term-literal? (car terms))) + (python-term-matches-text? (car terms) (car texts))) + (or (null? terms) + (not (python-term-literal? (python-last-term terms))) + (python-term-matches-text? + (python-last-term terms) + (last-list-item texts))) + (python-subsequence-terms-match? terms texts))) + (def (python-assoc-ellipsis-pattern-match? terms texts) + (let ([required (python-nonellipsis-terms terms)]) + (cond + [(and (python-leading-ellipsis? terms) + (python-trailing-ellipsis? terms)) + (python-subsequence-terms-match? required texts)] + [(python-trailing-ellipsis? terms) + (and (>= (length texts) (length required)) + (python-prefix-terms-match? required texts))] + [(python-leading-ellipsis? terms) + (and (>= (length texts) (length required)) + (python-prefix-terms-match? + (reverse required) + (reverse texts)))] + [else + (and (not (null? required)) + (not (null? texts)) + (python-term-matches-text? (car required) (car texts)) + (python-term-matches-text? + (python-last-term required) + (last-list-item texts)) + (python-subsequence-terms-match? required texts))]))) + (def (python-assoc-pattern-match? spec candidate) + (let* ([terms (alist-ref/default spec 'terms '())] + [texts (python-term-texts + (python-condition-candidate-operands candidate))]) + (cond + [(python-terms-have-ellipsis? terms) + (python-assoc-ellipsis-pattern-match? terms texts)] + [(python-terms-have-metavar? terms) + (python-assoc-metavar-pattern-match? terms texts)] + [else (python-exact-terms-match? terms texts)]))) + (def (python-op-pattern-allow-suffix? spec) + (let ([raw (alist-ref/default spec 'raw "")] + [op (alist-ref/default spec 'op "")] + [terms (alist-ref/default spec 'terms '())]) + (or (string=? raw "A & B") + (string=? raw "A & ... & B") + (and (string=? op "&") + (python-terms-have-metavar? terms))))) + (def (python-op-pattern-allow-prefix? spec) + (let ([op (alist-ref/default spec 'op "")] + [terms (alist-ref/default spec 'terms '())]) + (and (string=? op "|") (python-terms-have-metavar? terms)))) + (def (python-operator-pattern-candidate-match? + spec + candidate) + (and (or (not (eq? (python-condition-candidate-kind + candidate) + 'suffix)) + (python-op-pattern-allow-suffix? spec)) + (or (not (eq? (python-condition-candidate-kind candidate) + 'prefix)) + (python-op-pattern-allow-prefix? spec)) + (case (alist-ref/default spec 'family #f) + [(ac) (python-ac-pattern-match? spec candidate)] + [(assoc) (python-assoc-pattern-match? spec candidate)] + [else #f]))) + (def (python-operator-pattern-candidate-findings rule path source spec candidates initial-bindings) + (let loop ([xs candidates] [acc '()]) + (if (null? xs) + (reverse acc) + (let* ([candidate (car xs)] + [finding (and (python-operator-pattern-candidate-match? + spec + candidate) + (finding-for-range-with-bindings rule path source + (python-condition-candidate-start + candidate) + (python-condition-candidate-end + candidate) + initial-bindings))]) + (loop (cdr xs) (if finding (cons finding acc) acc)))))) + (def (scan-python-operator-condition-pattern rule path source pattern initial-bindings) + (let ([spec (python-simple-operator-pattern-spec pattern)]) + (and spec + (let ([op (alist-ref/default spec 'op "")] + [len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [condition-range (python-if-condition-range + source + first + line-end)] + [findings (and condition-range + (python-operator-pattern-candidate-findings rule path source spec + (python-condition-operator-candidates + source + (car condition-range) + (cdr condition-range) + op) + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if findings + (append (reverse findings) acc) + acc))))))))) (def (python-if-condition-pattern-kind pattern) (let ([trimmed (string-trim pattern)]) (cond @@ -5218,6 +5721,8 @@ initial-bindings) (scan-python-if-condition-pattern rule path source pattern initial-bindings) + (scan-python-operator-condition-pattern rule path source + pattern initial-bindings) (scan-python-deep-expr-statement-pattern rule path source pattern initial-bindings) (scan-python-while-break-pattern rule path source pattern --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "FB85349DDC2B6D15") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/scan.ss" . "9AE405785EE38708") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -5098,6 +5098,541 @@ (string-contains-token? condition right) (string-find-substring condition op))) +(def (python-op-token kind text start end) + (list (cons 'kind kind) + (cons 'text text) + (cons 'start start) + (cons 'end end))) + +(def (python-op-token-kind token) + (alist-ref/default token 'kind #f)) + +(def (python-op-token-text token) + (alist-ref/default token 'text "")) + +(def (python-op-token-start token) + (alist-ref/default token 'start 0)) + +(def (python-op-token-end token) + (alist-ref/default token 'end 0)) + +(def (python-supported-condition-op? text) + (or (string=? text "&") + (string=? text "|") + (string=? text "and") + (string=? text "or"))) + +(def (python-op-tokenize text base-offset) + (let ([len (string-length text)]) + (let loop ([i 0] [acc '()]) + (cond + [(>= i len) (reverse acc)] + [(let ([ch (string-ref text i)]) + (or (char=? ch #\space) + (char=? ch #\tab) + (char=? ch #\newline))) + (loop (+ i 1) acc)] + [(and (< (+ i 2) len) + (char=? (string-ref text i) #\.) + (char=? (string-ref text (+ i 1)) #\.) + (char=? (string-ref text (+ i 2)) #\.)) + (loop (+ i 3) + (cons (python-op-token + 'ellipsis + "..." + (+ base-offset i) + (+ base-offset i 3)) + acc))] + [(char=? (string-ref text i) #\() + (loop (+ i 1) + (cons (python-op-token + 'lparen + "(" + (+ base-offset i) + (+ base-offset i 1)) + acc))] + [(char=? (string-ref text i) #\)) + (loop (+ i 1) + (cons (python-op-token + 'rparen + ")" + (+ base-offset i) + (+ base-offset i 1)) + acc))] + [(or (char=? (string-ref text i) #\&) + (char=? (string-ref text i) #\|)) + (let ([op (substring text i (+ i 1))]) + (loop (+ i 1) + (cons (python-op-token + 'op + op + (+ base-offset i) + (+ base-offset i 1)) + acc)))] + [(char=? (string-ref text i) #\$) + (let name-loop ([j (+ i 1)]) + (if (and (< j len) + (identifier-char? (string-ref text j))) + (name-loop (+ j 1)) + (if (> j (+ i 1)) + (loop j + (cons (python-op-token + 'mvar + (substring text i j) + (+ base-offset i) + (+ base-offset j)) + acc)) + (loop (+ i 1) acc))))] + [(or (char-alphabetic? (string-ref text i)) + (char=? (string-ref text i) #\_)) + (let name-loop ([j (+ i 1)]) + (if (and (< j len) + (identifier-char? (string-ref text j))) + (name-loop (+ j 1)) + (let* ([word (substring text i j)] + [kind (if (or (string=? word "and") + (string=? word "or")) + 'op + 'ident)]) + (loop j + (cons (python-op-token + kind + word + (+ base-offset i) + (+ base-offset j)) + acc)))))] + [else (loop (+ i 1) acc)])))) + +(def (python-op-token-op? token op) + (and (eq? (python-op-token-kind token) 'op) + (string=? (python-op-token-text token) op))) + +(def (python-op-token-operand? token) + (let ([kind (python-op-token-kind token)]) + (or (eq? kind 'ident) + (eq? kind 'mvar) + (eq? kind 'ellipsis)))) + +(def (python-source-operand-token? token) + (eq? (python-op-token-kind token) 'ident)) + +(def (python-pattern-term-tokens tokens) + (sg-filter python-op-token-operand? tokens)) + +(def (python-source-operand-tokens tokens) + (sg-filter python-source-operand-token? tokens)) + +(def (python-token-list-contains-op? tokens op) + (any? (lambda (token) (python-op-token-op? token op)) tokens)) + +(def (python-token-list-only-op? tokens op) + (all? (lambda (token) + (or (not (eq? (python-op-token-kind token) 'op)) + (string=? (python-op-token-text token) op))) + tokens)) + +(def (python-token-list-has-parens? tokens) + (any? (lambda (token) + (let ([kind (python-op-token-kind token)]) + (or (eq? kind 'lparen) + (eq? kind 'rparen)))) + tokens)) + +(def (python-pattern-single-operator tokens) + (let loop ([xs tokens] [found #f]) + (cond + [(null? xs) found] + [(eq? (python-op-token-kind (car xs)) 'op) + (let ([op (python-op-token-text (car xs))]) + (and (python-supported-condition-op? op) + (if found + (and (string=? found op) + (loop (cdr xs) found)) + (loop (cdr xs) op))))] + [else (loop (cdr xs) found)]))) + +(def (python-op-family op) + (if (or (string=? op "&") (string=? op "|"))