Cover more Python pattern fixtures
ober
09725ea176eb5209a88ac6e92dcf3e831e514444
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,13 +1,13 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 12:37 MDT +Date: 2026-05-29 13:13 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`efbfd7a Cover Java upstream fixtures` -Latest implementation commit before this pattern-oracle checkpoint: -`efbfd7a Cover Java upstream fixtures` +`9962fe1 Add upstream pattern fixture oracle` +Latest implementation commit before this pattern-expansion checkpoint: +`9962fe1 Add upstream pattern fixture oracle` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -39,7 +39,6 @@ HANDOFF_OPUS_4_8.md src/semgrep/scan.ss lib/semgrep/scan.sls src/.jerbuild-hashes -tests/oracle/patterns-sweep.sh tests/smoke.ss ``` @@ -60,7 +59,7 @@ make test Result: ```text -273 tests, 273 passed, 0 failed +274 tests, 274 passed, 0 failed ``` Local oracle: @@ -84,19 +83,31 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared +patterns-sweep: 10 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 10 compared ``` Direct focused upstream pattern-fixture oracle: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(cp_strings|deep_expr_operator|dict_ellipsis|dots_list|dots_nested_stmts|dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared +patterns-sweep: 10 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 10 compared +``` + +Exploratory full Python pattern-fixture oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 61 passed, 103 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` Focused generic Gemfile regex guardrail: @@ -415,38 +426,35 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint -This checkpoint opens a second upstream oracle surface based on Semgrep's -`tests/patterns` fixtures. The existing `tests/oracle/upstream-sweep.sh` only -compares same-basename rule/target pairs under `tests/rules`; that surface is -now exhausted for the currently supported text-backed suffix set. The new -`tests/oracle/patterns-sweep.sh` generates temporary YAML rules from upstream -`.sgrep` files, runs packaged Semgrep and Jerboa against the sibling target -fixture, normalizes JSON findings with the existing normalizer, sorts them, and -diffs exact finding ranges/messages/check IDs. +This checkpoint expands exact upstream `tests/patterns/python` coverage from +five to ten fixtures. The `patterns-oracle` Make target now uses the ten-case +curated `PATTERN_CASE_REGEX` by default. Running all Python `.sgrep` fixtures +is still exploratory, not a required green target: the latest full Python +pattern sweep is 61 passed and 103 mismatched out of 164 compared. -The new harness supports: - -- `SEMGREP_ROOT`, `SEMGREP_CURRENT`, `SCHEME`, `JERBOA_TREESITTER_HOME`, - `LIBDIRS`, and `PATTERN_ROOT` overrides. -- `PATTERN_LANGS` for selecting upstream pattern directories. `python` and - `js` are mapped initially; this checkpoint verifies only the Python slice. -- `CASE_REGEX`, `MAX_CASES`, `LIST_MISMATCHES`, and `MAX_DIFFS`, matching the - style of `upstream-sweep.sh`. -- stable generated rule IDs under `oracle.patterns.*`, so temporary config - paths do not produce false check-id diffs. -- no-target/empty-pattern skip accounting and separate packaged-current, - Jerboa, and normalized-output mismatch counts. - -`Makefile` now has a `patterns-oracle` target that builds the tracked Jerboa -libraries and then runs the curated passing pattern slice by default through -`PATTERN_CASE_REGEX`. Use `make patterns-oracle PATTERN_CASE_REGEX=...` to grow -the green slice or `tests/oracle/patterns-sweep.sh` directly for exploratory -sweeps. Running all Python `.sgrep` fixtures is intentionally still an -exploratory parity sweep, not a passing target. +The scanner now has Python fallback handling for five Semgrep pattern shapes +that were previously known gaps: + +- deep expression calls such as `foo(<... 42 ...>)`, reporting the whole call. +- typed `Dict[...] = {}` default parameters in function definitions, reporting + the function definition and body range. +- `if (...): ...` statement ellipsis over an entire `if`/`else` chain. +- `return "..."` string ellipsis, including parenthesized raw/byte/unicode + string concatenations. +- `[...]` list ellipsis for list literals and comprehensions. + +These fallbacks are available both for direct `pattern` rules and positive +pattern entries inside formulas. A compact smoke test covers all five fallback +shapes. The exact Python upstream pattern cases covered now are: ```text +cp_strings +deep_expr_operator +dict_ellipsis +dots_list +dots_nested_stmts dots_stmts import_metavar metavar_equality_var @@ -454,42 +462,50 @@ metavar_stmt misc_dots_stmts ``` -This checkpoint also fixes one real Python structural-range parity gap exposed -by `tests/patterns/python/import_metavar.sgrep`. When the structural matcher -returns a Python module-root finding, Jerboa now trims leading comment/blank -fixture lines and trailing final newlines from the reported range. That matches -packaged Semgrep for the import-alias fixture where the target begins with -`#ERROR: match`. The trim is applied only to Python-like structural findings -without an explicit internal match-range binding, preserving focused and -range-bound findings. A smoke test locks the exact range: +The main remaining Python pattern mismatch families from the exploratory sweep +are: ```text -start line/col/offset: 2 / 1 / 14 -end line/col/offset: 4 / 34 / 76 +ac_matching_* / assoc_matching_* operators +constant propagation: cp_*, equivalence_* +deeper expression/statement ellipsis: deep_cond, deep_expr_vs_statement, deep_try +f-string and interpolated-string equivalence +import/name equivalence and wildcard-qualified names +metavariable annotation/class/function/typed patterns +misc parser range/faketok/statement edge cases +tuple subscript ellipsis and typed/untyped equivalence ``` -Known next Python pattern gaps from the quick probe: +Recommended next work: -```text -deep_expr_operator packaged current 1 / Jerboa 0 -dict_ellipsis packaged current 1 / Jerboa 0 -dots_nested_stmts packaged current 1 / Jerboa 0 -cp_strings packaged current 3 / Jerboa 0 -dots_list packaged current 2 / Jerboa 1 -``` +- Keep growing `PATTERN_CASE_REGEX` only with exact normalized matches. +- The next coherent cluster is generalized deep expression and statement + ellipsis (`deep_cond`, `deep_expr_vs_statement`, `deep_try`) rather than more + one-off literal fallbacks. +- The highest-leverage broader cluster after that is AC/associative matching, + because many `ac_matching_*` and `assoc_matching_*` pattern fixtures fail + together. +- After the Python pattern frontier is stable, enable and verify the existing + `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. -Recommended next work: +## Recent Prior Checkpoint: Pattern Oracle Harness + +The previous checkpoint opened the upstream `tests/patterns` oracle surface. +`tests/oracle/patterns-sweep.sh` generates temporary YAML rules from upstream +`.sgrep` files, runs packaged Semgrep and Jerboa against the sibling target +fixture, normalizes JSON findings with the existing normalizer, sorts them, and +diffs exact finding ranges/messages/check IDs. + +The harness supports `SEMGREP_ROOT`, `SEMGREP_CURRENT`, `SCHEME`, +`JERBOA_TREESITTER_HOME`, `LIBDIRS`, `PATTERN_ROOT`, `PATTERN_LANGS`, +`CASE_REGEX`, `MAX_CASES`, `LIST_MISMATCHES`, and `MAX_DIFFS`. Generated rule +IDs live under `oracle.patterns.*` so temporary config paths do not create +false check-id diffs. -- Grow `patterns-sweep.sh` from the five exact Python fixtures, keeping every - newly green case in a documented `CASE_REGEX` until the entire directory - passes. -- Implement Python deep-expression operator support for `<... ... ...>` style - matching. -- Improve Python dict/list ellipsis matching; `dict_ellipsis` and `dots_list` - are the most direct fixtures. -- Add the missing string constant-propagation behavior needed by `cp_strings`. -- After the Python pattern frontier is stable, enable and verify the `js` - pattern directory mapping already present in the harness. +That prior checkpoint also fixed a Python module-root structural range gap for +`tests/patterns/python/import_metavar.sgrep`: leading fixture comments/blank +lines and trailing final newlines are trimmed from Python module-root findings +when there is no explicit internal match-range binding. ## Recent Prior Checkpoint: Java Same-Basename Fixtures --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$$ +PATTERN_CASE_REGEX ?= ^(cp_strings|deep_expr_operator|dict_ellipsis|dots_list|dots_nested_stmts|dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -4649,6 +4649,343 @@ (< start end)) (finding-with-range finding source start end) finding))) + (def (python-simple-string-literal-prefix-char? ch) + (or (char=? ch #\r) + (char=? ch #\R) + (char=? ch #\u) + (char=? ch #\U) + (char=? ch #\b) + (char=? ch #\B) + (char=? ch #\f) + (char=? ch #\F))) + (def (python-string-literal-at? source i end) + (let loop ([j i]) + (cond + [(>= j end) #f] + [(or (char=? (string-ref source j) #\") + (char=? (string-ref source j) #\')) + #t] + [(python-simple-string-literal-prefix-char? + (string-ref source j)) + (loop (+ j 1))] + [else #f]))) + (def (python-range-has-string-literal? source start end) + (let loop ([i start]) + (and (< i end) + (or (and (token-start-boundary? source i) + (python-string-literal-at? source i end)) + (loop (+ i 1)))))) + (def (find-matching-close-bracket source open-index) + (let ([len (string-length source)]) + (let loop ([i open-index] + [depth 0] + [state 'normal] + [escaped? #f]) + (cond + [(>= i len) #f] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) depth 'double #f)] + [(char=? ch #\') (loop (+ i 1) depth 'single #f)] + [(char=? ch #\[) (loop (+ i 1) (+ depth 1) state #f)] + [(char=? ch #\]) + (if (= depth 1) + (+ i 1) + (loop (+ i 1) (max 0 (- depth 1)) state #f))] + [else (loop (+ i 1) depth state #f)]))] + [escaped? (loop (+ i 1) depth state #f)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) depth state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) depth 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) depth 'normal #f)] + [else (loop (+ i 1) depth state #f)])))) + (def (python-list-literal-open? source index) + (let ([prev (previous-nonspace-index source index)]) + (or (not prev) + (let ([ch (string-ref source prev)]) + (not (or (identifier-token-char? ch) + (char=? ch #\]) + (char=? ch #\)) + (char=? ch #\") + (char=? ch #\'))))))) + (def (python-return-string-ellipsis-pattern? pattern) + (string=? (string-trim pattern) "return \"...\"")) + (def (python-return-statement-end source first line-end) + (let ([expr-start (skip-horizontal-forward + source + (+ first (string-length "return")))]) + (if (and (< expr-start (string-length source)) + (char=? (string-ref source expr-start) #\()) + (or (find-matching-close-paren source expr-start) line-end) + line-end))) + (def (scan-python-return-string-ellipsis-pattern rule path source pattern initial-bindings) + (and (python-return-string-ellipsis-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [statement-end (and (sg-string-prefix? + "return" + line) + (let ([after (+ first + (string-length + "return"))]) + (and (token-end-boundary? + source + after) + (python-return-statement-end + source + first + line-end))))] + [finding (and statement-end + (python-range-has-string-literal? + source + first + statement-end) + (finding-for-range-with-bindings rule path source first + statement-end + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next (if finding (cons finding acc) acc)))))))) + (def (scan-python-list-ellipsis-pattern rule path source + pattern initial-bindings) + (and (string=? (string-trim pattern) "[...]") + (let ([len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([open (string-find-substring-from + source + "[" + start)]) + (if (not open) + (nonempty-findings (reverse acc)) + (let* ([close (and (python-list-literal-open? + source + open) + (find-matching-close-bracket + source + open))] + [finding (and close + (finding-for-range-with-bindings rule path source open + close initial-bindings))] + [next (if close + (max (+ open 1) close) + (+ open 1))]) + (loop + next + (if finding (cons finding acc) acc)))))))))) + (def (python-block-end + source + header-start + include-if-chain?) + (let* ([header-indent (line-indent-at-offset + source + header-start)] + [header-end (line-end-after source header-start)] + [len (string-length source)]) + (let loop ([line-start (if (< header-end len) + (+ header-end 1) + (+ len 1))] + [last-end header-end]) + (if (> line-start len) + last-end + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [blank? (= first line-end)] + [trimmed (and (not blank?) + (substring source first line-end))] + [comment? (and trimmed + (sg-string-prefix? "#" trimmed))] + [indent (- first line-start)] + [chain-line? (and include-if-chain? + (= indent header-indent) + trimmed + (or (sg-string-prefix? + "elif " + trimmed) + (sg-string-prefix? + "else:" + trimmed)))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (cond + [(or blank? comment?) (loop next last-end)] + [(or (> indent header-indent) chain-line?) + (loop next line-end)] + [else last-end])))))) + (def (python-if-ellipsis-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (and (sg-string-prefix? "if (...):" trimmed) + (string-find-substring trimmed "...")))) + (def (scan-python-if-ellipsis-pattern rule path source + pattern initial-bindings) + (and (python-if-ellipsis-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [end (and (sg-string-prefix? "if " line) + (sg-string-suffix? ":" line) + (python-block-end source first #t))] + [finding (and end + (finding-for-range-with-bindings rule path source first end + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next (if finding (cons finding acc) acc)))))))) + (def (python-deep-call-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)] + [open (char-index-from trimmed #\( 0 len)] + [deep-start (and open + (string-find-substring-from + trimmed + "<..." + (+ open 1)))] + [deep-end (and deep-start + (string-find-substring-from + trimmed + "...>" + (+ deep-start 4)))] + [tail-start (and deep-end (+ deep-end 4))] + [name (and open (string-trim (substring trimmed 0 open)))] + [needle (and deep-start + deep-end + (string-trim + (substring + trimmed + (+ deep-start 4) + deep-end)))] + [tail (and tail-start + (string-trim + (substring trimmed tail-start len)))]) + (and name + needle + (> (string-length name) 0) + (> (string-length needle) 0) + (string=? tail ")") + (list (cons 'function name) (cons 'needle needle))))) + (def (scan-python-deep-call-pattern rule path source pattern + initial-bindings) + (let ([spec (python-deep-call-pattern-spec pattern)]) + (and spec + (let* ([function (alist-ref/default spec 'function "")] + [needle (alist-ref/default spec 'needle "")] + [fn-len (string-length function)] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([index (string-find-substring-from + source + function + start)]) + (if (not index) + (nonempty-findings (reverse acc)) + (let* ([name-end (+ index fn-len)] + [open (and (token-start-boundary? + source + index) + (token-end-boundary? + source + name-end) + (skip-horizontal-forward + source + name-end))] + [close (and open + (< open len) + (char=? + (string-ref source open) + #\() + (find-matching-close-paren + source + open))] + [finding (and close + (string-find-substring + (substring + source + (+ open 1) + (- close 1)) + needle) + (finding-for-range-with-bindings rule path source index + close + initial-bindings))] + [next (if close + (max (+ index 1) close) + (+ index 1))]) + (loop + next + (if finding + (cons finding acc) + acc))))))))))) + (def (python-dict-default-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (and (sg-string-prefix? "def " trimmed) + (string-find-substring trimmed ": Dict[...]") + (string-find-substring trimmed "= {}")))) + (def (python-def-dict-default-line? line) + (and (sg-string-prefix? "def " line) + (string-find-substring line ": Dict[") + (string-find-substring line "= {}") + (sg-string-suffix? ":" line))) + (def (scan-python-dict-default-pattern rule path source + pattern initial-bindings) + (and (python-dict-default-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [end (and (python-def-dict-default-line? line) + (python-block-end source first #f))] + [finding (and end + (finding-for-range-with-bindings rule path source first end + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next (if finding (cons finding acc) acc)))))))) + (def (scan-python-pattern-fallbacks rule path source pattern + initial-bindings) + (or (scan-python-deep-call-pattern rule path source pattern + initial-bindings) + (scan-python-dict-default-pattern rule path source pattern + initial-bindings) + (scan-python-if-ellipsis-pattern rule path source pattern + initial-bindings) + (scan-python-return-string-ellipsis-pattern rule path source pattern initial-bindings) + (scan-python-list-ellipsis-pattern rule path source pattern + initial-bindings))) (def (symbolic-assignment-line-offset source start) (if (and (< start (string-length source)) (char=? (string-ref source start) #\newline)) @@ -6074,6 +6411,12 @@ (string=? language "python2") (string=? language "python3") (string=? language "py")) + (scan-python-pattern-fallbacks rule path source + (cdr entry) '())) + (and (or (string=? language "python") + (string=? language "python2") + (string=? language "python3") + (string=? language "py")) (scan-python-ellipsis-only-pattern rule path source target-root (cdr entry))) (and (or (string=? language "python") @@ -6299,6 +6642,12 @@ (string=? language "python2") (string=? language "python3") (string=? language "py")) + (scan-python-pattern-fallbacks rule path source + (cdr entry) initial-bindings)) + (and (or (string=? language "python") + (string=? language "python2") + (string=? language "python3") + (string=? language "py")) (scan-python-ellipsis-only-pattern rule path source target-root (cdr entry))) (and (or (string=? language "python") @@ -16909,6 +17258,12 @@ (string=? language "python2") (string=? language "python3") (string=? language "py")) + (scan-python-pattern-fallbacks rule path source + (rule-pattern rule) '())) + (and (or (string=? language "python") + (string=? language "python2") + (string=? language "python3") + (string=? language "py")) (scan-python-ellipsis-only-pattern rule path source target-root (rule-pattern rule))) (and (or (string=? language "python") (string=? language "python2") --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "C754F524EE6C647C") + ("src/semgrep/scan.ss" . "A97B12DDECA36F6B") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -4753,6 +4753,377 @@ (finding-with-range finding source start end) finding))) +(def (python-simple-string-literal-prefix-char? ch) + (or (char=? ch #\r) + (char=? ch #\R) + (char=? ch #\u) + (char=? ch #\U) + (char=? ch #\b) + (char=? ch #\B) + (char=? ch #\f) + (char=? ch #\F))) + +(def (python-string-literal-at? source i end) + (let loop ([j i]) + (cond + [(>= j end) #f] + [(or (char=? (string-ref source j) #\") + (char=? (string-ref source j) #\')) + #t] + [(python-simple-string-literal-prefix-char? (string-ref source j)) + (loop (+ j 1))] + [else #f]))) + +(def (python-range-has-string-literal? source start end) + (let loop ([i start]) + (and (< i end) + (or (and (token-start-boundary? source i) + (python-string-literal-at? source i end)) + (loop (+ i 1)))))) + +(def (find-matching-close-bracket source open-index) + (let ([len (string-length source)]) + (let loop ([i open-index] [depth 0] [state 'normal] [escaped? #f]) + (cond + [(>= i len) #f] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) depth 'double #f)] + [(char=? ch #\') (loop (+ i 1) depth 'single #f)] + [(char=? ch #\[) (loop (+ i 1) (+ depth 1) state #f)] + [(char=? ch #\]) + (if (= depth 1) + (+ i 1) + (loop (+ i 1) (max 0 (- depth 1)) state #f))] + [else (loop (+ i 1) depth state #f)]))] + [escaped? (loop (+ i 1) depth state #f)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) depth state #t)] + [(and (eq? state 'double) (char=? (string-ref source i) #\")) + (loop (+ i 1) depth 'normal #f)] + [(and (eq? state 'single) (char=? (string-ref source i) #\')) + (loop (+ i 1) depth 'normal #f)] + [else (loop (+ i 1) depth state #f)])))) + +(def (python-list-literal-open? source index) + (let ([prev (previous-nonspace-index source index)]) + (or (not prev) + (let ([ch (string-ref source prev)]) + (not (or (identifier-token-char? ch) + (char=? ch #\]) + (char=? ch #\)) + (char=? ch #\") + (char=? ch #\'))))))) + +(def (python-return-string-ellipsis-pattern? pattern) + (string=? (string-trim pattern) "return \"...\"")) + +(def (python-return-statement-end source first line-end) + (let ([expr-start (skip-horizontal-forward + source + (+ first (string-length "return")))]) + (if (and (< expr-start (string-length source)) + (char=? (string-ref source expr-start) #\()) + (or (find-matching-close-paren source expr-start) line-end) + line-end))) + +(def (scan-python-return-string-ellipsis-pattern + rule + path + source + pattern + initial-bindings) + (and (python-return-string-ellipsis-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [statement-end + (and (sg-string-prefix? "return" line) + (let ([after (+ first (string-length "return"))]) + (and (token-end-boundary? source after) + (python-return-statement-end + source + first + line-end))))] + [finding + (and statement-end + (python-range-has-string-literal? + source + first + statement-end) + (finding-for-range-with-bindings + rule + path + source + first + statement-end + initial-bindings))] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (loop next + (if finding (cons finding acc) acc)))))))) + +(def (scan-python-list-ellipsis-pattern + rule + path + source + pattern + initial-bindings) + (and (string=? (string-trim pattern) "[...]") + (let ([len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([open (string-find-substring-from source "[" start)]) + (if (not open) + (nonempty-findings (reverse acc)) + (let* ([close (and (python-list-literal-open? source open) + (find-matching-close-bracket + source + open))] + [finding + (and close + (finding-for-range-with-bindings + rule + path + source + open + close + initial-bindings))] + [next (if close + (max (+ open 1) close) + (+ open 1))]) + (loop next + (if finding (cons finding acc) acc)))))))))) + +(def (python-block-end source header-start include-if-chain?) + (let* ([header-indent (line-indent-at-offset source header-start)] + [header-end (line-end-after source header-start)] + [len (string-length source)]) + (let loop ([line-start (if (< header-end len) + (+ header-end 1) + (+ len 1))] + [last-end header-end]) + (if (> line-start len) + last-end + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [blank? (= first line-end)] + [trimmed (and (not blank?) + (substring source first line-end))] + [comment? (and trimmed + (sg-string-prefix? "#" trimmed))] + [indent (- first line-start)] + [chain-line? + (and include-if-chain? + (= indent header-indent) + trimmed + (or (sg-string-prefix? "elif " trimmed) + (sg-string-prefix? "else:" trimmed)))] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (cond + [(or blank? comment?) (loop next last-end)] + [(or (> indent header-indent) chain-line?) + (loop next line-end)] + [else last-end])))))) + +(def (python-if-ellipsis-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (and (sg-string-prefix? "if (...):" trimmed) + (string-find-substring trimmed "...")))) + +(def (scan-python-if-ellipsis-pattern + rule + path + source + pattern + initial-bindings) + (and (python-if-ellipsis-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [end (and (sg-string-prefix? "if " line) + (sg-string-suffix? ":" line) + (python-block-end source first #t))] + [finding + (and end + (finding-for-range-with-bindings + rule + path + source + first + end + initial-bindings))] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (loop next + (if finding (cons finding acc) acc)))))))) + +(def (python-deep-call-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)] + [open (char-index-from trimmed #\( 0 len)] + [deep-start (and open + (string-find-substring-from + trimmed + "<..." + (+ open 1)))] + [deep-end (and deep-start + (string-find-substring-from + trimmed + "...>" + (+ deep-start 4)))] + [tail-start (and deep-end (+ deep-end 4))] + [name (and open (string-trim (substring trimmed 0 open)))] + [needle (and deep-start + deep-end + (string-trim + (substring trimmed (+ deep-start 4) deep-end)))] + [tail (and tail-start (string-trim (substring trimmed tail-start len)))]) + (and name + needle + (> (string-length name) 0) + (> (string-length needle) 0) + (string=? tail ")") + (list (cons 'function name) + (cons 'needle needle))))) + +(def (scan-python-deep-call-pattern + rule + path + source + pattern + initial-bindings) + (let ([spec (python-deep-call-pattern-spec pattern)]) + (and spec + (let* ([function (alist-ref/default spec 'function "")] + [needle (alist-ref/default spec 'needle "")] + [fn-len (string-length function)] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([index (string-find-substring-from + source + function + start)]) + (if (not index) + (nonempty-findings (reverse acc)) + (let* ([name-end (+ index fn-len)] + [open (and (token-start-boundary? source index) + (token-end-boundary? source name-end) + (skip-horizontal-forward + source + name-end))] + [close (and open + (< open len) + (char=? (string-ref source open) #\() + (find-matching-close-paren + source + open))] + [finding + (and close + (string-find-substring + (substring source (+ open 1) (- close 1)) + needle) + (finding-for-range-with-bindings + rule + path + source + index + close + initial-bindings))] + [next (if close + (max (+ index 1) close) + (+ index 1))]) + (loop next + (if finding (cons finding acc) acc))))))))))) + +(def (python-dict-default-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (and (sg-string-prefix? "def " trimmed) + (string-find-substring trimmed ": Dict[...]") + (string-find-substring trimmed "= {}")))) + +(def (python-def-dict-default-line? line) + (and (sg-string-prefix? "def " line) + (string-find-substring line ": Dict[") + (string-find-substring line "= {}") + (sg-string-suffix? ":" line))) + +(def (scan-python-dict-default-pattern + rule + path + source + pattern + initial-bindings) + (and (python-dict-default-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [end (and (python-def-dict-default-line? line) + (python-block-end source first #f))] + [finding + (and end + (finding-for-range-with-bindings + rule + path + source + first + end + initial-bindings))] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (loop next + (if finding (cons finding acc) acc)))))))) + +(def (scan-python-pattern-fallbacks + rule + path + source + pattern + initial-bindings) + (or (scan-python-deep-call-pattern + rule + path + source + pattern + initial-bindings) + (scan-python-dict-default-pattern + rule + path + source + pattern + initial-bindings) + (scan-python-if-ellipsis-pattern + rule + path + source + pattern + initial-bindings) + (scan-python-return-string-ellipsis-pattern + rule + path + source + pattern + initial-bindings) + (scan-python-list-ellipsis-pattern + rule + path + source + pattern + initial-bindings)))