Cover Python import pattern fixtures
ober
3d6be1b03e12a3e1ae29e062a2d95419f5fece77
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,19 +1,18 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 16:16 MDT +Date: 2026-05-29 16:50 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`db6d0af Cover Python constant propagation fixtures` +`fbb1a28 Cover Python f-string pattern fixtures` Previous implementation checkpoint: -`db6d0af Cover Python constant propagation fixtures` +`fbb1a28 Cover Python f-string pattern fixtures` This checkpoint adds another Python pattern-fixture fallback slice covering -simple f-string ellipsis matching, bare-metavariable content matches inside -simple f-strings, adjacent/concatenated string assignment and call equivalence, -and the PEP 614 decorator-assignment grammar fixture. The curated pattern -oracle expands from 66 to 72 exact cases. +Python import-statement equivalence, imported call alias equivalence, and +top-level wildcard-import qualified-name shortening. The curated pattern +oracle expands from 72 to 79 exact cases. The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -74,7 +73,7 @@ make test Result: ```text -280 tests, 280 passed, 0 failed +281 tests, 281 passed, 0 failed ``` Local oracle: @@ -98,7 +97,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 72 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 72 compared +patterns-sweep: 79 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 79 compared ``` Focused AC/associative upstream pattern-fixture oracle: @@ -173,6 +172,30 @@ Result: patterns-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 6 compared ``` +Focused import/name upstream pattern-fixture oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(equivalence_naming_import|import_negatives|import_negatives2|imports|multi_import|multi_qualified_wildcard|wildcard_qualified)$' LIST_MISMATCHES=1 MAX_DIFFS=3000 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 7 compared +``` + +Focused scoped wildcard regression guard: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(scoped_wildcard|multi_qualified_wildcard|wildcard_qualified)$' LIST_MISMATCHES=1 MAX_DIFFS=2400 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 3 compared +``` + Exploratory full Python pattern-fixture oracle: ```sh @@ -182,7 +205,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATC Result: ```text -patterns-sweep: 120 passed, 44 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +patterns-sweep: 127 passed, 37 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` First-window same-basename upstream guardrail: @@ -526,35 +549,33 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint This checkpoint expands exact upstream `tests/patterns/python` coverage from -sixty-six to seventy-two fixtures. The `patterns-oracle` Make target now uses -the seventy-two-case curated `PATTERN_CASE_REGEX` by default. Running all +seventy-two to seventy-nine fixtures. The `patterns-oracle` Make target now +uses the seventy-nine-case curated `PATTERN_CASE_REGEX` by default. Running all Python `.sgrep` fixtures is still exploratory, not a required green target: the -latest full Python pattern sweep is 120 passed and 44 mismatched out of 164 +latest full Python pattern sweep is 127 passed and 37 mismatched out of 164 compared. -The scanner now has Python fallback handling for six additional upstream -f-string/interpolated/decorator fixture shapes: - -- `f"..."` matches simple f-string literals, including the whole f-string range. -- For the `dots_fstring` fixture, `f"..."` also emits the interpolation range - such as `{match}`. The `dots_fstring_with_match_stmt` fixture has a - fixture-shaped guard that suppresses interpolation subranges when the target - file contains top-level `match status:`, matching packaged Semgrep's current - behavior for that exact fixture. -- Bare metavariable pattern `$X` now emits a content-only range for simple - f-strings without interpolation, which covers `misc_metavar_vs_fstring`. -- `$X = "..."` matches single-line assignments whose right side evaluates to a - string through adjacent string literals or string concatenation. -- `$FUNC("...")` matches one-argument calls whose argument evaluates to a - string through adjacent string literals or string concatenation and binds the - called function name. -- `@why := $EXP` matches the PEP 614 decorator-assignment fixture lines and - binds the expression after `:=`. - -Smoke coverage now includes representative whole/subrange f-string matching, -the match-statement f-string guard, adjacent/concatenated string assignments, -adjacent/concatenated string call arguments, simple f-string content matching -for `$X`, and decorator-assignment lines. +The scanner now has Python fallback handling for seven additional upstream +import/name fixture shapes: + +- Import statement equivalence between `import foo.bar`, + `from foo import bar`, aliased imports, longer child imports, grouped imports, + and wildcard imports from the requested module. +- Multi-import containment for patterns such as `from z import x, y`, where + extra imported names are allowed in the same statement. +- Semgrep-compatible import finding ranges for comma-separated `import` + statements, trailing spaces, and parenthesized `from ... import (...)` forms. +- Imported call alias equivalence for patterns such as `subprocess.open(...)` + matching `sub_open(...)` and `sub.open(...)` after relevant imports. +- Top-level wildcard import qualified-name shortening, so `A.x` matches `x` + when `from A import *` is top-level, and `A.B.C.D.x` matches `C.D.x` when + `from A.B import *` is top-level. +- Scoped wildcard imports remain intentionally ignored for shortening; this is + covered by the focused `scoped_wildcard` regression guard. + +Smoke coverage now includes imported call aliases, import-statement +equivalence, multi-import containment, import range conventions, wildcard +qualified-name shortening, and the scoped wildcard guard. The exact Python upstream pattern cases covered now are: @@ -612,7 +633,11 @@ dots_stmts equivalence_constant_propagation equivalence_interpolated_str equivalence_interpolated_str2 +equivalence_naming_import import_metavar +import_negatives +import_negatives2 +imports index_tuple less_attributes less_typehint @@ -625,12 +650,15 @@ metavar_stmt metavar_typed misc_dots_stmts misc_metavar_vs_fstring +multi_import +multi_qualified_wildcard pip614-extended-decorator-grammer1 standalone_decorator standalone_decorator_single_arg static_method tuple_subscript_ellipsis tuple_subscript_ellipsis_with_match_stmt +wildcard_qualified ``` The main remaining Python pattern mismatch families from the exploratory sweep @@ -642,9 +670,6 @@ path-sensitive/range constant propagation: f-string expression/dataflow equivalence: equivalence_f_string, equivalence_f_string_2, equivalence_f_string_3, equivalence_f_string_4 -import/name equivalence and wildcard-qualified names: - equivalence_naming_import, import_negatives, import_negatives2, imports, - multi_import, multi_qualified_wildcard, wildcard_qualified metavariable class/function/dict/parameter patterns: metavar_class_def, metavar_dict, metavar_equality_param_vs_use, metavar_func_def, unordered_metavar2 @@ -665,8 +690,9 @@ literal/equivalence edge cases: Recommended next work: - Keep growing `PATTERN_CASE_REGEX` only with exact normalized matches. -- The next highest-leverage Python pattern clusters are import/name equivalence - and the remaining path-sensitive constant propagation cases. +- The next highest-leverage Python pattern clusters are the remaining + path-sensitive constant propagation cases and f-string expression/dataflow + equivalence. - A narrower alternative is the remaining f-string expression/dataflow set (`equivalence_f_string*`). The simple f-string/interpolated-string fixture slice is green now; do not re-open it unless a broader implementation can @@ -674,6 +700,34 @@ Recommended next work: - After the Python pattern frontier is stable, enable and verify the existing `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. +## Implementation Notes: Python Import/Name Slice + +Implementation is in `src/semgrep/scan.ss` in the Python fallback section: + +- `scan-python-import-equivalence-pattern` handles single-line non-metavariable + import patterns and matches equivalent absolute import statements in source. + It intentionally ignores relative imports and multi-line pattern bodies. +- `python-import-line-match-end` reproduces the current Semgrep import range + conventions needed by the fixtures: comma-separated `import` statements end + at the last matching imported item, trailing horizontal whitespace is + trimmed, and a closing parenthesis in grouped `from ... import (...)` lines + is excluded from the range. +- `scan-python-import-equivalent-call-pattern` handles dotted call patterns + with `(...)` and expands imported aliases through `python-import-map`. This + covers `from subprocess import open as sub_open` and `import subprocess as + sub` for `subprocess.open(...)`. +- `scan-python-star-import-qualified-pattern` handles qualified-name shortening + from top-level wildcard imports only. It scans both the original dotted + pattern and shortened tails, then sorts findings by source offset. +- `python-star-import-modules` deliberately requires the `from ... import *` + line to begin at column 1. This preserves `scoped_wildcard`, where an + indented wildcard import must not make outer `x` or inner `x` match `A.x`. + +These are still fixture-parity fallbacks rather than a complete import resolver. +The code does not model Python package execution semantics or scoped import +visibility beyond the top-level wildcard case needed by current Semgrep +fixtures. + ## Implementation Notes: Python F-String/Interpolated Slice Implementation is in `src/semgrep/scan.ss` in the Python fallback section: --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_interpolated_str|equivalence_interpolated_str2|import_metavar|index_tuple|less_attributes|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_equality_var|metavar_stmt|metavar_typed|misc_dots_stmts|misc_metavar_vs_fstring|pip614-extended-decorator-grammer1|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt)$$ +PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_equality_var|metavar_stmt|metavar_typed|misc_dots_stmts|misc_metavar_vs_fstring|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|wildcard_qualified)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -7755,6 +7755,367 @@ (+ line-end 1) (+ len 1))]) (loop next (if finding (cons finding acc) acc)))))))) + (def (python-import-clean-item item) + (let ([without-delimiters (let ([len (string-length item)]) + (let loop ([i 0] [acc '()]) + (cond + [(= i len) + (list->string (reverse acc))] + [(let ([ch (string-ref item i)]) + (or (char=? ch #\() + (char=? ch #\)))) + (loop (+ i 1) acc)] + [else + (loop + (+ i 1) + (cons + (string-ref item i) + acc))])))]) + (string-trim without-delimiters))) + (def (python-import-entry kind fqn) (cons kind fqn)) + (def (python-import-entry-kind entry) (car entry)) + (def (python-import-entry-fqn entry) (cdr entry)) + (def (python-import-items->entries items module) + (let loop ([xs (split-on-char items #\,)] [acc '()]) + (if (null? xs) + (reverse acc) + (let* ([clean (python-import-clean-item (car xs))]) + (let-values ([(imported alias) (split-alias clean)]) + (cond + [(string=? imported "") (loop (cdr xs) acc)] + [(string=? imported "*") + (loop + (cdr xs) + (if module + (cons (python-import-entry 'star module) acc) + acc))] + [else + (loop + (cdr xs) + (cons + (python-import-entry + 'normal + (if module + (string-append module "." imported) + imported)) + acc))])))))) + (def (python-import-line-without-comment line) + (let ([comment (first-index-of line #\#)]) + (if comment (substring line 0 comment) line))) + (def (python-absolute-module-name? module) + (and (> (string-length module) 0) + (not (char=? (string-ref module 0) #\.)))) + (def (python-import-line-entries line) + (let* ([trimmed (string-trim + (python-import-line-without-comment line))] + [len (string-length trimmed)]) + (cond + [(sg-string-prefix? "import " trimmed) + (python-import-items->entries (substring trimmed 7 len) #f)] + [(sg-string-prefix? "from " trimmed) + (let ([import-index (string-find-substring + trimmed + " import ")]) + (if import-index + (let ([module (string-trim + (substring trimmed 5 import-index))]) + (if (python-absolute-module-name? module) + (python-import-items->entries + (substring trimmed (+ import-index 8) len) + module) + '())) + '()))] + [else '()]))) + (def (python-import-pattern-requirements pattern) + (let ([trimmed (string-trim pattern)]) + (and (not (string-find-substring trimmed "\n")) + (not (string-find-substring trimmed "$")) + (not (string-find-substring trimmed "...")) + (let ([entries (python-import-line-entries trimmed)]) + (and (not (null? entries)) + (map python-import-entry-fqn entries)))))) + (def (python-import-entry-satisfies? entry required) + (let ([fqn (python-import-entry-fqn entry)]) + (if (eq? (python-import-entry-kind entry) 'star) + (string=? fqn required) + (or (string=? fqn required) + (sg-string-prefix? (string-append required ".") fqn))))) + (def (python-import-line-satisfies? entries requirements) + (all? + (lambda (required) + (any? + (lambda (entry) + (python-import-entry-satisfies? entry required)) + entries)) + requirements)) + (def (python-import-trimmed-end source start end) + (let ([last (skip-horizontal-backward source (- end 1))]) + (cond + [(< last start) start] + [(char=? (string-ref source last) #\)) last] + [else (+ last 1)]))) + (def (python-import-item-matches-requirement? + imported + requirements) + (let ([entry (python-import-entry 'normal imported)]) + (any? + (lambda (required) + (python-import-entry-satisfies? entry required)) + requirements))) + (def (python-import-line-matching-import-end + line + requirements) + (let ([len (string-length line)]) + (let loop ([start 7] [i 7] [best #f]) + (cond + [(> start len) best] + [(or (= i len) (char=? (string-ref line i) #\,)) + (let* ([item-end (python-import-trimmed-end line start i)] + [item (substring line start item-end)]) + (let-values ([(imported alias) + (split-alias + (python-import-clean-item item))]) + (loop + (+ i 1) + (+ i 1) + (if (and (not (string=? imported "")) + (python-import-item-matches-requirement? + imported + requirements)) + item-end + best))))] + [else (loop start (+ i 1) best)])))) + (def (python-import-line-match-end + source + first + line-end + requirements) + (let* ([line (substring source first line-end)] + [trimmed-line (string-trim line)]) + (if (sg-string-prefix? "import " trimmed-line) + (let ([relative-end (python-import-line-matching-import-end + trimmed-line + requirements)]) + (if relative-end + (+ first relative-end) + (python-import-trimmed-end source first line-end))) + (python-import-trimmed-end source first line-end)))) + (def (scan-python-import-equivalence-pattern rule path source pattern initial-bindings) + (let ([requirements (python-import-pattern-requirements + pattern)]) + (and requirements + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [entries (python-import-line-entries line)] + [finding (and (not (null? entries)) + (python-import-line-satisfies? + entries + requirements) + (finding-for-range-with-bindings rule path source first + (python-import-line-match-end + source + first + line-end + requirements) + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) + (def (python-dotted-call-ellipsis-callee pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)] + [suffix "(...)"]) + (and (> len (string-length suffix)) + (sg-string-suffix? suffix trimmed) + (let ([callee (substring + trimmed + 0 + (- len (string-length suffix)))]) + (and (dotted-name-pattern? callee) + (dotted-name? callee) + callee))))) + (def (python-import-local-callee-candidates source callee) + (unique-string-list + (let import-loop ([imports (python-import-map source)] + [acc (list callee)]) + (if (null? imports) + acc + (let* ([entry (car imports)] + [local (car entry)] + [fqns (cdr entry)]) + (import-loop + (cdr imports) + (let fqn-loop ([xs fqns] [inner acc]) + (if (null? xs) + inner + (let* ([fqn (car xs)] + [candidate (cond + [(and (string=? fqn callee) + (not (string=? + local + (first-dotted-segment + fqn)))) + local] + [(sg-string-prefix? + (string-append fqn ".") + callee) + (string-append + local + (substring + callee + (string-length fqn) + (string-length callee)))] + [else #f])]) + (fqn-loop + (cdr xs) + (if candidate + (cons candidate inner) + inner))))))))))) + (def (dotted-token-start-boundary? source index) + (or (= index 0) + (let ([ch (string-ref source (- index 1))]) + (and (not (identifier-token-char? ch)) + (not (char=? ch #\.)))))) + (def (dotted-token-end-boundary? source index) + (or (>= index (string-length source)) + (let ([ch (string-ref source index)]) + (and (not (identifier-token-char? ch)) + (not (char=? ch #\.)))))) + (def (scan-python-callee-occurrences rule path source callee + initial-bindings) + (let ([len (string-length source)] + [callee-len (string-length callee)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from + source + callee + start)]) + (if (not index) + (reverse acc) + (let* ([after-callee (+ index callee-len)] + [open (skip-horizontal-forward + source + after-callee)] + [close (and (< open len) + (char=? (string-ref source open) #\() + (find-matching-close-paren + source + open))] + [finding (and close + (dotted-token-start-boundary? + source + index) + (finding-for-range-with-bindings rule path source index close + initial-bindings))] + [next (max (+ index 1) + (if close close after-callee))]) + (loop next (if finding (cons finding acc) acc)))))))) + (def (insert-finding-by-start finding sorted) + (cond + [(null? sorted) (list finding)] + [(< (finding-start-offset finding) + (finding-start-offset (car sorted))) + (cons finding sorted)] + [else + (cons + (car sorted) + (insert-finding-by-start finding (cdr sorted)))])) + (def (sort-findings-by-start findings) + (let loop ([xs findings] [acc '()]) + (if (null? xs) + acc + (loop (cdr xs) (insert-finding-by-start (car xs) acc))))) + (def (scan-python-import-equivalent-call-pattern rule path source pattern initial-bindings) + (let ([callee (python-dotted-call-ellipsis-callee pattern)]) + (and callee + (let ([findings (apply + append + (map (lambda (candidate) + (scan-python-callee-occurrences rule path source candidate + initial-bindings)) + (python-import-local-callee-candidates + source + callee)))]) + (nonempty-findings + (sort-findings-by-start (dedupe-findings findings))))))) + (def (python-star-import-modules source) + (regex-fold-matches + "(^|\\n)from[ \\t]+([A-Za-z_][A-Za-z0-9_\\.]*)[ \\t]+import[ \\t]+\\*" + source + (lambda (match acc) + (let ([module (re-match-group match 2)]) + (if (member module acc) acc (cons module acc)))) + '())) + (def (python-star-import-tail-candidates source pattern) + (let ([trimmed (string-trim pattern)]) + (and (dotted-name-pattern? trimmed) + (let loop ([modules (python-star-import-modules source)] + [acc '()]) + (if (null? modules) + (unique-string-list (reverse acc)) + (let* ([module (car modules)] + [prefix (string-append module ".")] + [tail (and (sg-string-prefix? prefix trimmed) + (substring + trimmed + (string-length prefix) + (string-length trimmed)))]) + (loop + (cdr modules) + (if (and tail + (> (string-length tail) 0) + (dotted-name-pattern? tail)) + (cons tail acc) + acc)))))))) + (def (scan-python-dotted-token-occurrences rule path source + token initial-bindings) + (let ([token-len (string-length token)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from + source + token + start)]) + (if (not index) + (reverse acc) + (let* ([end (+ index token-len)] + [finding (and (dotted-token-start-boundary? + source + index) + (dotted-token-end-boundary? + source + end) + (finding-for-range-with-bindings rule path source index end + initial-bindings))]) + (loop + (max (+ index 1) end) + (if finding (cons finding acc) acc)))))))) + (def (scan-python-star-import-qualified-pattern rule path source pattern initial-bindings) + (let ([tails (python-star-import-tail-candidates + source + pattern)]) + (and tails + (not (null? tails)) + (let* ([tokens (unique-string-list + (cons (string-trim pattern) tails))] + [findings (apply + append + (map (lambda (token) + (scan-python-dotted-token-occurrences rule path source token + initial-bindings)) + tokens))]) + (nonempty-findings + (sort-findings-by-start (dedupe-findings findings))))))) (def (scan-python-pattern-fallbacks rule path source pattern initial-bindings) (or (scan-python-deep-call-pattern rule path source pattern @@ -7791,6 +8152,10 @@ (scan-python-typed-call-pattern rule path source pattern initial-bindings) (scan-python-annotated-fstring-sequence-pattern rule path source pattern initial-bindings) + (scan-python-import-equivalence-pattern rule path source + pattern initial-bindings) + (scan-python-import-equivalent-call-pattern rule path source pattern initial-bindings) + (scan-python-star-import-qualified-pattern rule path source pattern initial-bindings) (scan-python-cp-string-call-pattern rule path source pattern initial-bindings) (scan-python-cp-return-string-mvar-pattern rule path source pattern initial-bindings) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "43BFA7FE5147E40E") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/scan.ss" . "1D5A0FC144209DAE") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -7798,6 +7798,397 @@ (loop next (if finding (cons finding acc) acc)))))))) +(def (python-import-clean-item item) + (let ([without-delimiters + (let ([len (string-length item)]) + (let loop ([i 0] [acc '()]) + (cond + [(= i len) (list->string (reverse acc))] + [(let ([ch (string-ref item i)]) + (or (char=? ch #\() + (char=? ch #\)))) + (loop (+ i 1) acc)] + [else (loop (+ i 1) + (cons (string-ref item i) acc))])))]) + (string-trim without-delimiters))) + +(def (python-import-entry kind fqn) + (cons kind fqn)) + +(def (python-import-entry-kind entry) + (car entry)) + +(def (python-import-entry-fqn entry) + (cdr entry)) + +(def (python-import-items->entries items module) + (let loop ([xs (split-on-char items #\,)] [acc '()]) + (if (null? xs) + (reverse acc) + (let* ([clean (python-import-clean-item (car xs))]) + (let-values ([(imported alias) (split-alias clean)]) + (cond + [(string=? imported "") + (loop (cdr xs) acc)] + [(string=? imported "*") + (loop (cdr xs) + (if module + (cons (python-import-entry 'star module) acc) + acc))] + [else + (loop (cdr xs) + (cons (python-import-entry + 'normal + (if module + (string-append module "." imported) + imported)) + acc))])))))) + +(def (python-import-line-without-comment line) + (let ([comment (first-index-of line #\#)]) + (if comment + (substring line 0 comment) + line))) + +(def (python-absolute-module-name? module) + (and (> (string-length module) 0) + (not (char=? (string-ref module 0) #\.)))) + +(def (python-import-line-entries line) + (let* ([trimmed (string-trim (python-import-line-without-comment line))] + [len (string-length trimmed)]) + (cond + [(sg-string-prefix? "import " trimmed) + (python-import-items->entries + (substring trimmed 7 len) + #f)] + [(sg-string-prefix? "from " trimmed) + (let ([import-index (string-find-substring trimmed " import ")]) + (if import-index + (let ([module (string-trim (substring trimmed 5 import-index))]) + (if (python-absolute-module-name? module) + (python-import-items->entries + (substring trimmed (+ import-index 8) len) + module) + '())) + '()))] + [else '()]))) + +(def (python-import-pattern-requirements pattern) + (let ([trimmed (string-trim pattern)]) + (and (not (string-find-substring trimmed "\n")) + (not (string-find-substring trimmed "$")) + (not (string-find-substring trimmed "...")) + (let ([entries (python-import-line-entries trimmed)]) + (and (not (null? entries)) + (map python-import-entry-fqn entries)))))) + +(def (python-import-entry-satisfies? entry required) + (let ([fqn (python-import-entry-fqn entry)]) + (if (eq? (python-import-entry-kind entry) 'star) + (string=? fqn required) + (or (string=? fqn required) + (sg-string-prefix? (string-append required ".") fqn))))) + +(def (python-import-line-satisfies? entries requirements) + (all? (lambda (required) + (any? (lambda (entry) + (python-import-entry-satisfies? entry required)) + entries)) + requirements)) + +(def (python-import-trimmed-end source start end) + (let ([last (skip-horizontal-backward source (- end 1))]) + (cond + [(< last start) start] + [(char=? (string-ref source last) #\)) last] + [else (+ last 1)]))) + +(def (python-import-item-matches-requirement? imported requirements) + (let ([entry (python-import-entry 'normal imported)]) + (any? (lambda (required) + (python-import-entry-satisfies? entry required)) + requirements))) + +(def (python-import-line-matching-import-end line requirements) + (let ([len (string-length line)]) + (let loop ([start 7] [i 7] [best #f]) + (cond + [(> start len) best] + [(or (= i len) (char=? (string-ref line i) #\,)) + (let* ([item-end (python-import-trimmed-end line start i)] + [item (substring line start item-end)]) + (let-values ([(imported alias) + (split-alias (python-import-clean-item item))]) + (loop (+ i 1) + (+ i 1) + (if (and (not (string=? imported "")) + (python-import-item-matches-requirement? + imported + requirements)) + item-end + best))))] + [else (loop start (+ i 1) best)])))) + +(def (python-import-line-match-end source first line-end requirements) + (let* ([line (substring source first line-end)] + [trimmed-line (string-trim line)]) + (if (sg-string-prefix? "import " trimmed-line) + (let ([relative-end + (python-import-line-matching-import-end + trimmed-line + requirements)]) + (if relative-end + (+ first relative-end) + (python-import-trimmed-end source first line-end))) + (python-import-trimmed-end source first line-end)))) + +(def (scan-python-import-equivalence-pattern + rule + path + source + pattern + initial-bindings) + (let ([requirements (python-import-pattern-requirements pattern)]) + (and requirements + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [entries (python-import-line-entries line)] + [finding + (and (not (null? entries)) + (python-import-line-satisfies? + entries + requirements) + (finding-for-range-with-bindings + rule + path + source + first + (python-import-line-match-end + source + first + line-end + requirements) + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next + (if finding (cons finding acc) acc))))))))) + +(def (python-dotted-call-ellipsis-callee pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)] + [suffix "(...)"]) + (and (> len (string-length suffix)) + (sg-string-suffix? suffix trimmed) + (let ([callee (substring trimmed 0 (- len (string-length suffix)))]) + (and (dotted-name-pattern? callee) + (dotted-name? callee) + callee))))) + +(def (python-import-local-callee-candidates source callee) + (unique-string-list + (let import-loop ([imports (python-import-map source)] + [acc (list callee)]) + (if (null? imports) + acc + (let* ([entry (car imports)] + [local (car entry)] + [fqns (cdr entry)]) + (import-loop + (cdr imports) + (let fqn-loop ([xs fqns] [inner acc]) + (if (null? xs) + inner + (let* ([fqn (car xs)] + [candidate + (cond + [(and (string=? fqn callee) + (not (string=? local + (first-dotted-segment fqn)))) + local] + [(sg-string-prefix? + (string-append fqn ".") + callee) + (string-append + local + (substring callee + (string-length fqn) + (string-length callee)))] + [else #f])]) + (fqn-loop (cdr xs) + (if candidate + (cons candidate inner) + inner))))))))))) + +(def (dotted-token-start-boundary? source index) + (or (= index 0) + (let ([ch (string-ref source (- index 1))]) + (and (not (identifier-token-char? ch)) + (not (char=? ch #\.)))))) + +(def (dotted-token-end-boundary? source index) + (or (>= index (string-length source)) + (let ([ch (string-ref source index)]) + (and (not (identifier-token-char? ch)) + (not (char=? ch #\.)))))) + +(def (scan-python-callee-occurrences + rule + path + source + callee + initial-bindings) + (let ([len (string-length source)] + [callee-len (string-length callee)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from source callee start)]) + (if (not index) + (reverse acc) + (let* ([after-callee (+ index callee-len)] + [open (skip-horizontal-forward source after-callee)] + [close (and (< open len) + (char=? (string-ref source open) #\() + (find-matching-close-paren source open))] + [finding + (and close + (dotted-token-start-boundary? source index) + (finding-for-range-with-bindings + rule + path + source + index + close + initial-bindings))] + [next (max (+ index 1) + (if close close after-callee))]) + (loop next + (if finding (cons finding acc) acc)))))))) + +(def (insert-finding-by-start finding sorted) + (cond + [(null? sorted) (list finding)] + [(< (finding-start-offset finding) + (finding-start-offset (car sorted))) + (cons finding sorted)] + [else (cons (car sorted) + (insert-finding-by-start finding (cdr sorted)))])) + +(def (sort-findings-by-start findings) + (let loop ([xs findings] [acc '()]) + (if (null? xs) + acc + (loop (cdr xs) + (insert-finding-by-start (car xs) acc))))) + +(def (scan-python-import-equivalent-call-pattern + rule + path + source + pattern + initial-bindings) + (let ([callee (python-dotted-call-ellipsis-callee pattern)]) + (and callee + (let ([findings + (apply append + (map (lambda (candidate) + (scan-python-callee-occurrences + rule + path + source + candidate + initial-bindings)) + (python-import-local-callee-candidates + source + callee)))]) + (nonempty-findings + (sort-findings-by-start (dedupe-findings findings))))))) + +(def (python-star-import-modules source) + (regex-fold-matches + "(^|\\n)from[ \\t]+([A-Za-z_][A-Za-z0-9_\\.]*)[ \\t]+import[ \\t]+\\*" + source + (lambda (match acc) + (let ([module (re-match-group match 2)]) + (if (member module acc) + acc + (cons module acc)))) + '())) + +(def (python-star-import-tail-candidates source pattern) + (let ([trimmed (string-trim pattern)]) + (and (dotted-name-pattern? trimmed) + (let loop ([modules (python-star-import-modules source)] [acc '()]) + (if (null? modules) + (unique-string-list (reverse acc)) + (let* ([module (car modules)] + [prefix (string-append module ".")] + [tail (and (sg-string-prefix? prefix trimmed) + (substring trimmed + (string-length prefix) + (string-length trimmed)))]) + (loop (cdr modules) + (if (and tail + (> (string-length tail) 0) + (dotted-name-pattern? tail)) + (cons tail acc) + acc)))))))) + +(def (scan-python-dotted-token-occurrences + rule + path + source + token + initial-bindings) + (let ([token-len (string-length token)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from source token start)]) + (if (not index) + (reverse acc) + (let* ([end (+ index token-len)] + [finding + (and (dotted-token-start-boundary? source index) + (dotted-token-end-boundary? source end) + (finding-for-range-with-bindings