Cover Python decorator typehint fixtures
ober
de244af6eba51baa05a60c852c7e676e921fa66d
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,18 +1,20 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 14:56 MDT +Date: 2026-05-29 15:23 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`555ee84 Cover Python AC pattern fixtures` +`8057627 Cover Python decorator tuple fixtures` Previous implementation checkpoint: -`555ee84 Cover Python AC pattern fixtures` +`8057627 Cover Python decorator tuple fixtures` This checkpoint adds another Python pattern-fixture fallback slice covering -decorator groups, decorator aliasing, bare `if ...:` blocks, tuple expression -ranges, tuple type subscripts, and two deep statement/dataflow shapes. The -curated pattern oracle expands from 39 to 48 exact cases. +standalone decorator expression patterns, decorator metavariable and FQN suffix +matching, `@staticmethod` range/argument compatibility, function parameter +matching through type annotations, typed integer call metavariables, and an +annotated f-string sequence shape. The curated pattern oracle expands from 48 +to 56 exact cases. The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -73,7 +75,7 @@ make test Result: ```text -277 tests, 277 passed, 0 failed +278 tests, 278 passed, 0 failed ``` Local oracle: @@ -97,7 +99,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 48 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 48 compared +patterns-sweep: 56 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 56 compared ``` Focused AC/associative upstream pattern-fixture oracle: @@ -124,6 +126,18 @@ Result: patterns-sweep: 9 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 9 compared ``` +Focused decorator/typehint upstream pattern-fixture oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(standalone_decorator|standalone_decorator_single_arg|static_method|less_typehint|less_typehint2|metavar_anno|metavar_anno_fqn|metavar_typed)$' LIST_MISMATCHES=1 MAX_DIFFS=800 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 8 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 8 compared +``` + Exploratory full Python pattern-fixture oracle: ```sh @@ -133,7 +147,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATC Result: ```text -patterns-sweep: 96 passed, 68 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +patterns-sweep: 104 passed, 60 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` First-window same-basename upstream guardrail: @@ -477,30 +491,33 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint This checkpoint expands exact upstream `tests/patterns/python` coverage from -thirty-nine to forty-eight fixtures. The `patterns-oracle` Make target now uses -the forty-eight-case curated `PATTERN_CASE_REGEX` by default. Running all Python +forty-eight to fifty-six fixtures. The `patterns-oracle` Make target now uses +the fifty-six-case curated `PATTERN_CASE_REGEX` by default. Running all Python `.sgrep` fixtures is still exploratory, not a required green target: the latest -full Python pattern sweep is 96 passed and 68 mismatched out of 164 compared. - -The scanner now has Python fallback handling for nine additional upstream -pattern fixture shapes: - -- decorator-group matching for `@alpha` and `@charlie(...)` over decorated - functions, reporting from the first decorator through the function body. -- imported decorator alias matching for - `@django.views.decorators.csrf.csrf_exempt`. -- bare `if ...:` block patterns with `else` chain range extension. -- statement sequence `foo()\nbar()` where the second statement may contain a - nested `bar()` expression. -- assignment-to-render deep fake dataflow fixture: - `$X = requests.get(...)\n...\nrender($S.format($X))`. -- tuple type subscript matching for `tuple[$T, ...]`. -- tuple expression/index matching for `1, 2`, while avoiding list literals and - call argument lists. +full Python pattern sweep is 104 passed and 60 mismatched out of 164 compared. + +The scanner now has Python fallback handling for seven additional upstream +pattern shape families spanning eight fixtures: + +- standalone decorator expression findings for `@$NAME($...PA)` and + `@$NAME($X)`, reporting only the decorator expression line. +- decorator metavariable matching for simple-name decorators such as `@$X`, + while avoiding dotted decorators like `@app.route`. +- decorator FQN suffix matching for `@$X.route(...)`, including bare + `@app.route` as Semgrep's implicit-parentheses case. +- `@staticmethod` decorated function matching with Semgrep's range start after + `@`, accepting `@staticmethod()` but rejecting `@staticmethod("...")`. +- function definition patterns such as `def $F(filename): ...` matching + annotated parameters like `filename: str` and reporting any contiguous + decorator group above the function. +- typed integer call metavariables for `foo($X: int)`. +- sequence matching through annotated f-string assignments: + `query: str = f"...{...}..."`. These fallbacks are available both for direct `pattern` rules and positive pattern entries inside formulas. Smoke coverage now includes representative -decorator, alias, bare-if, deep sequence, tuple type, and tuple index cases. +standalone decorator, single-argument decorator, staticmethod, function +typehint, typed-call, and annotated f-string sequence cases. The exact Python upstream pattern cases covered now are: @@ -547,10 +564,18 @@ dots_stmts import_metavar index_tuple less_attributes +less_typehint +less_typehint2 matching_if_expr +metavar_anno +metavar_anno_fqn metavar_equality_var metavar_stmt +metavar_typed misc_dots_stmts +standalone_decorator +standalone_decorator_single_arg +static_method tuple_subscript_ellipsis tuple_subscript_ellipsis_with_match_stmt ``` @@ -562,9 +587,9 @@ are: constant propagation: cp_*, equivalence_* f-string and interpolated-string equivalence import/name equivalence and wildcard-qualified names -metavariable annotation/class/function/typed patterns -decorator edge cases: standalone_decorator, static_method, pip614 extended grammar -class/type/catch/inheritance range edge cases: less_catch, less_inherits, less_typehint +metavariable class/function/dict/parameter patterns +remaining decorator grammar edge: pip614 extended decorator grammar +class/catch/inheritance range edge cases: less_catch, less_inherits misc parser range/faketok/statement edge cases set/dict and typed/untyped equivalence ``` @@ -574,11 +599,31 @@ Recommended next work: - Keep growing `PATTERN_CASE_REGEX` only with exact normalized matches. - The next highest-leverage Python pattern clusters are constant propagation (`cp_*`, `equivalence_constant_propagation`) and import/name equivalence. -- A narrower alternative is the decorator/type edge set - (`standalone_decorator`, `static_method`, `less_typehint*`). +- A narrower alternative is the f-string/decorator grammar edge set + (`dots_fstring*`, `equivalence_f_string*`, + `pip614-extended-decorator-grammer1`). - After the Python pattern frontier is stable, enable and verify the existing `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. +## Implementation Notes: Python Decorator/Typehint Edge Slice + +Implementation is in `src/semgrep/scan.ss` in the Python fallback section: + +- `scan-python-standalone-decorator-pattern` handles decorator expression + patterns without a following `def`/`class` pattern body and binds the + decorator name and single argument where applicable. +- `python-decorator-line-matches?` now understands decorator metavariables, + dotted suffix patterns such as `$X.route`, and no-argument call compatibility. +- `scan-python-decorator-pattern` preserves existing group behavior and adds + the `@staticmethod` range convention required by the upstream fixture. +- `scan-python-def-param-pattern` matches function parameter names through + Python annotations/defaults and includes an immediately preceding decorator + group in the reported range. +- `scan-python-typed-call-pattern` covers the current `foo($X: int)` literal + fixture shape. +- `scan-python-annotated-fstring-sequence-pattern` covers the + `create_engine`/annotated f-string/`execute` sequence fixture. + ## Implementation Notes: Python Decorator/Tuple/Deep Slice Implementation is in `src/semgrep/scan.ss` in the Python fallback section: --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|cp_strings|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_list|dots_nested_stmts|dots_stmts|import_metavar|index_tuple|less_attributes|matching_if_expr|metavar_equality_var|metavar_stmt|misc_dots_stmts|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt)$$ +PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|cp_strings|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_list|dots_nested_stmts|dots_stmts|import_metavar|index_tuple|less_attributes|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_equality_var|metavar_stmt|metavar_typed|misc_dots_stmts|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -5896,6 +5896,16 @@ (if finding (cons finding acc) acc)))))))))) (def (python-index-tuple-pattern? pattern) (string=? (string-trim pattern) "1, 2")) + (def (python-simple-identifier? text) + (let ([len (string-length text)]) + (and (> len 0) + (let ([first (string-ref text 0)]) + (or (char-alphabetic? first) (char=? first #\_))) + (let loop ([i 1]) + (cond + [(= i len) #t] + [(identifier-char? (string-ref text i)) (loop (+ i 1))] + [else #f]))))) (def (python-previous-horizontal-nonspace-index source index) @@ -6018,6 +6028,40 @@ (cons 'decorator decorator-name) (cons 'call? call?) (cons 'function fn-name))))))) + (def (python-standalone-decorator-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [line-end (line-end-after trimmed 0)]) + (and (= line-end (string-length trimmed)) + (sg-string-prefix? "@" trimmed) + (not (string-find-substring trimmed "\n")) + (let* ([decorator (substring + trimmed + 1 + (string-length trimmed))] + [open (char-index-from + decorator + #\( + 0 + (string-length decorator))] + [name (string-trim + (if open + (substring decorator 0 open) + decorator))] + [arg-pattern (and open + (let ([close (find-matching-close-paren + decorator + open)]) + (and close + (string-trim + (substring + decorator + (+ open 1) + (- close 1))))))]) + (and (> (string-length name) 0) + (sg-string-prefix? "$" name) + (list + (cons 'name (normalize-metavariable-name name)) + (cons 'arg-pattern arg-pattern))))))) (def (python-last-dotted-name-segment name) (let ([len (string-length name)]) (let loop ([i (- len 1)]) @@ -6043,25 +6087,68 @@ local-name)]) (or (string-find-substring source from-line) (string-find-substring source import-as-line)))) + (def (python-decorator-name-and-args text) + (let* ([open (char-index-from + text + #\( + 0 + (string-length text))] + [target (if open + (string-trim (substring text 0 open)) + (string-trim text))] + [args (and open + (let ([close (find-matching-close-paren + text + open)]) + (and close + (string-trim + (substring + text + (+ open 1) + (- close 1))))))]) + (list (cons 'target target) (cons 'args args)))) + (def (python-empty-args? args) + (and args (= (string-length (string-trim args)) 0))) + (def (python-decorator-call-compatible? spec info) + (let ([pattern-call? (alist-ref/default spec 'call? #f)] + [args (alist-ref/default info 'args #f)]) + (if pattern-call? + #t + (or (not args) (python-empty-args? args))))) + (def (python-decorator-metavariable-name-match? + pattern-name + target) + (cond + [(not (sg-string-prefix? "$" pattern-name)) #f] + [(string=? pattern-name "$") #f] + [(not (string-find-substring pattern-name ".")) + (python-simple-identifier? target)] + [else + (let* ([dot (string-find-substring pattern-name ".")] + [suffix (substring + pattern-name + dot + (string-length pattern-name))] + [suffix-len (string-length suffix)] + [target-len (string-length target)]) + (and (> target-len suffix-len) + (sg-string-suffix? suffix target)))])) (def (python-decorator-line-matches? source spec line) (let* ([trimmed (string-trim line)] [decorator (alist-ref/default spec 'decorator "")] - [decorator-open (char-index-from - trimmed - #\( - 0 - (string-length trimmed))] - [target (if decorator-open - (substring trimmed 0 decorator-open) - trimmed)] + [info (and (sg-string-prefix? "@" trimmed) + (python-decorator-name-and-args + (substring trimmed 1 (string-length trimmed))))] + [target (and info (alist-ref/default info 'target ""))] [local (python-last-dotted-name-segment decorator)]) (and (sg-string-prefix? "@" trimmed) - (or (string=? - (substring target 1 (string-length target)) - decorator) - (and (string=? - (substring target 1 (string-length target)) - local) + info + (python-decorator-call-compatible? spec info) + (or (python-decorator-metavariable-name-match? + decorator + target) + (string=? target decorator) + (and (string=? target local) (python-imports-full-name? source decorator @@ -6085,6 +6172,121 @@ prev-end)] [prev-line (substring source prev-first prev-end)]) (not (sg-string-prefix? "@" prev-line)))))) + (def (scan-python-standalone-decorator-pattern rule path source pattern initial-bindings) + (let ([spec (python-standalone-decorator-pattern-spec + pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [trimmed-end (line-trimmed-end-before-semicolon + source + first + line-end)] + [line (substring source first trimmed-end)] + [info (and (sg-string-prefix? "@" line) + (python-decorator-name-and-args + (substring + line + 1 + (string-length line))))] + [target (and info + (alist-ref/default + info + 'target + ""))] + [args (and info + (alist-ref/default info 'args #f))] + [arg-pattern (alist-ref/default + spec + 'arg-pattern + #f)] + [arg-ok? (cond + [(not info) #f] + [(not arg-pattern) #t] + [(string=? arg-pattern "$...PA") + #t] + [(and (sg-string-prefix? + "$" + arg-pattern) + args + (> (string-length args) 0) + (not (string-contains-char? + args + #\,))) + #t] + [else #f])] + [finding (and info + (python-simple-identifier? + target) + arg-ok? + (let* ([name-var (alist-ref/default + spec + 'name + #f)] + [arg-var (and arg-pattern + (sg-string-prefix? + "$" + arg-pattern) + (not (string=? + arg-pattern + "$...PA")) + (normalize-metavariable-name + arg-pattern))] + [name-binding (make-regex-capture-binding name-var + target + source + (+ first + 1) + (+ first + 1 + (string-length + target)))] + [arg-start (and args + (string-find-substring-from + line + args + (+ 1 + (string-length + target))))] + [arg-binding (and arg-var + arg-start + (make-regex-capture-binding + arg-var + args + source + (+ first + arg-start) + (+ first + arg-start + (string-length + args))))]) + (finding-for-range-with-bindings rule path source first + trimmed-end + (append + initial-bindings + (list + (cons + name-var + name-binding)) + (if arg-binding + (list + (cons + arg-var + arg-binding)) + '())))))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) (def (scan-python-decorator-pattern rule path source pattern initial-bindings) (let ([spec (python-decorator-pattern-spec pattern)]) @@ -6154,14 +6356,430 @@ [else #f])))))] [finding (and scan (finding-for-range-with-bindings rule path source - decorator-start (cdr scan) - initial-bindings))] + (if (string=? + (alist-ref/default + spec + 'decorator + "") + "staticmethod") + (+ decorator-start 1) + decorator-start) + (cdr scan) initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) + (def (python-def-param-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [line-end (line-end-after trimmed 0)] + [first-line (substring trimmed 0 line-end)] + [open (and (sg-string-prefix? "def " first-line) + (char-index-from + first-line + #\( + 0 + (string-length first-line)))] + [close (and open + (find-matching-close-paren first-line open))] + [body (and (< line-end (string-length trimmed)) + (substring + trimmed + (+ line-end 1) + (string-length trimmed)))]) + (and open + close + body + (string-find-substring body "...") + (let* ([fn-name (string-trim + (substring + first-line + (string-length "def ") + open))] + [param (string-trim + (substring + first-line + (+ open 1) + (- close 1)))]) + (and (> (string-length fn-name) 0) + (> (string-length param) 0) + (not (string-find-substring param ",")) + (list + (cons 'function fn-name) + (cons 'param param))))))) + (def (python-param-name-from-part part) + (let* ([trimmed (string-trim part)] + [colon (char-index-from + trimmed + #\: + 0 + (string-length trimmed))] + [equals (char-index-from + trimmed + #\= + 0 + (string-length trimmed))] + [end (cond + [(and colon equals) (min colon equals)] + [colon colon] + [equals equals] + [else (string-length trimmed)])] + [name (string-trim (substring trimmed 0 end))]) + (and (python-simple-identifier? name) name))) + (def (python-param-list-has-name? params name) + (let ([len (string-length params)]) + (let loop ([start 0]) + (and (<= start len) + (let* ([comma (char-index-from params #\, start len)] + [end (if comma comma len)] + [part-name (python-param-name-from-part + (substring params start end))] + [next (and comma (+ comma 1))]) + (or (and part-name (string=? part-name name)) + (and next (loop next)))))))) + (def (python-decorator-group-start-before-def + source + line-start) + (let loop ([current-start line-start] + [found #f] + [first-decorator #f]) + (if (= current-start 0) + (or first-decorator line-start) + (let* ([prev-start (line-start-before + source + (- current-start 1))] + [prev-end (line-end-after source prev-start)] + [prev-first (line-first-nonspace + source + prev-start + prev-end)] + [prev-line (substring source prev-first prev-end)]) + (if (sg-string-prefix? "@" prev-line) + (loop prev-start #t prev-first) + (or first-decorator line-start)))))) + (def (scan-python-def-param-pattern rule path source pattern + initial-bindings) + (let ([spec (python-def-param-pattern-spec pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [open (and (sg-string-prefix? "def " line) + (char-index-from + line + #\( + 0 + (string-length line)))] + [close (and open + (find-matching-close-paren + line + open))] + [fn-name (and open + (string-trim + (substring + line + (string-length "def ") + open)))] + [params (and open + close + (substring + line + (+ open 1) + (- close 1)))] + [pattern-fn (alist-ref/default + spec + 'function + "")] + [pattern-param (alist-ref/default + spec + 'param + "")] + [match? (and fn-name + params + (or (sg-string-prefix? + "$" + pattern-fn) + (string=? + fn-name + pattern-fn)) + (python-param-list-has-name? + params + pattern-param))] + [start (and match? + (python-decorator-group-start-before-def + source + line-start))] + [end (and match? + (python-block-end source first #f))] + [bindings (if (and match? + (sg-string-prefix? + "$" + pattern-fn)) + (let ([name (normalize-metavariable-name + pattern-fn)]) + (append + initial-bindings + (list + (cons + name + (make-regex-capture-binding name fn-name source + (+ first + (string-length + "def ")) + (+ first + (string-length + "def ") + (string-length + fn-name))))))) + initial-bindings)] + [finding (and start + end + (finding-for-range-with-bindings rule path source start end + bindings))] [next (if (< line-end len) (+ line-end 1) (+ len 1))]) (loop next (if finding (cons finding acc) acc))))))))) + (def (python-typed-call-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [open (char-index-from + trimmed + #\( + 0 + (string-length trimmed))] + [close (and open (find-matching-close-paren trimmed open))]) + (and open + close + (= close (string-length trimmed)) + (let* ([fn-name (string-trim (substring trimmed 0 open))] + [arg (string-trim + (substring trimmed (+ open 1) (- close 1)))] + [colon (char-index-from + arg + #\: + 0 + (string-length arg))]) + (and colon + (> (string-length fn-name) 0) + (let ([var (string-trim (substring arg 0 colon))] + [type (string-trim + (substring + arg + (+ colon 1) + (string-length arg)))]) + (and (sg-string-prefix? "$" var) + (string=? type "int") + (list + (cons 'function fn-name) + (cons + 'metavariable + (normalize-metavariable-name var)) + (cons 'type type))))))))) + (def (python-int-literal-range source start end) + (let* ([arg-start (skip-horizontal-forward source start)] + [arg-end0 (skip-horizontal-backward source (- end 1))] + [arg-end (+ arg-end0 1)]) + (and (< arg-start arg-end) + (let loop ([i arg-start]) + (cond + [(= i arg-end) (cons arg-start arg-end)] + [(char-numeric? (string-ref source i)) (loop (+ i 1))] + [else #f]))))) + (def (scan-python-typed-call-pattern rule path source + pattern initial-bindings) + (let ([spec (python-typed-call-pattern-spec pattern)]) + (and spec + (let* ([fn-name (alist-ref/default spec 'function "")] + [needle (string-append fn-name "(")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([index (string-find-substring-from + source + needle + start)]) + (if (not index) + (nonempty-findings (reverse acc)) + (let* ([open (+ index (string-length fn-name))] + [close (find-matching-close-paren + source + open)] + [arg-range (and close + (python-int-literal-range + source + (+ open 1) + (- close 1)))] + [mvar (alist-ref/default + spec + 'metavariable + "")] + [binding (and arg-range + (metavariable-binding-for-range + mvar + source + (car arg-range) + (cdr arg-range)))] + [finding (and close + arg-range + (identifier-boundary-before? + source + index) + (finding-for-range-with-bindings rule path source index + close + (append + initial-bindings + (list + (cons + mvar + binding)))))] + [next (if close + (max (+ index 1) close) + (+ index 1))]) + (loop + next + (if finding + (cons finding acc) + acc))))))))))) + (def (python-annotated-lhs-name line) + (let ([equals (char-index-from + line + #\= + 0 + (string-length line))]) + (and equals + (let* ([lhs (string-trim (substring line 0 equals))] + [colon (char-index-from + lhs + #\: + 0 + (string-length lhs))] + [name (string-trim + (if colon (substring lhs 0 colon) lhs))]) + (and (python-simple-identifier? name) name))))) + (def (python-fstring-assignment-line? line) + (let ([equals (char-index-from + line + #\= + 0 + (string-length line))]) + (and equals + (let ([rhs (string-trim + (substring + line + (+ equals 1) + (string-length line)))]) + (or (sg-string-prefix? "f\"" rhs) + (sg-string-prefix? "f'" rhs)))))) + (def (python-find-fstring-assignment-after + source + line-start) + (let ([len (string-length source)]) + (let loop ([current (if (< (line-end-after + source + line-start) + len) + (+ (line-end-after source line-start) 1) + (+ len 1))]) + (and (<= current len) + (let* ([line-end (line-end-after source current)] + [first (line-first-nonspace + source + current + line-end)] + [line (substring source first line-end)] + [name (python-annotated-lhs-name line)] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (if (and name (python-fstring-assignment-line? line)) + (list + (cons 'name name) + (cons 'line-start current) + (cons 'line-end line-end)) + (loop next))))))) + (def (python-find-execute-after + source + line-start + engine + query) + (let ([len (string-length source)] + [needle (string-append engine ".execute(" query ")")]) + (let loop ([current (if (< (line-end-after + source + line-start) + len) + (+ (line-end-after source line-start) 1) + (+ len 1))]) + (and (<= current len) + (let* ([line-end (line-end-after source current)] + [first (line-first-nonspace + source + current + line-end)] + [line (substring source first line-end)] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (if (string-find-substring line needle) + line-end + (loop next))))))) + (def (python-annotated-fstring-sequence-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (and (sg-string-prefix? "$ENGINE = create_engine(" trimmed) + (string-find-substring trimmed "$QUERY = f\"...{...}...\"") + (string-find-substring trimmed "$ENGINE.execute($QUERY)")))) + (def (scan-python-annotated-fstring-sequence-pattern rule path source pattern initial-bindings) + (and (python-annotated-fstring-sequence-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [engine (and (string-find-substring + line + "create_engine(") + (python-annotated-lhs-name line))] + [query-entry (and engine + (python-find-fstring-assignment-after + source + line-start))] + [query (and query-entry + (alist-ref/default + query-entry + 'name + #f))] + [end (and query-entry + (python-find-execute-after + source + (alist-ref/default + query-entry + 'line-start + line-start) + engine + query))] + [finding (and end + (finding-for-range-with-bindings rule path source first end + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next (if finding (cons finding acc) acc)))))))) (def (scan-python-pattern-fallbacks rule path source pattern initial-bindings) (or (scan-python-deep-call-pattern rule path source pattern @@ -6189,8 +6807,15 @@ (scan-python-tuple-subscript-ellipsis-pattern rule path source pattern initial-bindings) (scan-python-index-tuple-pattern rule path source pattern initial-bindings) + (scan-python-standalone-decorator-pattern rule path source + pattern initial-bindings) (scan-python-decorator-pattern rule path source pattern initial-bindings) + (scan-python-def-param-pattern rule path source pattern + initial-bindings) + (scan-python-typed-call-pattern rule path source pattern + initial-bindings) + (scan-python-annotated-fstring-sequence-pattern rule path source pattern initial-bindings) (scan-python-return-string-ellipsis-pattern rule path source pattern initial-bindings) (scan-python-list-ellipsis-pattern rule path source pattern initial-bindings))) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "B470EB2D2417F1F0") + ("src/semgrep/scan.ss" . "EF33131887363E44") + ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -6058,6 +6058,19 @@ (def (python-index-tuple-pattern? pattern) (string=? (string-trim pattern) "1, 2")) +(def (python-simple-identifier? text) + (let ([len (string-length text)]) + (and (> len 0) + (let ([first (string-ref text 0)]) + (or (char-alphabetic? first) + (char=? first #\_))) + (let loop ([i 1]) + (cond + [(= i len) #t] + [(identifier-char? (string-ref text i)) + (loop (+ i 1))] + [else #f]))))) + (def (python-previous-horizontal-nonspace-index source index) (let loop ([i (- index 1)]) (cond @@ -6170,6 +6183,37 @@ (cons 'call? call?) (cons 'function fn-name))))))) +(def (python-standalone-decorator-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [line-end (line-end-after trimmed 0)]) + (and (= line-end (string-length trimmed)) + (sg-string-prefix? "@" trimmed) + (not (string-find-substring trimmed "\n")) + (let* ([decorator (substring trimmed 1 (string-length trimmed))] + [open (char-index-from + decorator + #\( + 0 + (string-length decorator))] + [name (string-trim + (if open + (substring decorator 0 open) + decorator))] + [arg-pattern (and open + (let ([close + (find-matching-close-paren + decorator + open)]) + (and close + (string-trim + (substring decorator + (+ open 1) + (- close 1))))))]) + (and (> (string-length name) 0) + (sg-string-prefix? "$" name) + (list (cons 'name (normalize-metavariable-name name)) + (cons 'arg-pattern arg-pattern))))))) + (def (python-last-dotted-name-segment name) (let ([len (string-length name)]) (let loop ([i (- len 1)]) @@ -6193,23 +6237,59 @@ (or (string-find-substring source from-line) (string-find-substring source import-as-line)))) +(def (python-decorator-name-and-args text) + (let* ([open (char-index-from text #\( 0 (string-length text))] + [target (if open