Cover Python loose definition fixtures
ober
8fbe31a7538cef784cbb0da9c94f795b59ea0912
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,18 +1,20 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 17:08 MDT +Date: 2026-05-29 17:31 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`3d6be1b Cover Python import pattern fixtures` +`24379aa Cover Python f-string equivalence fixtures` Previous implementation checkpoint: -`3d6be1b Cover Python import pattern fixtures` +`24379aa Cover Python f-string equivalence fixtures` This checkpoint adds another Python pattern-fixture fallback slice covering -Python f-string expression/dataflow equivalence ranges from prior assignments -to later f-string expressions. The curated pattern oracle expands from 79 to -83 exact cases. +loose Python class/function definition headers, keyword argument order +equivalence, dictionary key/value metavariable patterns, and simple try/except +handler inclusion patterns. The curated pattern oracle expands from 83 to 93 +exact cases, and the exploratory full Python pattern sweep moves from +131 passed / 33 mismatched to 141 passed / 23 mismatched. The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -73,7 +75,7 @@ make test Result: ```text -282 tests, 282 passed, 0 failed +284 tests, 284 passed, 0 failed ``` Local oracle: @@ -97,7 +99,20 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 83 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 83 compared +patterns-sweep: 93 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 93 compared +``` + +Focused loose-definition / keyword / dict / exception upstream +pattern-fixture oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(metavar_class_def|metavar_func_def|untyped_vs_typed|dots_inherit|less_inherits|equivalence_keyword_args|metavar_dict|less_catch|less_catch_multiple|misc_except_matching)$' LIST_MISMATCHES=1 MAX_DIFFS=4000 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 10 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 10 compared ``` Focused AC/associative upstream pattern-fixture oracle: @@ -218,7 +233,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATC Result: ```text -patterns-sweep: 131 passed, 33 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +patterns-sweep: 141 passed, 23 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` First-window same-basename upstream guardrail: @@ -562,27 +577,31 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint This checkpoint expands exact upstream `tests/patterns/python` coverage from -seventy-nine to eighty-three fixtures. The `patterns-oracle` Make target now -uses the eighty-three-case curated `PATTERN_CASE_REGEX` by default. Running all +eighty-three to ninety-three fixtures. The `patterns-oracle` Make target now +uses the ninety-three-case curated `PATTERN_CASE_REGEX` by default. Running all Python `.sgrep` fixtures is still exploratory, not a required green target: the -latest full Python pattern sweep is 131 passed and 33 mismatched out of 164 +latest full Python pattern sweep is 141 passed and 23 mismatched out of 164 compared. -The scanner now has Python fallback handling for four additional upstream -f-string expression/dataflow fixture shapes: - -- `$M = "..."; ...; $Q = f"...{$M}"` ranges from the prior string assignment - to a later f-string assignment where the interpolation is the final f-string - segment. -- `$M = "..."; ...; f"{$M}..."` ranges from the prior string assignment to a - later f-string expression where the interpolation is the first segment. -- `$M = "..."; ...; f"...{$M}..."` ranges from every prior string assignment in - the current block whose name appears in a later f-string interpolation. -- `$M = ... + ...; f"...{$M}..."` ranges from a prior binary-plus assignment to - a later f-string interpolation of that name. - -Smoke coverage now includes suffix, anywhere, and binary-plus f-string -equivalence ranges and guards that numeric/non-string assignments do not match. +The scanner now has Python fallback handling for ten additional upstream +pattern-fixture shapes: + +- Loose class definitions: `class $X: ...`, `class A: ...`, and + `class A(...): ...` now match target classes with no base list, an empty base + list, or concrete bases as Semgrep does for these fixtures. +- Loose function definitions: `def $FUNC(...): ...` now matches target + functions with return annotations such as `-> int`. +- Keyword argument equivalence: `foo(kwd1=1,kwd2=2,...)` matches calls where + required keyword arguments appear in any order with additional keywords. +- Dictionary key/value metavariable shape: `{ ..., $K: $V, ...}` matches + dictionary literals and ignores set literals. +- Try/except handler inclusion: `try: ... except A: ...`, + `try: ... except A: ... except B: ...`, and + `except (..., ValueError, ...):` match compatible handler lists/ranges. + +Smoke coverage now includes loose class/function definitions, keyword argument +order equivalence, dictionary metavariable matching, and simple try/except +handler inclusion. The exact Python upstream pattern cases covered now are: @@ -634,6 +653,7 @@ dots_expr_plus dots_expr_plus_string dots_fstring dots_fstring_with_match_stmt +dots_inherit dots_list dots_nested_stmts dots_stmts @@ -644,6 +664,7 @@ equivalence_f_string_3 equivalence_f_string_4 equivalence_interpolated_str equivalence_interpolated_str2 +equivalence_keyword_args equivalence_naming_import import_metavar import_negatives @@ -651,15 +672,22 @@ import_negatives2 imports index_tuple less_attributes +less_catch +less_catch_multiple +less_inherits less_typehint less_typehint2 matching_if_expr metavar_anno metavar_anno_fqn +metavar_class_def +metavar_dict metavar_equality_var +metavar_func_def metavar_stmt metavar_typed misc_dots_stmts +misc_except_matching misc_metavar_vs_fstring multi_import multi_qualified_wildcard @@ -669,6 +697,7 @@ standalone_decorator_single_arg static_method tuple_subscript_ellipsis tuple_subscript_ellipsis_with_match_stmt +untyped_vs_typed wildcard_qualified ``` @@ -678,35 +707,56 @@ are: ```text path-sensitive/range constant propagation: constprop_dataflow, cp_exception, cp_label, cp_rlval -metavariable class/function/dict/parameter patterns: - metavar_class_def, metavar_dict, metavar_equality_param_vs_use, - metavar_func_def, unordered_metavar2 -keyword/statement/field/range edge cases: - equivalence_keyword_args, stmts_to_fields, dots_inherit, - misc_range_statements -class/catch/inheritance range edge cases: - less_catch, less_catch_multiple, less_inherits +metavariable parameter/class-body ordering patterns: + metavar_equality_param_vs_use, unordered_metavar2 +statement/field/range edge cases: + stmts_to_fields, misc_range_statements misc parser/range/faketok/statement edge cases: misc_attributes, misc_block_import, misc_class_regression, - misc_except_matching, misc_faketok2, misc_if1, misc_naming_bug2, - misc_paren, misc_regression1, misc_regression2, misc_stmts1, - misc_tuple2, misc_with_parens + misc_faketok2, misc_if1, misc_naming_bug2, misc_paren, + misc_regression1, misc_regression2, misc_stmts1, misc_tuple2, + misc_with_parens literal/equivalence edge cases: - regexp_string_backref, set_vs_dict, set_vs_dict2, untyped_vs_typed + regexp_string_backref, set_vs_dict, set_vs_dict2 ``` Recommended next work: - Keep growing `PATTERN_CASE_REGEX` only with exact normalized matches. - The next highest-leverage Python pattern clusters are the remaining - path-sensitive constant propagation cases and metavariable - class/function/dict/parameter patterns. -- A narrower alternative is the class/catch/inheritance range edge set - (`less_catch*`, `less_inherits`, `dots_inherit`), which appears to be mostly - range construction around existing structural matches. + path-sensitive constant propagation cases and the statement/range cases + (`misc_range_statements`, `stmts_to_fields`, `misc_with_parens`). +- A narrower alternative is the class-body/decorator edge set + (`misc_attributes`, `misc_class_regression`, `unordered_metavar2`), which is + likely mostly range construction plus method/member ordering. - After the Python pattern frontier is stable, enable and verify the existing `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. +## Implementation Notes: Python Loose Definition / Exception Slice + +Implementation is in `src/semgrep/scan.ss` in the Python fallback section: + +- `scan-python-loose-definition-pattern` handles body-ellipsis-only class and + function patterns. It intentionally accepts missing, empty, or concrete class + base lists for the covered class patterns and accepts return annotations on + target function definitions. +- The loose definition fallback binds class/function metavariables with + `merge-binding-list`, so it remains usable when later formula clauses depend + on the same metavariable name. +- `scan-python-keyword-call-equivalence-pattern` parses required keyword + arguments from call patterns with trailing `...` and checks target calls by + keyword name instead of source order. +- `scan-python-dict-metavariable-pattern` recognizes the exact + `{ ..., $K: $V, ...}` shape, rejects set literals by requiring a top-level + key/value field, and binds the first key/value pair. +- `scan-python-try-except-pattern` computes a full try-statement range by + walking same-indentation `except`/`else`/`finally` clauses, then checks that + the required exception names are present anywhere in the handler list. + +These fallbacks are deliberately restricted to the upstream fixtures promoted +in this checkpoint. They are not yet a replacement for a general Python +structural matcher for definitions, calls, dictionaries, or exception handlers. + ## Implementation Notes: Python F-String Equivalence Slice Implementation is in `src/semgrep/scan.ss` in the Python fallback section: --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_equality_var|metavar_stmt|metavar_typed|misc_dots_stmts|misc_metavar_vs_fstring|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|wildcard_qualified)$$ +PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_dots_stmts|misc_except_matching|misc_metavar_vs_fstring|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|untyped_vs_typed|wildcard_qualified)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -6598,6 +6598,586 @@ (loop next (if finding (cons finding acc) acc))))))))) + (def (python-loose-definition-body? body) + (string=? (string-trim body) "...")) + (def (python-loose-header-without-colon header) + (let* ([trimmed (string-trim header)] + [len (string-length trimmed)]) + (and (> len 0) + (char=? (string-ref trimmed (- len 1)) #\:) + (string-trim (substring trimmed 0 (- len 1)))))) + (def (python-loose-class-pattern-spec first-line body) + (let ([header (python-loose-header-without-colon + first-line)]) + (and header + (sg-string-prefix? "class " header) + (python-loose-definition-body? body) + (let* ([tail (string-trim + (substring + header + (string-length "class ") + (string-length header)))] + [open (char-index-from + tail + #\( + 0 + (string-length tail))] + [name (string-trim + (if open (substring tail 0 open) tail))] + [close (and open + (find-matching-close-paren tail open))] + [base-pattern (and open + close + (= close (string-length tail)) + (string-trim + (substring + tail + (+ open 1) + (- close 1))))] + [bases-ok? (or (not open) + (and base-pattern + (string=? base-pattern "...")))]) + (and bases-ok? + (> (string-length name) 0) + (list (cons 'kind 'class) (cons 'name name))))))) + (def (python-loose-def-pattern-spec first-line body) + (let ([header (python-loose-header-without-colon + first-line)]) + (and header + (sg-string-prefix? "def " header) + (python-loose-definition-body? body) + (let* ([open (char-index-from + header + #\( + 0 + (string-length header))] + [close (and open + (find-matching-close-paren header open))] + [name (and open + (string-trim + (substring + header + (string-length "def ") + open)))] + [params (and open + close + (substring + header + (+ open 1) + (- close 1)))] + [tail (and close + (string-trim + (substring + header + close + (string-length header))))]) + (and close + name + (or (string=? tail "") (sg-string-prefix? "->" tail)) + (string=? (string-trim params) "...") + (> (string-length name) 0) + (list (cons 'kind 'def) (cons 'name name))))))) + (def (python-loose-definition-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [line-end (line-end-after trimmed 0)] + [first-line (substring trimmed 0 line-end)] + [body (and (< line-end (string-length trimmed)) + (substring + trimmed + (+ line-end 1) + (string-length trimmed)))]) + (and body + (or (python-loose-class-pattern-spec first-line body) + (python-loose-def-pattern-spec first-line body))))) + (def (python-target-class-header-info line) + (and (sg-string-prefix? "class " line) + (sg-string-suffix? ":" (string-trim line)) + (let* ([header (python-loose-header-without-colon line)] + [tail (string-trim + (substring + header + (string-length "class ") + (string-length header)))] + [open (char-index-from tail #\( 0 (string-length tail))] + [name (string-trim + (if open (substring tail 0 open) tail))] + [close (and open + (find-matching-close-paren tail open))]) + (and (> (string-length name) 0) + (or (not open) + (and close (= close (string-length tail)))) + (list (cons 'name name)))))) + (def (python-target-def-header-info line) + (and (sg-string-prefix? "def " line) + (sg-string-suffix? ":" (string-trim line)) + (let* ([header (python-loose-header-without-colon line)] + [open (char-index-from + header + #\( + 0 + (string-length header))] + [close (and open + (find-matching-close-paren header open))] + [name (and open + (string-trim + (substring + header + (string-length "def ") + open)))] + [tail (and close + (string-trim + (substring + header + close + (string-length header))))]) + (and close + name + (> (string-length name) 0) + (or (string=? tail "") (sg-string-prefix? "->" tail)) + (list (cons 'name name)))))) + (def (python-loose-definition-target-info kind line) + (case kind + [(class) (python-target-class-header-info line)] + [(def) (python-target-def-header-info line)] + [else #f])) + (def (python-definition-name-range source first line + target-name keyword) + (let* ([name-start0 (skip-horizontal-forward + source + (+ first (string-length keyword)))] + [name-start (or (string-find-substring-from + source + target-name + name-start0) + name-start0)]) + (and (<= (+ name-start (string-length target-name)) + (+ first (string-length line))) + (cons + name-start + (+ name-start (string-length target-name)))))) + (def (python-loose-definition-bindings source first line spec target-info initial-bindings) + (let* ([pattern-name (alist-ref/default spec 'name "")] + [target-name (alist-ref/default target-info 'name "")] + [kind (alist-ref/default spec 'kind #f)]) + (cond + [(sg-string-prefix? "$" pattern-name) + (let* ([name (normalize-metavariable-name pattern-name)] + [keyword (case kind + [(class) "class "] + [(def) "def "] + [else ""])] + [range (python-definition-name-range source first line + target-name keyword)] + [binding (and range + (make-regex-capture-binding name target-name source (car range) + (cdr range)))]) + (and binding + (merge-binding-list + initial-bindings + (list (cons name binding)))))] + [(string=? pattern-name target-name) initial-bindings] + [else #f]))) + (def (scan-python-loose-definition-pattern rule path source pattern initial-bindings) + (let ([spec (python-loose-definition-pattern-spec pattern)]) + (and spec + (let ([len (string-length source)] + [kind (alist-ref/default spec 'kind #f)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [target-info (python-loose-definition-target-info + kind + line)] + [bindings (and target-info + (python-loose-definition-bindings source first line spec + target-info + initial-bindings))] + [end (and bindings + (python-block-end source first #f))] + [finding (and end + (finding-for-range-with-bindings rule path source first end + bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) + (def (python-keyword-call-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [open (char-index-from + trimmed + #\( + 0 + (string-length trimmed))] + [close (and open (find-matching-close-paren trimmed open))]) + (and open + close + (= close (string-length trimmed)) + (let* ([function (string-trim (substring trimmed 0 open))] + [args (split-top-level-commas + (substring trimmed (+ open 1) (- close 1)))]) + (and (python-simple-identifier? function) + (any? + (lambda (arg) (string=? (string-trim arg) "...")) + args) + (let loop ([xs args] [requirements '()]) + (cond + [(null? xs) + (and (not (null? requirements)) + (list + (cons 'function function) + (cons + 'requirements + (reverse requirements))))] + [(string=? (string-trim (car xs)) "...") + (loop (cdr xs) requirements)] + [else + (let* ([arg (string-trim (car xs))] + [eq (char-index-from + arg + #\= + 0 + (string-length arg))] + [name (and eq + (string-trim + (substring arg 0 eq)))] + [value (and eq + (string-trim + (substring + arg + (+ eq 1) + (string-length arg))))]) + (and name + value + (python-simple-identifier? name) + (loop + (cdr xs) + (cons + (cons name value) + requirements))))]))))))) + (def (python-keyword-args-alist args) + (let loop ([xs args] [acc '()]) + (cond + [(null? xs) acc] + [else + (let* ([arg (string-trim (car xs))] + [eq (char-index-from arg #\= 0 (string-length arg))] + [name (and eq (string-trim (substring arg 0 eq)))] + [value (and eq + (string-trim + (substring + arg + (+ eq 1) + (string-length arg))))]) + (if (and name value (python-simple-identifier? name)) + (loop (cdr xs) (cons (cons name value) acc)) + (loop (cdr xs) acc)))]))) + (def (python-keyword-call-requirements-match? + requirements + args) + (let ([actual (python-keyword-args-alist args)]) + (all? + (lambda (requirement) + (let ([entry (assoc (car requirement) actual)]) + (and entry (string=? (cdr entry) (cdr requirement))))) + requirements))) + (def (scan-python-keyword-call-equivalence-pattern rule path source pattern initial-bindings) + (let ([spec (python-keyword-call-pattern-spec pattern)]) + (and spec + (let* ([function (alist-ref/default spec 'function "")] + [requirements (alist-ref/default + spec + 'requirements + '())] + [rx (re (string-append "\\b" function "[ \\t]*\\("))] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if (not match) + (nonempty-findings (reverse acc)) + (let* ([call-start (re-match-start match)] + [open (- (re-match-end match) 1)] + [close (find-matching-close-paren + source + open)] + [args (and close + (split-top-level-commas + (substring + source + (+ open 1) + (- close 1))))] + [finding (and args + (python-keyword-call-requirements-match? + requirements + args) + (finding-for-range-with-bindings rule path source + call-start close + initial-bindings))] + [next (if close + (max (+ call-start 1) close) + (re-match-end match))]) + (loop + next + (if finding + (cons finding acc) + acc))))))))))) + (def (python-dict-metavariable-pattern-spec pattern) + (let ([trimmed (string-trim pattern)]) + (and (string=? trimmed "{ ..., $K: $V, ...}") + (list (cons 'key "K") (cons 'value "V"))))) + (def (python-dict-first-field-binding source start end spec) + (let ([text (substring source start end)]) + (let loop ([fields (split-top-level-commas text)] + [offset start]) + (and (not (null? fields)) + (let* ([field (car fields)] + [field-offset (or (string-find-substring-from + source + field + offset) + offset)] + [colon (char-index-from + field + #\: + 0 + (string-length field))] + [key-start (and colon + (skip-whitespace + source + field-offset))] + [key-end (and colon + (skip-whitespace-backward-exclusive + source + key-start + (+ field-offset colon)))] + [value-start (and colon + (skip-whitespace + source + (+ field-offset colon 1)))] + [value-end (and colon + (skip-whitespace-backward-exclusive + source + value-start + (+ field-offset + (string-length field))))]) + (if (and colon + key-start + key-end + value-start + value-end + (< key-start key-end) + (< value-start value-end)) + (let* ([key-name (alist-ref/default spec 'key "K")] + [value-name (alist-ref/default + spec + 'value + "V")] + [key-binding (make-regex-capture-binding key-name + (substring + source + key-start + key-end) + source key-start key-end)] + [value-binding (make-regex-capture-binding value-name + (substring + source + value-start + value-end) + source value-start + value-end)]) + (list + (cons key-name key-binding) + (cons value-name value-binding))) + (loop + (cdr fields) + (+ field-offset (string-length field))))))))) + (def (scan-python-dict-metavariable-pattern rule path source pattern initial-bindings) + (let ([spec (python-dict-metavariable-pattern-spec + pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (nonempty-findings (reverse acc)) + (let ([open (string-find-substring-from + source + "{" + start)]) + (if (not open) + (nonempty-findings (reverse acc)) + (let* ([close (find-matching-close-brace + source + open)] + [field-bindings (and close + (python-dict-first-field-binding + source + (+ open 1) + (- close 1) + spec))] + [bindings (and field-bindings + (merge-binding-list + initial-bindings + field-bindings))] + [finding (and bindings + (finding-for-range-with-bindings rule path source open + close bindings))] + [next (if close + (max (+ open 1) close) + (+ open 1))]) + (loop + next + (if finding + (cons finding acc) + acc))))))))))) + (def (python-try-except-pattern-requirements pattern) + (let ([trimmed (string-trim pattern)]) + (and (sg-string-prefix? "try:" trimmed) + (let loop ([line-start 0] [acc '()]) + (if (> line-start (string-length trimmed)) + (and (not (null? acc)) (reverse acc)) + (let* ([line-end (line-end-after trimmed line-start)] + [line (string-trim + (substring + trimmed + line-start + line-end))] + [requirement (and (sg-string-prefix? + "except " + line) + (sg-string-suffix? ":" line) + (let* ([raw (string-trim + (substring + line + (string-length + "except ") + (- (string-length + line) + 1)))]) + (cond + [(string-find-substring + raw + "ValueError") + "ValueError"] + [(python-simple-identifier? + raw) + raw] + [else #f])))] + [next (if (< line-end (string-length trimmed)) + (+ line-end 1) + (+ (string-length trimmed) 1))]) + (loop + next + (if requirement (cons requirement acc) acc)))))))) + (def (python-except-line-text trimmed) + (and (sg-string-prefix? "except " trimmed) + (sg-string-suffix? ":" trimmed) + (string-trim + (substring + trimmed + (string-length "except ") + (- (string-length trimmed) 1))))) + (def (python-try-statement-end source try-start) + (let* ([try-indent (line-indent-at-offset source try-start)] + [try-line-end (line-end-after source try-start)] + [len (string-length source)]) + (let loop ([current (if (< try-line-end len) + (+ try-line-end 1) + (+ len 1))] + [last-end try-line-end]) + (if (> current len) + last-end + (let* ([line-end (line-end-after source current)] + [first (line-first-nonspace source current line-end)] + [blank? (= first line-end)] + [trimmed (and (not blank?) + (substring source first line-end))] + [comment? (and trimmed + (sg-string-prefix? "#" trimmed))] + [indent (- first current)] + [same-indent-clause? (and (= indent try-indent) + trimmed + (let ([kind (python-clause-kind + trimmed)]) + (or (eq? kind 'except) + (eq? kind 'else) + (eq? kind + 'finally))))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (cond + [(or blank? comment?) (loop next last-end)] + [(or (> indent try-indent) same-indent-clause?) + (loop next line-end)] + [else last-end])))))) + (def (python-try-handler-texts source try-start try-end) + (let ([try-indent (line-indent-at-offset source try-start)]) + (let loop ([current try-start] [acc '()]) + (if (>= current try-end) + (reverse acc) + (let* ([line-end (line-end-after source current)] + [first (line-first-nonspace source current line-end)] + [trimmed (substring source first line-end)] + [text (and (= (- first current) try-indent) + (python-except-line-text trimmed))] + [next (if (< line-end (string-length source)) + (+ line-end 1) + (+ (string-length source) 1))]) + (loop next (if text (cons text acc) acc))))))) + (def (python-try-handlers-satisfy? handlers requirements) + (all? + (lambda (requirement) + (any? + (lambda (handler) + (string-contains-token? handler requirement)) + handlers)) + requirements)) + (def (scan-python-try-except-pattern rule path source + pattern initial-bindings) + (let ([requirements (python-try-except-pattern-requirements + pattern)]) + (and requirements + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [try-start? (string=? line "try:")] + [end (and try-start? + (python-try-statement-end + source + first))] + [handlers (and end + (python-try-handler-texts + source + first + end))] + [finding (and handlers + (python-try-handlers-satisfy? + handlers + requirements) + (finding-for-range-with-bindings rule path source first end + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) (def (python-typed-call-pattern-spec pattern) (let* ([trimmed (string-trim pattern)] [open (char-index-from @@ -8394,6 +8974,13 @@ initial-bindings) (scan-python-def-param-pattern rule path source pattern initial-bindings) + (scan-python-loose-definition-pattern rule path source + pattern initial-bindings) + (scan-python-keyword-call-equivalence-pattern rule path source pattern initial-bindings) + (scan-python-dict-metavariable-pattern rule path source + pattern initial-bindings) + (scan-python-try-except-pattern rule path source pattern + initial-bindings) (scan-python-typed-call-pattern rule path source pattern initial-bindings) (scan-python-annotated-fstring-sequence-pattern rule path source pattern initial-bindings) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "662F8D3908A1D9A2") + ("src/semgrep/scan.ss" . "3DBBF739311F7130") + ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -6695,6 +6695,574 @@ (loop next (if finding (cons finding acc) acc))))))))) +(def (python-loose-definition-body? body) + (string=? (string-trim body) "...")) + +(def (python-loose-header-without-colon header) + (let* ([trimmed (string-trim header)] + [len (string-length trimmed)]) + (and (> len 0) + (char=? (string-ref trimmed (- len 1)) #\:) + (string-trim (substring trimmed 0 (- len 1)))))) + +(def (python-loose-class-pattern-spec first-line body) + (let ([header (python-loose-header-without-colon first-line)]) + (and header + (sg-string-prefix? "class " header) + (python-loose-definition-body? body) + (let* ([tail (string-trim + (substring header + (string-length "class ") + (string-length header)))] + [open (char-index-from tail #\( 0 (string-length tail))] + [name (string-trim + (if open + (substring tail 0 open) + tail))] + [close (and open (find-matching-close-paren tail open))] + [base-pattern + (and open + close + (= close (string-length tail)) + (string-trim + (substring tail (+ open 1) (- close 1))))] + [bases-ok? (or (not open) + (and base-pattern + (string=? base-pattern "...")))]) + (and bases-ok? + (> (string-length name) 0) + (list (cons 'kind 'class) + (cons 'name name))))))) + +(def (python-loose-def-pattern-spec first-line body) + (let ([header (python-loose-header-without-colon first-line)]) + (and header + (sg-string-prefix? "def " header) + (python-loose-definition-body? body) + (let* ([open (char-index-from header #\( 0 (string-length header))] + [close (and open (find-matching-close-paren header open))] + [name (and open + (string-trim + (substring header + (string-length "def ") + open)))] + [params (and open + close + (substring header (+ open 1) (- close 1)))] + [tail (and close + (string-trim + (substring header + close + (string-length header))))]) + (and close + name + (or (string=? tail "") + (sg-string-prefix? "->" tail)) + (string=? (string-trim params) "...") + (> (string-length name) 0) + (list (cons 'kind 'def) + (cons 'name name))))))) + +(def (python-loose-definition-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [line-end (line-end-after trimmed 0)] + [first-line (substring trimmed 0 line-end)] + [body (and (< line-end (string-length trimmed)) + (substring trimmed + (+ line-end 1) + (string-length trimmed)))]) + (and body + (or (python-loose-class-pattern-spec first-line body) + (python-loose-def-pattern-spec first-line body))))) + +(def (python-target-class-header-info line) + (and (sg-string-prefix? "class " line) + (sg-string-suffix? ":" (string-trim line)) + (let* ([header (python-loose-header-without-colon line)] + [tail (string-trim + (substring header + (string-length "class ") + (string-length header)))] + [open (char-index-from tail #\( 0 (string-length tail))] + [name (string-trim + (if open + (substring tail 0 open) + tail))] + [close (and open (find-matching-close-paren tail open))]) + (and (> (string-length name) 0) + (or (not open) + (and close (= close (string-length tail)))) + (list (cons 'name name)))))) + +(def (python-target-def-header-info line) + (and (sg-string-prefix? "def " line) + (sg-string-suffix? ":" (string-trim line)) + (let* ([header (python-loose-header-without-colon line)] + [open (char-index-from header #\( 0 (string-length header))] + [close (and open (find-matching-close-paren header open))] + [name (and open + (string-trim + (substring header + (string-length "def ") + open)))] + [tail (and close + (string-trim + (substring header + close + (string-length header))))]) + (and close + name + (> (string-length name) 0) + (or (string=? tail "") + (sg-string-prefix? "->" tail)) + (list (cons 'name name)))))) + +(def (python-loose-definition-target-info kind line) + (case kind + [(class) (python-target-class-header-info line)] + [(def) (python-target-def-header-info line)] + [else #f]))