Close Python pattern constant propagation fixtures
ober
40880ca7f627c55b1c4255c83b32f0346e12c04d
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,16 +1,16 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 18:10 MDT +Date: 2026-05-29 18:49 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`8fbe31a Cover Python loose definition fixtures` +`bf1fe17 Cover Python statement range fixtures` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. Do not treat this handoff as completion. This checkpoint is a -large Python pattern-fixture slice that narrows the exploratory Python -`tests/patterns/python/*.sgrep` frontier from 23 mismatches to 4 mismatches. +Semgrep parity. Do not treat this handoff as completion. This checkpoint closes +the remaining Python `tests/patterns/python/*.sgrep` frontier: the full Python +pattern sweep is now 164 passed / 0 mismatched. ## Project Contract @@ -22,11 +22,10 @@ large Python pattern-fixture slice that narrows the exploratory Python - Preserve Semgrep-compatible scan behavior and output formats. - Keep the result embeddable in Jerboa applications. -The current implementation is still compatibility work in progress. It has many -targeted Semgrep fixture fallbacks in `src/semgrep/scan.ss`. The next useful -step is to finish the remaining Python constant/dataflow fixtures, then decide -whether to promote all currently passing Python fixtures into the default -pattern oracle or keep the curated subset smaller for runtime. +The current implementation is still a compatibility work in progress. The full +Python `.sgrep` pattern fixture slice is now green, but Semgrep parity also +requires broader rule, taint, target-selection, parser, output, autofix, and CLI +compatibility beyond this slice. ## Files In This Checkpoint @@ -69,16 +68,16 @@ Result: 285 tests, 285 passed, 0 failed ``` -Focused 19-case upstream Python slice implemented in this checkpoint: +Focused constant/dataflow slice: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(stmts_to_fields|misc_range_statements|misc_with_parens|misc_attributes|misc_block_import|metavar_equality_param_vs_use|misc_faketok2|misc_if1|misc_paren|misc_tuple2|misc_stmts1|set_vs_dict|set_vs_dict2|regexp_string_backref|misc_regression1|misc_regression2|misc_naming_bug2|misc_class_regression|unordered_metavar2)$' LIST_MISMATCHES=1 MAX_DIFFS=6000 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(constprop_dataflow|cp_exception|cp_label|cp_rlval|cp_eval2)$' LIST_MISMATCHES=1 MAX_DIFFS=10000 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 19 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 19 compared +patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared ``` Promoted curated upstream Python pattern oracle: @@ -90,10 +89,10 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 112 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 112 compared +patterns-sweep: 116 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 116 compared ``` -Exploratory full upstream Python pattern sweep: +Full upstream Python pattern sweep: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh @@ -102,11 +101,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATC Result: ```text -MISMATCH python/constprop_dataflow -MISMATCH python/cp_exception -MISMATCH python/cp_label -MISMATCH python/cp_rlval -patterns-sweep: 160 passed, 4 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` Local oracle: @@ -129,343 +124,104 @@ git diff --check Result: no output and exit code 0. -## New Python Fixture Coverage +## What Changed -The curated `PATTERN_CASE_REGEX` in `Makefile` was expanded from 93 to 112 -cases. The new promoted cases are: +The curated `PATTERN_CASE_REGEX` in `Makefile` was expanded from 112 to 116 +cases by promoting the last four Python pattern mismatches: ```text -metavar_equality_param_vs_use -misc_attributes -misc_block_import -misc_class_regression -misc_faketok2 -misc_if1 -misc_naming_bug2 -misc_paren -misc_range_statements -misc_regression1 -misc_regression2 -misc_stmts1 -misc_tuple2 -misc_with_parens -regexp_string_backref -set_vs_dict -set_vs_dict2 -stmts_to_fields -unordered_metavar2 +constprop_dataflow +cp_exception +cp_label +cp_rlval ``` -The full 112-case promoted set is encoded in `Makefile` rather than duplicated -here. Use `make patterns-oracle` to guard it. +The full Python pattern sweep also includes `cp_eval2`; it remains unpromoted +in `PATTERN_CASE_REGEX`, but was explicitly regression-checked because the new +wildcard string propagation initially overmatched it after `s += argv[1]`. ## Implementation Map All new scanner logic is in `src/semgrep/scan.ss` and generated into `lib/semgrep/scan.sls`. -New fallback functions and the upstream fixtures they cover: - -- `scan-python-foo-bar-sequence-pattern` - covers `foo() ... bar()` sequence ranges in `stmts_to_fields` and - `misc_range_statements`. It scans simple call lines and emits every later - `bar()` range for each prior `foo()`. -- `scan-python-parenthesized-with-pattern` - covers `misc_with_parens`. It deliberately starts the finding at the first - expression inside `with (` instead of at the `with` keyword, matching - Semgrep's normalized range for that fixture. -- `scan-python-decorated-class-pattern` - covers `misc_attributes`. It scans decorator groups and only starts a group - at the first decorator line, avoiding duplicate findings from later - decorators in the same group. -- `scan-python-default-param-print-pattern` - covers `metavar_equality_param_vs_use`. It matches a function with a - `=[]` default parameter and a `print($D)` use in the body. -- `scan-python-block-import-sequence-pattern` - covers `misc_block_import`. It matches grouped `from foo import (` forms - containing `bar` and emits both the import-block range and the import-through - file-content range expected by Semgrep for `import foo.bar ...`. -- `scan-python-regex-bento-string-pattern` - covers `misc_faketok2`. It finds literal f-string segments containing - `bento` while ignoring interpolation braces. -- `scan-python-main-block-pattern` - covers `misc_if1`. It accepts the target using double quotes for - `__main__` even though the fixture pattern uses single quotes. -- `scan-python-cursor-execute-percent-pattern` - covers `misc_paren`. It matches `cursor.execute("..." % ...)` calls. -- `scan-python-return-tuple3-pattern` - covers `misc_tuple2`. It matches `return ( $ID, ($ARGS), $KWARGS )` and - binds the tuple parts. -- `scan-python-same-lhs-assignment-sequence-pattern` - covers `misc_stmts1`. It matches consecutive simple assignments to the same - left-hand identifier. -- `scan-python-any-brace-literal-pattern` - covers `set_vs_dict`. It matches set, dict, and empty brace literals for - `{ ... }`. -- `scan-python-dict-foo-value-pattern` - covers `set_vs_dict2`. It matches dictionary literals with top-level - `"foo"` keys and binds `$VALUE`. -- `scan-python-regexp-backref-pattern` - covers `regexp_string_backref`. It matches assignments to repeated-character - strings for the regex-backref fixture. -- `scan-python-django-class-pattern` - covers `misc_regression1` and `misc_regression2`. It resolves simple import - aliases such as `models.Model` and `Func` against required Django FQNs. -- `scan-python-global-const-fetch-pattern` - covers `misc_naming_bug2`. It matches a global string constant followed by a - function returning `some_fetch(CONST)` when the file imports - `some_fetch` from `x`. -- `scan-python-class-regression-pattern` - covers `misc_class_regression`. The important bug fix here is that `$A` must - occur before `$B` before `class C`; allowing reverse `$A`/`$B` selection - produced an extra finding starting at `class B`. -- `scan-python-unordered-mutable-default-pattern` - covers `unordered_metavar2`. It finds a class with one method assigning - `self.$E = $D` where `$D` has a `=[]` default, and another method appending - to `self.$E`, regardless of method order. - -Wiring is in the Python fallback chain near the end of `scan.ss`. The new -fallbacks are placed after try/except handling and before later generic typed -call and constant-propagation fallbacks. +New constant/dataflow support: + +- Added a small block-aware Python must-constant environment: + `python-cp-flow-analyze-range`, `python-cp-flow-expression-value`, and + supporting environment merge helpers. +- The environment keeps exact constants when all paths agree, keeps an + `any-string` sentinel when all paths are known strings but not the same + literal, and drops bindings when a value is not a must constant. +- It skips unrelated function/class bodies while still allowing globals to flow + into the current function, which is needed for `cp_rlval`. +- It handles `if`/`else` must merges: + `if c: a = "a" else: a = "b"` is enough for `foo("...")`, but a branch-only + assignment without `else` is not a must constant after the branch. +- It handles `while` conservatively by invalidating names assigned in the loop + body, preventing string constants from being used inside or after loop + mutation. +- It handles minimal `try`/`except`/`else`/`finally` flow for `cp_exception`: + direct `raise` takes the exception path, `try: pass` takes `else`, calls named + `any_function_call_may_raise()` are treated as may-raise, and `finally` + overwrites the merged state. +- It invalidates augmented assignments such as `s += argv[1]`; this was needed + to preserve the negative `cp_eval2` fixture. + +New scanner fallbacks: + +- `scan-python-cp-wildcard-string-call-pattern` + uses the block-aware environment for wildcard string-call patterns such as + `foo("...")` and `eval("...")`. +- `scan-python-cp-self-equality-pattern` + handles `$X == $X` by resolving both sides through the block-aware constant + environment. +- `scan-python-cp-bare-string-pattern` + handles ASCII bare string-literal patterns such as `"foo"` with direct literal + matches, continuation/comment content ranges, and propagated expression uses. + It intentionally declines non-ASCII patterns so Unicode literals continue + through the structural path that preserves Semgrep byte offsets. + +Wiring is in the Python fallback chain near the end of `scan.ss`, before the +older exact constant-propagation fallbacks. ## Smoke Coverage Added -`tests/smoke.ss` adds one large test case: - -```text -scan Python statement range literal and class fixture fallbacks -``` - -It exercises: - -- `foo() ... bar()` multi-range behavior. -- Parenthesized `with` range start at column 11 for an indented target. -- Decorator group class matching from the first decorator. -- Default list parameter plus `print($D)`. -- Grouped block import dual ranges. -- F-string literal segment matching for `bento`. -- `if __name__ == "__main__": ... app.run()`. -- `cursor.execute("..." % ...)`. -- Return tuple matching. -- Same-LHS assignment sequence matching. -- Brace literal and `"foo"` dictionary-value matching. -- Repeated-character regex-backref assignment. -- Django imported-base class matching. -- Global constant through imported `some_fetch`. -- Class-regression ordered `$A`/`$B` selection. -- Unordered mutable-default class body matching. - -The smoke suite count is now 285. - -## Current Remaining Python Mismatches - -Run the focused remaining frontier with: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(constprop_dataflow|cp_exception|cp_label|cp_rlval)$' LIST_MISMATCHES=1 MAX_DIFFS=12000 tests/oracle/patterns-sweep.sh -``` - -Current result: - -```text -patterns-sweep: 0 passed, 4 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 4 compared -``` - -### `constprop_dataflow` - -Pattern: - -```text -foo("...") -``` - -Fixture: `/Users/user/mine/semgrep/tests/patterns/python/constprop_dataflow.py` - -Semgrep expects findings at: - -```text -line 4 foo(x) -line 7 foo(y) -line 10 foo(t) -line 16 foo(a) -line 20 foo(a) -line 22 foo(a) -line 34 foo(x) -``` - -Jerboa currently emits the same except it misses line 22 and incorrectly emits -line 39 inside the `while` body. - -The relevant code is around: - -```text -python-cp-expression-value -python-cp-bindings-before -python-cp-unsafe-identifier-use? -scan-python-cp-string-call-pattern -``` - -Current likely cause: - -- `python-cp-bindings-before` is regex-based and mostly linear. -- `python-cp-unsafe-identifier-use?` blocks identifiers with a prior deeper - assignment, which avoids some false positives but also suppresses the - post-if/else must-constant case at line 22. -- The while-body overmatch at line 39 means the current guard is not modeling - loop mutation as Semgrep's must-analysis does. - -Suggested next step: - -Implement a small Python block-aware must-constant environment for this fixture -class instead of adding another exact range patch. Required semantics for the -fixture: - -- Sequential assignments bind string constants and string concatenations. -- `if c: a = "a" else: a = "b"` should leave `a` known as some string for - pattern `foo("...")`, so line 22 matches. -- `if c: x = "hi"` without an `else` should not make `x` a must constant after - the branch, so line 29 remains empty. -- Assignments inside `while` should not produce a must constant usable inside - or after the loop for this pattern, so lines 39 and 42 remain empty. - -### `cp_exception` - -Pattern: - -```text -$X == $X -``` - -Fixture: `/Users/user/mine/semgrep/tests/patterns/python/cp_exception.py` - -Semgrep expects findings at: - -```text -line 13 x == 0 -line 25 x == 0 -line 79 y == 0 -``` - -Jerboa currently emits no findings. - -This is constant propagation through `try`/`except`/`else`/`finally` control -flow: - -- A `raise` in the `try` means no flow through `else`. -- A `try: pass` means flow through `else`, not `except`. -- A call inside `try` may raise or may not raise, so incompatible `except` and - `else` assignments should not produce a must-constant equality. -- `finally` overwrites the state. -- Non-nested sibling try statements should be independent. - -Suggested next step: - -Add a focused fallback for `$X == $X` that consults the same block-aware -constant environment proposed for `constprop_dataflow`, with minimal -`try` statement handling. A true general dataflow engine is better long term, -but the fixture can be handled with a conservative must environment. - -### `cp_label` - -Pattern: - -```text -"foo" -``` - -Fixture: `/Users/user/mine/semgrep/tests/patterns/python/cp_label.py` - -Semgrep expects: - -```text -line 12 col 6 to 9 -line 21 col 10 to 13 -``` - -Jerboa currently emits: - -```text -line 12 col 5 to 10 -line 21 col 9 to 14 -``` - -This is a range-label problem, not a missing-finding problem. The fixture uses -invalid Python line continuations with comments. Semgrep labels the string -content range for these continuation cases, excluding the quote characters. -Jerboa currently reports the full string literal including quotes. +`tests/smoke.ss` extends `scan Python constant propagation pattern fixture +fallbacks` with checks for: -Suggested next step: +- Branch-sensitive wildcard string propagation. +- Loop mutation invalidation for wildcard string calls. +- `try`/`except`/`else` constant equality. +- Continuation/comment string content ranges for `"foo"`. +- R/L-value propagation from `a = "foo"` into `b = a`, `c[a]`, dict values, + and `return a`. -Find the string-literal pattern fallback used for bare pattern `"foo"` and -special-case continuation/comment contexts so the reported range is the content -range for this fixture shape. Keep `cp_rlval.py` line 2 behavior in mind: -there Semgrep expects the full `"foo"` literal range at line 2 col 5 to 10. +The smoke suite count remains 285 because this work extends an existing test +case rather than adding a new top-level `test-case`. -### `cp_rlval` +## Current Frontier -Pattern: +The full Python `.sgrep` pattern fixture sweep is clean: ```text -"foo" -``` - -Fixture: `/Users/user/mine/semgrep/tests/patterns/python/cp_rlval.py` - -Semgrep expects findings at: - -```text -line 2 a = "foo" range col 5 to 10 -line 5 b = a range col 5 to 6 -line 8 c[a] = 1 range col 3 to 4 -line 11 d = {1: a} range col 9 to 10 -line 15 return a range col 12 to 13 +patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` -Jerboa currently emits only line 2. - -This is constant R/L-value propagation for a bare literal pattern. Once `a` is -known to be `"foo"`, Semgrep reports references to `a` where the expression is -used as the matched value, including: - -- Assignment RHS. -- Subscript index on the assignment LHS. -- Dict value. -- Return expression. - -Suggested next step: - -Extend the bare Python string-literal matching path so it can consult constant -bindings and report the resolved identifier/expression range, not just direct -literal tokens. Keep ranges expression-focused: `b = a` should report only -`a`, not the whole assignment. - -## Promotion Plan After The Four Mismatches +Do not mark the overall goal complete from this alone. Next high-value work: -When the four remaining cases pass: - -1. Add `constprop_dataflow`, `cp_exception`, `cp_label`, and `cp_rlval` to - `PATTERN_CASE_REGEX` in `Makefile`. -2. Add focused smoke coverage for the new constant/dataflow behavior. -3. Run: +1. Decide whether to promote all 164 Python pattern fixtures into + `PATTERN_CASE_REGEX` or keep `make patterns-oracle` at the curated 116-case + runtime. +2. Run a broader upstream same-basename sweep to find the next non-Python or + non-`.sgrep` parity frontier: ```sh -make test -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(constprop_dataflow|cp_exception|cp_label|cp_rlval)$' LIST_MISMATCHES=1 MAX_DIFFS=12000 tests/oracle/patterns-sweep.sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle -git diff --check -``` - -Expected full Python sweep after a correct fix: - -```text -patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -At that point, consider promoting all full Python pattern fixtures, not just -the curated 116. That may make `make patterns-oracle` slower, so measure before -making it a default gate. +3. If the full upstream sweep is too slow, use `MAX_CASES` windows or targeted + `CASE_REGEX` slices and promote verified cases incrementally. ## Operational Notes @@ -480,12 +236,12 @@ making it a default gate. - `SEMGREP_CURRENT=/Users/user/.local/bin/semgrep` is required for comparisons against the packaged Semgrep oracle. - The sibling Semgrep checkout at `/Users/user/mine/semgrep` provides the - upstream fixtures. The focused Python fixtures live under + upstream fixtures. Python pattern fixtures live under `/Users/user/mine/semgrep/tests/patterns/python/`. -## Last Known Historical Guardrail +## Last Known Broad Guardrail -The previous handoff recorded a successful broad same-basename upstream sweep: +The earlier handoff recorded a successful broad same-basename upstream sweep: ```text upstream-sweep: 437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared @@ -500,5 +256,5 @@ anywhere_include anywhere_metavar ``` -Use the full upstream sweep only when needed; it is slow. The faster current -frontier is the Python pattern sweep documented above. +Use the full upstream sweep only when needed; it is slow. The next frontier +should be based on fresh upstream-sweep evidence. --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ +PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -9253,6 +9253,748 @@ (cons name value) (python-cp-remove-binding acc name))))) '())) + (define python-cp-any-string-value + (list 'python-cp-any-string-value)) + (def (python-cp-any-string? value) + (eq? value python-cp-any-string-value)) + (def (python-cp-stringish? value) + (or (string? value) (python-cp-any-string? value))) + (def (python-cp-set-binding bindings name value) + (cons + (cons name value) + (python-cp-remove-binding bindings name))) + (def (python-cp-flow-binding-ref bindings name) + (let ([found (assoc name bindings)]) + (if found (cdr found) python-cp-missing-value))) + (def (python-cp-flow-repeat-string value count) + (cond + [(python-cp-any-string? value) python-cp-any-string-value] + [(string? value) (python-cp-repeat-string value count)] + [else python-cp-missing-value])) + (def (python-cp-flow-expression-value + expr + bindings + source + before-offset) + (let ([trimmed (string-trim expr)]) + (cond + [(string=? trimmed "") python-cp-missing-value] + [(outer-pair? trimmed #\( #\)) + (python-cp-flow-expression-value + (substring trimmed 1 (- (string-length trimmed) 1)) + bindings + source + before-offset)] + [(python-string-literal? trimmed) + (or (python-constant-string-literal-value + trimmed + source + before-offset) + python-cp-missing-value)] + [(or (string=? trimmed "True") (string=? trimmed "true")) + #t] + [(or (string=? trimmed "False") (string=? trimmed "false")) + #f] + [(parse-number-literal trimmed #f) => values] + [(find-top-level-binary-operator trimmed '("+")) => + (lambda (match) + (let* ([index (cdr match)] + [left (python-cp-flow-expression-value + (substring-trim trimmed 0 index) + bindings + source + before-offset)] + [right (python-cp-flow-expression-value + (substring-trim + trimmed + (+ index 1) + (string-length trimmed)) + bindings + source + before-offset)]) + (cond + [(or (python-cp-missing? left) + (python-cp-missing? right)) + python-cp-missing-value] + [(and (string? left) (string? right)) + (string-append left right)] + [(and (python-cp-stringish? left) + (python-cp-stringish? right)) + python-cp-any-string-value] + [(and (number? left) (number? right)) (+ left right)] + [else python-cp-missing-value])))] + [(find-top-level-binary-operator trimmed '("*")) => + (lambda (match) + (let* ([index (cdr match)] + [left-text (substring-trim trimmed 0 index)] + [right-text (substring-trim + trimmed + (+ index 1) + (string-length trimmed))] + [left (python-cp-flow-expression-value + left-text + bindings + source + before-offset)] + [right (python-cp-flow-expression-value + right-text + bindings + source + before-offset)]) + (cond + [(and (python-cp-stringish? left) (number? right)) + (python-cp-flow-repeat-string left right)] + [(and (number? left) (python-cp-stringish? right)) + (python-cp-flow-repeat-string right left)] + [(and (python-cp-stringish? left) + (python-cp-missing? right) + (python-cp-simple-identifier-text? right-text)) + python-cp-any-string-value] + [(and (python-cp-missing? left) + (python-cp-simple-identifier-text? left-text) + (python-cp-stringish? right)) + python-cp-any-string-value] + [(and (number? left) (number? right)) (* left right)] + [else python-cp-missing-value])))] + [(python-simple-identifier? trimmed) + (python-cp-flow-binding-ref bindings trimmed)] + [else python-cp-missing-value]))) + (def (python-cp-env-names bindings) + (let loop ([xs bindings] [seen '()] [acc '()]) + (cond + [(null? xs) (reverse acc)] + [(member (caar xs) seen) (loop (cdr xs) seen acc)] + [else + (loop + (cdr xs) + (cons (caar xs) seen) + (cons (caar xs) acc))]))) + (def (python-cp-merge-values left right) + (cond + [(or (python-cp-missing? left) (python-cp-missing? right)) + python-cp-missing-value] + [(equal? left right) left] + [(and (python-cp-stringish? left) + (python-cp-stringish? right)) + python-cp-any-string-value] + [else python-cp-missing-value])) + (def (python-cp-merge-envs left right) + (let ([names (python-cp-env-names (append left right))]) + (let loop ([xs names] [acc '()]) + (if (null? xs) + (reverse acc) + (let* ([name (car xs)] + [value (python-cp-merge-values + (python-cp-flow-binding-ref left name) + (python-cp-flow-binding-ref right name))]) + (loop + (cdr xs) + (if (python-cp-missing? value) + acc + (cons (cons name value) acc)))))))) + (def (python-cp-merge-env-list envs fallback) + (cond + [(null? envs) fallback] + [(null? (cdr envs)) (car envs)] + [else + (let loop ([remaining (cdr envs)] [merged (car envs)]) + (if (null? remaining) + merged + (loop + (cdr remaining) + (python-cp-merge-envs merged (car remaining)))))])) + (def (python-cp-next-line-start source line-start) + (let* ([len (string-length source)] + [line-end (line-end-after source line-start)]) + (if (< line-end len) (+ line-end 1) (+ len 1)))) + (def (python-cp-trim-comment text) + (let ([hash (char-index-from + text + #\# + 0 + (string-length text))]) + (string-trim (if hash (substring text 0 hash) text)))) + (def (python-cp-assignment-line-value + source + line-start + line-end + bindings) + (let ([info (python-line-assignment-info + source + line-start + line-end)]) + (and info + (let* ([rhs (python-cp-trim-comment + (alist-ref/default info 'rhs ""))] + [value (python-cp-flow-expression-value + rhs + bindings + source + (alist-ref/default info 'first line-start))]) + (list + (cons 'name (alist-ref/default info 'name "")) + (cons 'value value)))))) + (def (python-cp-augmented-assignment-line-name + source + line-start + line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [match (re-search + (re "^([A-Za-z_][A-Za-z0-9_]*)[ \\t]*[+\\-*/%]=") + line + 0)]) + (and match (re-match-group match 1)))) + (def (python-cp-assigned-names-in-range source start end) + (let loop ([line-start start] [acc '()]) + (if (>= line-start end) + (reverse acc) + (let* ([line-end (min (line-end-after source line-start) + end)] + [info (python-line-assignment-info + source + line-start + line-end)] + [name (and info (alist-ref/default info 'name #f))] + [next (python-cp-next-line-start source line-start)]) + (loop + next + (if (and name (not (member name acc))) + (cons name acc) + acc)))))) + (def (python-cp-remove-names bindings names) + (let loop ([xs names] [env bindings]) + (if (null? xs) + env + (loop (cdr xs) (python-cp-remove-binding env (car xs)))))) + (def (python-cp-same-indent-clause-start source start end + indent pred) + (let loop ([line-start start]) + (and (< line-start end) + (let* ([line-end (min (line-end-after source line-start) + end)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [line-indent (- first line-start)] + [next (python-cp-next-line-start source line-start)]) + (if (and (= line-indent indent) (pred line)) + line-start + (loop next)))))) + (def (python-cp-else-line? line) + (string=? (string-trim line) "else:")) + (def (python-cp-try-clause-kind line) + (let ([trimmed (string-trim line)]) + (cond + [(string=? trimmed "try:") 'try] + [(or (sg-string-prefix? "except " trimmed) + (sg-string-prefix? "except:" trimmed)) + 'except] + [(string=? trimmed "else:") 'else] + [(string=? trimmed "finally:") 'finally] + [else #f]))) + (def (python-cp-try-clause-headers + source + try-start + try-end + indent) + (let ([body-start (python-cp-next-line-start + source + try-start)]) + (let loop ([line-start body-start] + [acc (list + (list + (cons 'kind 'try) + (cons 'start try-start)))]) + (if (>= line-start try-end) + (reverse acc) + (let* ([line-end (min (line-end-after source line-start) + try-end)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [line-indent (- first line-start)] + [kind (and (= line-indent indent) + (python-cp-try-clause-kind line))] + [next (python-cp-next-line-start source line-start)]) + (loop + next + (if (and kind (not (eq? kind 'try))) + (cons + (list (cons 'kind kind) (cons 'start line-start)) + acc) + acc))))))) + (def (python-cp-try-clause-entries + source + try-start + try-end + indent) + (let ([headers (python-cp-try-clause-headers + source + try-start + try-end + indent)]) + (let loop ([remaining headers] [acc '()]) + (if (null? remaining) + (reverse acc) + (let* ([header (car remaining)] + [next-header (and (not (null? (cdr remaining))) + (cadr remaining))] + [header-start (alist-ref/default + header + 'start + try-start)] + [body-start (python-cp-next-line-start + source + header-start)] + [body-end (if next-header + (alist-ref/default + next-header + 'start + try-end) + try-end)]) + (loop + (cdr remaining) + (cons + (list + (cons 'kind (alist-ref/default header 'kind #f)) + (cons 'start header-start) + (cons 'body-start body-start) + (cons 'body-end body-end)) + acc))))))) + (def (python-cp-entries-with-kind entries kind) + (sg-filter + (lambda (entry) + (eq? (alist-ref/default entry 'kind #f) kind)) + entries)) + (def (python-cp-entry-with-kind entries kind) + (let ([matches (python-cp-entries-with-kind entries kind)]) + (and (not (null? matches)) (car matches)))) + (def (python-cp-try-throw-kind source start end) + (let ([body (substring source start end)]) + (cond + [(string-find-substring body "raise ") 'definite] + [(string-find-substring + body + "any_function_call_may_raise()") + 'may] + [else 'none]))) + (def (python-cp-flow-analyze-entry source entry env) + (python-cp-flow-analyze-range + source + (alist-ref/default entry 'body-start 0) + (alist-ref/default entry 'body-end 0) + env)) + (def (python-cp-flow-analyze-if source first line-start + block-end range-end env) + (let* ([indent (- first line-start)] + [body-start (python-cp-next-line-start source line-start)] + [else-start (python-cp-same-indent-clause-start source body-start block-end indent + python-cp-else-line?)] + [then-end (or else-start block-end)] + [then-env (python-cp-flow-analyze-range + source + body-start + then-end + env)] + [else-env (if else-start + (python-cp-flow-analyze-range + source + (python-cp-next-line-start source else-start) + block-end + env) + env)]) + (cond + [(and (< range-end block-end) + (or (not else-start) (<= range-end else-start))) + then-env] + [(and else-start (< range-end block-end)) else-env] + [else (python-cp-merge-envs then-env else-env)]))) + (def (python-cp-flow-analyze-try source first line-start + block-end env) + (let* ([indent (- first line-start)] + [entries (python-cp-try-clause-entries + source + line-start + block-end + indent)] + [try-entry (python-cp-entry-with-kind entries 'try)] + [except-entries (python-cp-entries-with-kind + entries + 'except)] + [else-entry (python-cp-entry-with-kind entries 'else)] + [finally-entry (python-cp-entry-with-kind entries 'finally)] + [try-env (if try-entry + (python-cp-flow-analyze-entry + source + try-entry + env) + env)] + [throw-kind (if try-entry + (python-cp-try-throw-kind + source + (alist-ref/default try-entry 'body-start 0) + (alist-ref/default try-entry 'body-end 0)) + 'none)] + [normal-env (and (not (eq? throw-kind 'definite)) + (if else-entry + (python-cp-flow-analyze-entry + source + else-entry + try-env) + try-env))] + [except-envs (map (lambda (entry) + (python-cp-flow-analyze-entry + source + entry + env)) + except-entries)] + [merged (case throw-kind + [(definite) + (python-cp-merge-env-list except-envs env)] + [(none) normal-env] + [else + (python-cp-merge-env-list + (if normal-env + (cons normal-env except-envs) + except-envs) + env)])] + [with-finally (if finally-entry + (python-cp-flow-analyze-entry + source + finally-entry + merged) + merged)]) + with-finally)) + (def (python-cp-flow-analyze-assignment + source + line-start + line-end + env) + (let ([augmented (python-cp-augmented-assignment-line-name + source + line-start + line-end)]) + (if augmented + (python-cp-remove-binding env augmented) + (let ([assignment (python-cp-assignment-line-value + source + line-start + line-end + env)]) + (if assignment + (let ([name (alist-ref/default assignment 'name "")] + [value (alist-ref/default + assignment + 'value + python-cp-missing-value)]) + (if (python-cp-missing? value) + (python-cp-remove-binding env name) + (python-cp-set-binding env name value))) + env))))) + (def (python-cp-flow-analyze-range source start end env) + (let loop ([line-start start] [bindings env]) + (if (>= line-start end) + bindings + (let* ([line-end (min (line-end-after source line-start) + end)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [trimmed (string-trim line)] + [next (python-cp-next-line-start source line-start)]) + (cond