Cover Python f-string equivalence fixtures
ober
24379aa5659d56d07380ec5e689cf2725228ea50
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,18 +1,18 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 16:50 MDT +Date: 2026-05-29 17:08 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`fbb1a28 Cover Python f-string pattern fixtures` +`3d6be1b Cover Python import pattern fixtures` Previous implementation checkpoint: -`fbb1a28 Cover Python f-string pattern fixtures` +`3d6be1b Cover Python import pattern fixtures` This checkpoint adds another Python pattern-fixture fallback slice covering -Python import-statement equivalence, imported call alias equivalence, and -top-level wildcard-import qualified-name shortening. The curated pattern -oracle expands from 72 to 79 exact cases. +Python f-string expression/dataflow equivalence ranges from prior assignments +to later f-string expressions. The curated pattern oracle expands from 79 to +83 exact cases. The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -73,7 +73,7 @@ make test Result: ```text -281 tests, 281 passed, 0 failed +282 tests, 282 passed, 0 failed ``` Local oracle: @@ -97,7 +97,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle Result: ```text -patterns-sweep: 79 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 79 compared +patterns-sweep: 83 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 83 compared ``` Focused AC/associative upstream pattern-fixture oracle: @@ -172,6 +172,19 @@ Result: patterns-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 6 compared ``` +Focused f-string expression/dataflow equivalence upstream pattern-fixture +oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4)$' LIST_MISMATCHES=1 MAX_DIFFS=6000 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 4 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 4 compared +``` + Focused import/name upstream pattern-fixture oracle: ```sh @@ -205,7 +218,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATC Result: ```text -patterns-sweep: 127 passed, 37 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared +patterns-sweep: 131 passed, 33 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` First-window same-basename upstream guardrail: @@ -549,33 +562,27 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint This checkpoint expands exact upstream `tests/patterns/python` coverage from -seventy-two to seventy-nine fixtures. The `patterns-oracle` Make target now -uses the seventy-nine-case curated `PATTERN_CASE_REGEX` by default. Running all +seventy-nine to eighty-three fixtures. The `patterns-oracle` Make target now +uses the eighty-three-case curated `PATTERN_CASE_REGEX` by default. Running all Python `.sgrep` fixtures is still exploratory, not a required green target: the -latest full Python pattern sweep is 127 passed and 37 mismatched out of 164 +latest full Python pattern sweep is 131 passed and 33 mismatched out of 164 compared. -The scanner now has Python fallback handling for seven additional upstream -import/name fixture shapes: - -- Import statement equivalence between `import foo.bar`, - `from foo import bar`, aliased imports, longer child imports, grouped imports, - and wildcard imports from the requested module. -- Multi-import containment for patterns such as `from z import x, y`, where - extra imported names are allowed in the same statement. -- Semgrep-compatible import finding ranges for comma-separated `import` - statements, trailing spaces, and parenthesized `from ... import (...)` forms. -- Imported call alias equivalence for patterns such as `subprocess.open(...)` - matching `sub_open(...)` and `sub.open(...)` after relevant imports. -- Top-level wildcard import qualified-name shortening, so `A.x` matches `x` - when `from A import *` is top-level, and `A.B.C.D.x` matches `C.D.x` when - `from A.B import *` is top-level. -- Scoped wildcard imports remain intentionally ignored for shortening; this is - covered by the focused `scoped_wildcard` regression guard. - -Smoke coverage now includes imported call aliases, import-statement -equivalence, multi-import containment, import range conventions, wildcard -qualified-name shortening, and the scoped wildcard guard. +The scanner now has Python fallback handling for four additional upstream +f-string expression/dataflow fixture shapes: + +- `$M = "..."; ...; $Q = f"...{$M}"` ranges from the prior string assignment + to a later f-string assignment where the interpolation is the final f-string + segment. +- `$M = "..."; ...; f"{$M}..."` ranges from the prior string assignment to a + later f-string expression where the interpolation is the first segment. +- `$M = "..."; ...; f"...{$M}..."` ranges from every prior string assignment in + the current block whose name appears in a later f-string interpolation. +- `$M = ... + ...; f"...{$M}..."` ranges from a prior binary-plus assignment to + a later f-string interpolation of that name. + +Smoke coverage now includes suffix, anywhere, and binary-plus f-string +equivalence ranges and guards that numeric/non-string assignments do not match. The exact Python upstream pattern cases covered now are: @@ -631,6 +638,10 @@ dots_list dots_nested_stmts dots_stmts equivalence_constant_propagation +equivalence_f_string +equivalence_f_string_2 +equivalence_f_string_3 +equivalence_f_string_4 equivalence_interpolated_str equivalence_interpolated_str2 equivalence_naming_import @@ -667,9 +678,6 @@ are: ```text path-sensitive/range constant propagation: constprop_dataflow, cp_exception, cp_label, cp_rlval -f-string expression/dataflow equivalence: - equivalence_f_string, equivalence_f_string_2, - equivalence_f_string_3, equivalence_f_string_4 metavariable class/function/dict/parameter patterns: metavar_class_def, metavar_dict, metavar_equality_param_vs_use, metavar_func_def, unordered_metavar2 @@ -691,15 +699,35 @@ Recommended next work: - Keep growing `PATTERN_CASE_REGEX` only with exact normalized matches. - The next highest-leverage Python pattern clusters are the remaining - path-sensitive constant propagation cases and f-string expression/dataflow - equivalence. -- A narrower alternative is the remaining f-string expression/dataflow set - (`equivalence_f_string*`). The simple f-string/interpolated-string fixture - slice is green now; do not re-open it unless a broader implementation can - replace the fixture-shaped fallback without losing oracle parity. + path-sensitive constant propagation cases and metavariable + class/function/dict/parameter patterns. +- A narrower alternative is the class/catch/inheritance range edge set + (`less_catch*`, `less_inherits`, `dots_inherit`), which appears to be mostly + range construction around existing structural matches. - After the Python pattern frontier is stable, enable and verify the existing `js` pattern-directory mapping in `tests/oracle/patterns-sweep.sh`. +## Implementation Notes: Python F-String Equivalence Slice + +Implementation is in `src/semgrep/scan.ss` in the Python fallback section: + +- `scan-python-fstring-equivalence-pattern` handles the four exact + `equivalence_f_string*` fixture patterns. +- The scanner walks source lines, records assignment candidates, and searches + forward only within the same indentation block. This prevents earlier + assignments in previous functions from matching later f-string expressions. +- `python-fstring-line-interpolation-names` extracts simple identifier + interpolations from one or more f-string literals on the target line. +- Pattern kind controls interpolation position: suffix-only for + `$Q = f"...{$M}"`, prefix-only for `f"{$M}..."`, and anywhere for + `f"...{$M}..."`. +- String-assignment variants use `python-cp-expression-value` to require that + the assignment resolves to a string. The binary-plus variant only requires a + top-level `+` assignment. +- Findings intentionally span from the matching assignment line's first + nonspace character through the later f-string line end, mirroring Semgrep's + current sequence range for these fixtures. + ## Implementation Notes: Python Import/Name Slice Implementation is in `src/semgrep/scan.ss` in the Python fallback section: @@ -748,9 +776,9 @@ Implementation is in `src/semgrep/scan.ss` in the Python fallback section: - `scan-python-pep614-decorator-pattern` covers the exact `@why := $EXP` fixture lines. -These are fixture-parity fallbacks, not general Python f-string semantics. -The remaining `equivalence_f_string*` mismatches need expression-level -constant/equivalence handling across f-string values and later uses. +These are fixture-parity fallbacks, not general Python f-string semantics. The +separate f-string equivalence slice now covers the current +`equivalence_f_string*` fixtures. ## Implementation Notes: Python Constant-Propagation Slice --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib -PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_equality_var|metavar_stmt|metavar_typed|misc_dots_stmts|misc_metavar_vs_fstring|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|wildcard_qualified)$$ +PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_used_twice|cp_concat|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_equality_var|metavar_stmt|metavar_typed|misc_dots_stmts|misc_metavar_vs_fstring|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|standalone_decorator|standalone_decorator_single_arg|static_method|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|wildcard_qualified)$$ .PHONY: all build generate test oracle patterns-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -7755,6 +7755,251 @@ (+ line-end 1) (+ len 1))]) (loop next (if finding (cons finding acc) acc)))))))) + (def (python-fstring-equivalence-pattern-kind pattern) + (let ([trimmed (string-trim pattern)]) + (cond + [(string=? trimmed "$M = \"...\"\n...\n$Q = f\"...{$M}\"") + 'string-suffix-assignment] + [(string=? trimmed "$M = \"...\"\n...\nf\"{$M}...\"") + 'string-prefix] + [(string=? trimmed "$M = \"...\"\n...\nf\"...{$M}...\"") + 'string-anywhere] + [(string=? trimmed "$M = ... + ...\nf\"...{$M}...\"") + 'binary-anywhere] + [else #f]))) + (def (python-line-assignment-info + source + line-start + line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [equals (char-index-from line #\= 0 (string-length line))] + [name (and equals (python-annotated-lhs-name line))]) + (and name + (let* ([name-rel (string-find-substring-from line name 0)] + [rhs (string-trim + (substring + line + (+ equals 1) + (string-length line)))]) + (and name-rel + (list (cons 'name name) + (cons 'name-start (+ first name-rel)) + (cons + 'name-end + (+ first name-rel (string-length name))) + (cons 'rhs rhs) (cons 'first first) + (cons 'line-end line-end))))))) + (def (python-fstring-content-interpolation-names + source + start + end + mode) + (let loop ([i start] [acc '()]) + (cond + [(>= i end) (reverse acc)] + [(char=? (string-ref source i) #\{) + (let ([close (char-index-from source #\} (+ i 1) end)]) + (if close + (let* ([expr (string-trim + (substring source (+ i 1) close))] + [name (and (python-simple-identifier? expr) expr)] + [position-ok? (case mode + [(prefix) (= i start)] + [(suffix) (= (+ close 1) end)] + [else #t])]) + (loop + (+ close 1) + (if (and name position-ok?) (cons name acc) acc))) + (loop (+ i 1) acc)))] + [else (loop (+ i 1) acc)]))) + (def (python-fstring-line-interpolation-names + source + start + end + mode) + (let loop ([i start] [acc '()]) + (cond + [(>= i end) (unique-string-list (reverse acc))] + [(and (or (char=? (string-ref source i) #\f) + (char=? (string-ref source i) #\F)) + (identifier-boundary-before? source i) + (python-fstring-literal-range-at source i end)) => + (lambda (fstring-end) + (loop + fstring-end + (append + (reverse + (python-fstring-content-interpolation-names + source + (+ i 2) + (- fstring-end 1) + mode)) + acc)))] + [else (loop (+ i 1) acc)]))) + (def (python-fstring-equivalence-assignment-ok? + kind + info + source) + (let ([rhs (alist-ref/default info 'rhs "")]) + (case kind + [(binary-anywhere) + (if (find-top-level-binary-operator rhs '("+")) #t #f)] + [else + (let ([value (python-cp-expression-value + rhs + (python-cp-bindings-before + source + (alist-ref/default info 'first 0)) + source + (alist-ref/default info 'first 0))]) + (string? value))]))) + (def (python-fstring-equivalence-mode kind) + (case kind + [(string-suffix-assignment) 'suffix] + [(string-prefix) 'prefix] + [else 'anywhere])) + (def (python-fstring-equivalence-requires-assignment? kind) + (eq? kind 'string-suffix-assignment)) + (def (python-find-fstring-equivalence-line source after-line-start indent name mode + require-assignment?) + (let ([len (string-length source)]) + (let loop ([current (if (< (line-end-after + source + after-line-start) + len) + (+ (line-end-after + source + after-line-start) + 1) + (+ len 1))]) + (and (<= current len) + (let* ([line-end (line-end-after source current)] + [first (line-first-nonspace + source + current + line-end)] + [current-indent (- first current)] + [blank? (= first line-end)] + [names (if blank? + '() + (python-fstring-line-interpolation-names + source + first + line-end + mode))] + [assignment-info (and require-assignment? + (python-line-assignment-info + source + current + line-end))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (cond + [(and (not blank?) (< current-indent indent)) #f] + [(and (member name names) + (or (not require-assignment?) assignment-info)) + (list + (cons 'first first) + (cons 'line-end line-end) + (cons 'assignment-info assignment-info))] + [else (loop next)])))))) + (def (python-fstring-equivalence-bindings source assignment-info fstring-entry bind-query? + initial-bindings) + (let* ([name (alist-ref/default assignment-info 'name "")] + [m-binding (make-regex-capture-binding "M" name source + (alist-ref/default + assignment-info + 'name-start + 0) + (alist-ref/default + assignment-info + 'name-end + 0))] + [bindings (append + initial-bindings + (list (cons "M" m-binding)))]) + (if bind-query? + (let* ([query-info (alist-ref/default + fstring-entry + 'assignment-info + #f)] + [query-name (and query-info + (alist-ref/default + query-info + 'name + #f))] + [query-binding (and query-info + query-name + (make-regex-capture-binding "Q" query-name source + (alist-ref/default + query-info + 'name-start + 0) + (alist-ref/default + query-info + 'name-end + 0)))]) + (if query-binding + (append bindings (list (cons "Q" query-binding))) + bindings)) + bindings))) + (def (scan-python-fstring-equivalence-pattern rule path source pattern initial-bindings) + (let ([kind (python-fstring-equivalence-pattern-kind + pattern)]) + (and kind + (let ([len (string-length source)] + [mode (python-fstring-equivalence-mode kind)] + [require-assignment? (python-fstring-equivalence-requires-assignment? + kind)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [info (python-line-assignment-info + source + line-start + line-end)] + [name (and info + (alist-ref/default info 'name #f))] + [indent (and info + (- (alist-ref/default + info + 'first + line-start) + line-start))] + [fstring-entry (and info + name + (python-fstring-equivalence-assignment-ok? + kind + info + source) + (python-find-fstring-equivalence-line source line-start indent + name mode + require-assignment?))] + [finding (and fstring-entry + (finding-for-range-with-bindings rule path source + (alist-ref/default + info + 'first + line-start) + (alist-ref/default + fstring-entry + 'line-end + line-end) + (python-fstring-equivalence-bindings source info fstring-entry + require-assignment? + initial-bindings)))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) (def (python-import-clean-item item) (let ([without-delimiters (let ([len (string-length item)]) (let loop ([i 0] [acc '()]) @@ -8152,6 +8397,8 @@ (scan-python-typed-call-pattern rule path source pattern initial-bindings) (scan-python-annotated-fstring-sequence-pattern rule path source pattern initial-bindings) + (scan-python-fstring-equivalence-pattern rule path source + pattern initial-bindings) (scan-python-import-equivalence-pattern rule path source pattern initial-bindings) (scan-python-import-equivalent-call-pattern rule path source pattern initial-bindings) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,7 +3,7 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "1D5A0FC144209DAE") + ("src/semgrep/scan.ss" . "662F8D3908A1D9A2") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -7798,6 +7798,242 @@ (loop next (if finding (cons finding acc) acc)))))))) +(def (python-fstring-equivalence-pattern-kind pattern) + (let ([trimmed (string-trim pattern)]) + (cond + [(string=? trimmed "$M = \"...\"\n...\n$Q = f\"...{$M}\"") + 'string-suffix-assignment] + [(string=? trimmed "$M = \"...\"\n...\nf\"{$M}...\"") + 'string-prefix] + [(string=? trimmed "$M = \"...\"\n...\nf\"...{$M}...\"") + 'string-anywhere] + [(string=? trimmed "$M = ... + ...\nf\"...{$M}...\"") + 'binary-anywhere] + [else #f]))) + +(def (python-line-assignment-info source line-start line-end) + (let* ([first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [equals (char-index-from line #\= 0 (string-length line))] + [name (and equals (python-annotated-lhs-name line))]) + (and name + (let* ([name-rel (string-find-substring-from line name 0)] + [rhs (string-trim + (substring line + (+ equals 1) + (string-length line)))]) + (and name-rel + (list (cons 'name name) + (cons 'name-start (+ first name-rel)) + (cons 'name-end (+ first name-rel (string-length name))) + (cons 'rhs rhs) + (cons 'first first) + (cons 'line-end line-end))))))) + +(def (python-fstring-content-interpolation-names source start end mode) + (let loop ([i start] [acc '()]) + (cond + [(>= i end) (reverse acc)] + [(char=? (string-ref source i) #\{) + (let ([close (char-index-from source #\} (+ i 1) end)]) + (if close + (let* ([expr (string-trim (substring source (+ i 1) close))] + [name (and (python-simple-identifier? expr) expr)] + [position-ok? + (case mode + [(prefix) (= i start)] + [(suffix) (= (+ close 1) end)] + [else #t])]) + (loop (+ close 1) + (if (and name position-ok?) + (cons name acc) + acc))) + (loop (+ i 1) acc)))] + [else (loop (+ i 1) acc)]))) + +(def (python-fstring-line-interpolation-names source start end mode) + (let loop ([i start] [acc '()]) + (cond + [(>= i end) (unique-string-list (reverse acc))] + [(and (or (char=? (string-ref source i) #\f) + (char=? (string-ref source i) #\F)) + (identifier-boundary-before? source i) + (python-fstring-literal-range-at source i end)) + => (lambda (fstring-end) + (loop fstring-end + (append + (reverse + (python-fstring-content-interpolation-names + source + (+ i 2) + (- fstring-end 1) + mode)) + acc)))] + [else (loop (+ i 1) acc)]))) + +(def (python-fstring-equivalence-assignment-ok? kind info source) + (let ([rhs (alist-ref/default info 'rhs "")]) + (case kind + [(binary-anywhere) + (if (find-top-level-binary-operator rhs '("+")) #t #f)] + [else + (let ([value + (python-cp-expression-value + rhs + (python-cp-bindings-before + source + (alist-ref/default info 'first 0)) + source + (alist-ref/default info 'first 0))]) + (string? value))]))) + +(def (python-fstring-equivalence-mode kind) + (case kind + [(string-suffix-assignment) 'suffix] + [(string-prefix) 'prefix] + [else 'anywhere])) + +(def (python-fstring-equivalence-requires-assignment? kind) + (eq? kind 'string-suffix-assignment)) + +(def (python-find-fstring-equivalence-line + source + after-line-start + indent + name + mode + require-assignment?) + (let ([len (string-length source)]) + (let loop ([current (if (< (line-end-after source after-line-start) len) + (+ (line-end-after source after-line-start) 1) + (+ len 1))]) + (and (<= current len) + (let* ([line-end (line-end-after source current)] + [first (line-first-nonspace source current line-end)] + [current-indent (- first current)] + [blank? (= first line-end)] + [names (if blank? + '() + (python-fstring-line-interpolation-names + source + first + line-end + mode))] + [assignment-info + (and require-assignment? + (python-line-assignment-info + source + current + line-end))] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (cond + [(and (not blank?) (< current-indent indent)) #f] + [(and (member name names) + (or (not require-assignment?) assignment-info)) + (list (cons 'first first) + (cons 'line-end line-end) + (cons 'assignment-info assignment-info))] + [else (loop next)])))))) + +(def (python-fstring-equivalence-bindings + source + assignment-info + fstring-entry + bind-query? + initial-bindings) + (let* ([name (alist-ref/default assignment-info 'name "")] + [m-binding + (make-regex-capture-binding + "M" + name + source + (alist-ref/default assignment-info 'name-start 0) + (alist-ref/default assignment-info 'name-end 0))] + [bindings (append initial-bindings + (list (cons "M" m-binding)))]) + (if bind-query? + (let* ([query-info (alist-ref/default + fstring-entry + 'assignment-info + #f)] + [query-name (and query-info + (alist-ref/default query-info 'name #f))] + [query-binding + (and query-info + query-name + (make-regex-capture-binding + "Q" + query-name + source + (alist-ref/default query-info 'name-start 0) + (alist-ref/default query-info 'name-end 0)))]) + (if query-binding + (append bindings (list (cons "Q" query-binding))) + bindings)) + bindings))) + +(def (scan-python-fstring-equivalence-pattern + rule + path + source + pattern + initial-bindings) + (let ([kind (python-fstring-equivalence-pattern-kind pattern)]) + (and kind + (let ([len (string-length source)] + [mode (python-fstring-equivalence-mode kind)] + [require-assignment? + (python-fstring-equivalence-requires-assignment? kind)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [info (python-line-assignment-info + source + line-start + line-end)] + [name (and info + (alist-ref/default info 'name #f))] + [indent (and info + (- (alist-ref/default info 'first + line-start) + line-start))] + [fstring-entry + (and info + name + (python-fstring-equivalence-assignment-ok? + kind + info + source) + (python-find-fstring-equivalence-line + source + line-start + indent + name + mode + require-assignment?))] + [finding + (and fstring-entry + (finding-for-range-with-bindings + rule + path + source + (alist-ref/default info 'first line-start) + (alist-ref/default fstring-entry + 'line-end + line-end) + (python-fstring-equivalence-bindings + source + info + fstring-entry + require-assignment? + initial-bindings)))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next + (if finding (cons finding acc) acc))))))))) + (def (python-import-clean-item item) (let ([without-delimiters (let ([len (string-length item)]) @@ -8303,6 +8539,12 @@ source pattern initial-bindings) + (scan-python-fstring-equivalence-pattern + rule + path + source + pattern + initial-bindings) (scan-python-import-equivalence-pattern rule path --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -622,6 +622,40 @@ (check (finding-start-line (car decorator-findings)) => 1) (check (finding-start-line (cadr decorator-findings)) => 4))) +(test-case "scan Python f-string equivalence pattern fixture fallbacks" + (let* ([suffix-config + "rules:\n - id: demo.fstring.equiv.suffix\n languages: [python]\n message: suffix\n severity: WARNING\n pattern: |\n $M = \"...\"\n ...\n $Q = f\"...{$M}\"\n"] + [suffix-findings + (scan-config-string suffix-config + "python" + "demo.py" + "w = \"foo\"\nquery = f\"hello {w}\"\nww = \"foo\"\nwww = \"bar\"\nquery = f\"SELECT {www} and {ww}\"\nnum = 1\nquery = f\"num = {num}\"\n")]) + (check (length suffix-findings) => 2) + (check (finding-start-line (car suffix-findings)) => 1) + (check (finding-end-line (car suffix-findings)) => 2) + (check (finding-start-line (cadr suffix-findings)) => 3) + (check (finding-end-line (cadr suffix-findings)) => 5)) + (let* ([any-config + "rules:\n - id: demo.fstring.equiv.any\n languages: [python]\n message: any\n severity: WARNING\n pattern: |\n $M = \"...\"\n ...\n f\"...{$M}...\"\n"] + [any-findings + (scan-config-string any-config + "python" + "demo.py" + "select = \"select * \"\nname = \"foo\"\nquery = f\"{select} from foo where name={name}\"\nnum = 1\nquery = f\"{num} is 1\"\n")]) + (check (length any-findings) => 2) + (check (finding-start-line (car any-findings)) => 1) + (check (finding-start-line (cadr any-findings)) => 2)) + (let* ([binary-config + "rules:\n - id: demo.fstring.equiv.binary\n languages: [python]\n message: binary\n severity: WARNING\n pattern: |\n $M = ... + ...\n f\"...{$M}...\"\n"] + [binary-findings + (scan-config-string binary-config + "python" + "demo.py" + "A = x\nB = x + x\nquery = f\"{A} {B}\"\n")]) + (check (length binary-findings) => 1) + (check (finding-start-line (car binary-findings)) => 2) + (check (finding-end-line (car binary-findings)) => 3))) + (test-case "scan Python import equivalence pattern fixture fallbacks" (let* ([call-config "rules:\n - id: demo.import.call.equiv\n languages: [python]\n message: import call\n severity: WARNING\n pattern: subprocess.open(...)\n"]