Cover JavaScript template literal fixtures
ober
bb63b69e35f7846f375793336a0486f77215974c
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,23 +1,23 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 21:33 MDT +Date: 2026-05-29 22:35 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`9490f84 Cover JavaScript equivalence fixtures` +`0109749 Cover JavaScript object field fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Do not treat this handoff as completion. This handoff records the clean Python `tests/patterns/python/*.sgrep` frontier and the promoted JavaScript `tests/patterns/js/*.sgrep` -constant-propagation/equivalence/object-field slice. +constant-propagation/equivalence/object-field/template-literal slice. Current headline: - Full Python pattern sweep: 164 passed / 0 mismatched. -- Full JavaScript pattern sweep: 63 passed / 69 mismatched, improved from - 56 passed / 76 mismatched by closing the JS object/destructuring slice. +- Full JavaScript pattern sweep: 69 passed / 63 mismatched, improved from + 63 passed / 69 mismatched by closing the JS template-literal slice. - Broad same-basename upstream sweep: 437 passed / 0 mismatched / 0 Jerboa errors, with 3 current-Semgrep oracle errors. @@ -74,7 +74,7 @@ make test Result: ```text -288 tests, 288 passed, 0 failed +289 tests, 289 passed, 0 failed ``` Focused constant/dataflow slice: @@ -157,7 +157,8 @@ Result: upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` -Promoted JavaScript constant-propagation/equivalence/object-field oracle: +Promoted JavaScript constant-propagation/equivalence/object-field/template +oracle: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle @@ -166,7 +167,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle Result: ```text -patterns-sweep: 22 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 22 compared +patterns-sweep: 28 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 28 compared ``` Full upstream JavaScript pattern sweep: @@ -178,7 +179,19 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES= Result: ```text -patterns-sweep: 63 passed, 69 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 69 passed, 63 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +``` + +Focused JavaScript template-literal slice: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(dots_template_literals|ellipsis_metavar_template|ellipsis_metavar_tagged_template|metavar_template|template_string_w_metavar|misc_tagged_template_literal)$' LIST_MISMATCHES=1 MAX_DIFFS=100 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 6 compared ``` Broad same-basename upstream sweep: @@ -225,6 +238,9 @@ cp_label cp_throw cp_undeclared cp_undeclared1 +dots_template_literals +ellipsis_metavar_tagged_template +ellipsis_metavar_template infer_const infer_const_1 infer_const_regexp @@ -232,9 +248,12 @@ equivalence_constant_propagation2 equivalence_constant_propagation3 equivalence_keyword_args equivalence_varlet +metavar_template misc_object_directly +misc_tagged_template_literal object_duplicate_key object_numeric_key +template_string_w_metavar unordered_destructuring unordered_fields unordered_fields2 @@ -347,6 +366,27 @@ JavaScript object/destructuring support added in this checkpoint: `let_is_var: false` option check as var/let equivalence so rule-level semantics are preserved. +JavaScript template-literal support added in this checkpoint: + +- `scan-javascript-template-literal-pattern` is wired into the JavaScript + `pattern` path after object fallbacks and before JS constant propagation. +- The fallback parses simple backtick pattern forms and scans matching source + template literals while skipping `${...}` expression bodies correctly enough + for the upstream fixture shapes. +- Assignment patterns such as ``$X = `...${...}...` `` match assignment ranges + from the bound left-hand side through Semgrep's template range endpoint. +- Bare template patterns such as `` `$LEFT${...}$RIGHT` `` and + `` `$...LEFT${...}$...RIGHT` `` match template literals with interpolations, + including the Semgrep-compatible range that excludes the closing backtick. +- Tagged template patterns such as ``$T`...` `` match from the tag start through + the Semgrep-compatible template endpoint and bind the tag metavariable. +- Interpolation metavariable patterns such as `` `...${$X}...` `` bind the + expression content and follow Semgrep's narrower endpoint before the closing + interpolation brace. +- Parenthesized template-call patterns such as + ``prisma.$executeRaw(`...`)`` match calls whose single argument is a template + literal, while intentionally not matching tagged template calls. + ## Smoke Coverage Added `tests/smoke.ss` extends `scan Python constant propagation pattern fixture @@ -373,7 +413,12 @@ This checkpoint adds a new top-level smoke case, field matching, numeric key normalization, duplicate keys, unordered object declarations, and unordered destructuring declarations. -The smoke suite count is now 288. +This checkpoint adds a new top-level smoke case, +`scan JavaScript template literal pattern fixture fallbacks`, covering +assignment-to-template patterns, bare template metavariables, interpolation +metavariables, tagged templates, and parenthesized template-call patterns. + +The smoke suite count is now 289. ## Current Frontier @@ -384,10 +429,11 @@ patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 s ``` The full JavaScript `.sgrep` pattern fixture sweep is not clean yet, but the -promoted JS constant-propagation/equivalence/object-field slice is green: +promoted JS constant-propagation/equivalence/object-field/template-literal slice +is green: ```text -patterns-sweep: 63 passed, 69 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 69 passed, 63 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` Do not mark the overall goal complete from this alone. Next high-value work: @@ -395,18 +441,20 @@ Do not mark the overall goal complete from this alone. Next high-value work: 1. Continue the JavaScript pattern frontier. The next concentrated clusters from the full JS sweep are deep/caching expression matching, import equivalence/import metavariable variations, JSX/XML dots/less/metavariable - range parity, tagged/template literal metavariables, and arrow/class/partial - pattern shapes. + range parity, and arrow/class/partial pattern shapes. 2. Keep `cp_array_destructure` as a regression guard: the JS equality fallback must invalidate constants after destructuring reassignment and avoid matching `imageOrNull == null`. 3. Keep `misc_var_is_var` as a regression guard for `let_is_var: false` when touching declaration equivalence or declaration object/destructuring fallbacks. -4. Decide whether to promote all 164 Python pattern fixtures into +4. Keep the six promoted template fixtures as a range regression guard when + touching template-literal scanning, especially `template_string_w_metavar`, + whose Semgrep range ends before the closing interpolation brace. +5. Decide whether to promote all 164 Python pattern fixtures into `PATTERN_CASE_REGEX` or keep `make patterns-oracle` at the curated 116-case runtime. -5. Rerun the broad upstream same-basename sweep after any fallback that can +6. Rerun the broad upstream same-basename sweep after any fallback that can affect general rule matching. ## Operational Notes --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ -JS_PATTERN_CASE_REGEX ?= ^(cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|misc_object_directly|object_duplicate_key|object_numeric_key|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ +JS_PATTERN_CASE_REGEX ?= ^(cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|metavar_template|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|template_string_w_metavar|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ .PHONY: all build generate test oracle patterns-oracle patterns-js-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -16767,6 +16767,546 @@ source pattern) (scan-javascript-member-ellipsis-plus-pattern-with-bindings rule path source pattern '())) + (def (javascript-template-literal-end source start) + (let ([len (string-length source)]) + (let loop ([i (+ start 1)] [escaped? #f]) + (cond + [(>= i len) #f] + [escaped? (loop (+ i 1) #f)] + [(char=? (string-ref source i) #\\) (loop (+ i 1) #t)] + [(and (< (+ i 1) len) + (char=? (string-ref source i) #\$) + (char=? (string-ref source (+ i 1)) #\{)) + (let ([close (js-cp-template-expression-close + source + (+ i 1) + len)]) + (and close (loop (+ close 1) #f)))] + [(char=? (string-ref source i) #\`) (+ i 1)] + [else (loop (+ i 1) #f)])))) + (def (javascript-template-expression-ranges + source + template-start + template-end) + (let loop ([i (+ template-start 1)] [escaped? #f] [acc '()]) + (cond + [(>= i (- template-end 1)) (reverse acc)] + [escaped? (loop (+ i 1) #f acc)] + [(char=? (string-ref source i) #\\) (loop (+ i 1) #t acc)] + [(and (< (+ i 1) (- template-end 1)) + (char=? (string-ref source i) #\$) + (char=? (string-ref source (+ i 1)) #\{)) + (let* ([close (js-cp-template-expression-close + source + (+ i 1) + (- template-end 1))] + [range (and close + (trim-range-horizontal + source + (+ i 2) + close))] + [entry (and range + (list + (cons 'dollar-start i) + (cons 'expr-start (car range)) + (cons 'expr-end (cdr range)) + (cons 'close close)))]) + (and close + (loop (+ close 1) #f (if entry (cons entry acc) acc))))] + [else (loop (+ i 1) #f acc)]))) + (def (javascript-template-pattern-metavariable-name text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (and (> len 1) + (char=? (string-ref trimmed 0) #\$) + (not (string=? trimmed "$")) + (not (string=? trimmed "...")) + (normalize-metavariable-name trimmed)))) + (def (javascript-template-pattern-expression-metavariable + content + expr-open + expr-close) + (let ([expr (string-trim + (substring content (+ expr-open 2) expr-close))]) + (and (> (string-length expr) 0) + (char=? (string-ref expr 0) #\$) + (not (string=? expr "$...")) + (not (string=? expr "...")) + (not (sg-string-prefix? "$..." expr)) + (normalize-metavariable-name expr)))) + (def (javascript-template-content-spec template-text) + (let* ([len (string-length template-text)] + [content (substring template-text 1 (- len 1))] + [expr-open (string-find-substring content "${")] + [expr-close (and expr-open + (char-index-from + content + #\} + (+ expr-open 2) + (string-length content)))] + [before (and expr-open (substring content 0 expr-open))] + [after (and expr-close + (substring + content + (+ expr-close 1) + (string-length content)))]) + (list (cons 'requires-expression (and expr-open #t)) + (cons + 'expression-metavariable + (and expr-open + expr-close + (javascript-template-pattern-expression-metavariable + content + expr-open + expr-close))) + (cons + 'left-metavariable + (and before + (javascript-template-pattern-metavariable-name before))) + (cons + 'right-metavariable + (and after + (javascript-template-pattern-metavariable-name after))) + (cons + 'wildcard + (or (string-find-substring content "...") + (string-find-substring content "$")))))) + (def (javascript-template-text-at pattern start end) + (let* ([template (string-trim + (substring pattern start end))] + [len (string-length template)]) + (and (>= len 2) + (char=? (string-ref template 0) #\`) + (char=? (string-ref template (- len 1)) #\`) + template))) + (def (javascript-template-call-pattern-spec + trimmed + len + tick + open-paren) + (let* ([close-paren (find-matching-close-paren + trimmed + open-paren)] + [function (string-trim (substring trimmed 0 open-paren))] + [arg-start (skip-pattern-horizontal + trimmed + (+ open-paren 1) + tick)] + [arg-end (and close-paren + (javascript-skip-whitespace-backward-exclusive + trimmed + (+ open-paren 1) + (- close-paren 1)))] + [template-end (javascript-template-literal-end + trimmed + tick)] + [template-text (and template-end + (= arg-start tick) + (= arg-end template-end) + (javascript-template-text-at + trimmed + tick + template-end))] + [template-spec (and template-text + (javascript-template-content-spec + template-text))]) + (and close-paren + (= close-paren len) + (not (string=? function "")) + template-spec + (list + (cons 'kind 'call) + (cons 'function function) + (cons 'template template-spec))))) + (def (javascript-template-noncall-pattern-spec + trimmed + len + tick) + (let* ([template-end (javascript-template-literal-end + trimmed + tick)] + [template-text (and template-end + (= (skip-pattern-horizontal + trimmed + template-end + len) + len) + (javascript-template-text-at + trimmed + tick + template-end))] + [template-spec (and template-text + (javascript-template-content-spec + template-text))]) + (and template-spec + (cond + [(char-index-from trimmed #\= 0 tick) => + (lambda (eq-index) + (let* ([lhs (string-trim + (substring trimmed 0 eq-index))] + [rhs-start (skip-pattern-horizontal + trimmed + (+ eq-index 1) + tick)] + [metavariable (javascript-template-pattern-metavariable-name + lhs)]) + (and metavariable + (= rhs-start tick) + (list + (cons 'kind 'assignment) + (cons 'metavariable metavariable) + (cons 'template template-spec)))))] + [(= tick 0) + (and (alist-ref/default template-spec 'wildcard #f) + (list + (cons 'kind 'bare) + (cons 'template template-spec)))] + [else + (let* ([tag (string-trim (substring trimmed 0 tick))] + [tag-metavariable (javascript-template-pattern-metavariable-name + tag)]) + (and tag-metavariable + (list + (cons 'kind 'tagged) + (cons 'tag-metavariable tag-metavariable) + (cons 'template template-spec))))])))) + (def (javascript-template-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)] + [tick (char-index-from trimmed #\` 0 len)]) + (and tick + (let ([open-paren (char-index-from trimmed #\( 0 tick)]) + (if open-paren + (javascript-template-call-pattern-spec + trimmed + len + tick + open-paren) + (javascript-template-noncall-pattern-spec + trimmed + len + tick)))))) + (def (javascript-template-binding-entry + name + source + start + end) + (and name + start + end + (cons + name + (make-regex-capture-binding name + (substring source start end) source start end)))) + (def (javascript-template-content-bindings spec source template-start template-end initial-bindings) + (let* ([expressions (javascript-template-expression-ranges + source + template-start + template-end)] + [requires-expression? (alist-ref/default + spec + 'requires-expression + #f)]) + (and (or (not requires-expression?) + (not (null? expressions))) + (let* ([first-expr (and (not (null? expressions)) + (car expressions))] + [expr-metavariable (alist-ref/default + spec + 'expression-metavariable + #f)] + [left-metavariable (alist-ref/default + spec + 'left-metavariable + #f)] + [right-metavariable (alist-ref/default + spec + 'right-metavariable + #f)] + [expr-binding (and first-expr + (javascript-template-binding-entry + expr-metavariable + source + (alist-ref/default + first-expr + 'expr-start + #f) + (alist-ref/default + first-expr + 'expr-end + #f)))] + [left-binding (and first-expr + (javascript-template-binding-entry + left-metavariable + source + (+ template-start 1) + (alist-ref/default + first-expr + 'dollar-start + #f)))] + [right-binding (and first-expr + (javascript-template-binding-entry + right-metavariable + source + (+ (alist-ref/default + first-expr + 'close + #f) + 1) + (- template-end 1)))] + [additions (sg-filter + (lambda (entry) entry) + (list + expr-binding + left-binding + right-binding))]) + (merge-binding-list initial-bindings additions))))) + (def (javascript-template-finding-end + spec + source + template-start + template-end) + (let ([expr-metavariable (alist-ref/default + spec + 'expression-metavariable + #f)]) + (if expr-metavariable + (let ([expressions (javascript-template-expression-ranges + source + template-start + template-end)]) + (if (null? expressions) + (- template-end 1) + (alist-ref/default + (car expressions) + 'expr-end + (- template-end 1)))) + (- template-end 1)))) + (def (javascript-template-tag-char? ch) + (or (identifier-char? ch) (char=? ch #\.))) + (def (javascript-template-tag-range source template-start) + (let ([end (skip-horizontal-backward + source + (- template-start 1))]) + (and (>= end 0) + (javascript-template-tag-char? (string-ref source end)) + (let loop ([start end]) + (if (and (>= start 0) + (javascript-template-tag-char? + (string-ref source start))) + (loop (- start 1)) + (cons (+ start 1) (+ end 1))))))) + (def (javascript-template-assignment-range + source + template-start) + (let* ([line-start (line-start-before + source + template-start)] + [eq-index (last-char-index-before + source + #\= + line-start + template-start)] + [after-eq (and eq-index + (skip-horizontal-forward + source + (+ eq-index 1)))] + [lhs-end0 (and eq-index + (skip-horizontal-backward + source + (- eq-index 1)))]) + (and eq-index + (= after-eq template-start) + lhs-end0 + (identifier-char? (string-ref source lhs-end0)) + (let loop ([start lhs-end0]) + (if (and (>= start line-start) + (let ([ch (string-ref source start)]) + (or (identifier-char? ch) (char=? ch #\.)))) + (loop (- start 1)) + (cons (+ start 1) (+ lhs-end0 1))))))) + (def (scan-javascript-template-bare-or-tagged-pattern rule path source spec initial-bindings) + (let ([kind (alist-ref/default spec 'kind #f)] + [template-spec (alist-ref/default spec 'template '())] + [tag-metavariable (alist-ref/default + spec + 'tag-metavariable + #f)] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([template-start (string-find-substring-from + source + "`" + start)]) + (if (not template-start) + (nonempty-findings (reverse acc)) + (let* ([template-end (javascript-template-literal-end + source + template-start)] + [tag-range (and template-end + (javascript-template-tag-range + source + template-start))] + [range-start (case kind + [(bare) + (and (not tag-range) + template-start)] + [(tagged) + (and tag-range + (car tag-range))] + [else #f])] + [tag-binding (and tag-metavariable + tag-range + (javascript-template-binding-entry + tag-metavariable + source + (car tag-range) + (cdr tag-range)))] + [initial (and range-start + (if tag-binding + (merge-binding-list + initial-bindings + (list tag-binding)) + initial-bindings))] + [bindings (and initial + (javascript-template-content-bindings template-spec source + template-start template-end + initial))] + [finding (and bindings + (finding-for-range-with-bindings rule path source range-start + (javascript-template-finding-end + template-spec + source + template-start + template-end) + bindings))] + [next (if template-end + (max (+ template-start 1) + template-end) + (+ template-start 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) + (def (scan-javascript-template-assignment-pattern rule path source spec initial-bindings) + (let ([name (alist-ref/default spec 'metavariable #f)] + [template-spec (alist-ref/default spec 'template '())] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([template-start (string-find-substring-from + source + "`" + start)]) + (if (not template-start) + (nonempty-findings (reverse acc)) + (let* ([template-end (javascript-template-literal-end + source + template-start)] + [assignment-range (and template-end + (javascript-template-assignment-range + source + template-start))] + [lhs-binding (and assignment-range + (javascript-template-binding-entry + name + source + (car assignment-range) + (cdr assignment-range)))] + [initial (and lhs-binding + (merge-binding-list + initial-bindings + (list lhs-binding)))] + [bindings (and initial + (javascript-template-content-bindings template-spec source + template-start template-end + initial))] + [finding (and bindings + (finding-for-range-with-bindings rule path source + (car assignment-range) + (javascript-template-finding-end + template-spec + source + template-start + template-end) + bindings))] + [next (if template-end + (max (+ template-start 1) + template-end) + (+ template-start 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) + (def (scan-javascript-template-call-pattern rule path source + spec initial-bindings) + (let* ([function (alist-ref/default spec 'function "")] + [template-spec (alist-ref/default spec 'template '())] + [needle (string-append function "(")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([call-start (string-find-substring-from + source + needle + start)]) + (if (not call-start) + (nonempty-findings (reverse acc)) + (let* ([open (+ call-start (string-length function))] + [close (find-matching-close-paren source open)] + [arg-start (and close + (skip-horizontal-forward + source + (+ open 1)))] + [arg-end (and close + (javascript-skip-whitespace-backward-exclusive + source + (+ open 1) + (- close 1)))] + [template-end (and arg-start + (< arg-start len) + (char=? + (string-ref + source + arg-start) + #\`) + (javascript-template-literal-end + source + arg-start))] + [bindings (and template-end + (= template-end arg-end) + (javascript-template-content-bindings template-spec source arg-start + template-end + initial-bindings))] + [finding (and bindings + (finding-for-range-with-bindings rule path source call-start + close bindings))] + [next (if close + (max (+ call-start 1) close) + (+ call-start 1))]) + (loop + next + (if finding (cons finding acc) acc))))))))) + (def (scan-javascript-template-literal-pattern-with-bindings rule path source pattern initial-bindings) + (let ([spec (javascript-template-pattern-spec pattern)]) + (and spec + (case (alist-ref/default spec 'kind #f) + [(bare tagged) + (scan-javascript-template-bare-or-tagged-pattern rule path source spec initial-bindings)] + [(assignment) + (scan-javascript-template-assignment-pattern rule path source spec initial-bindings)] + [(call) + (scan-javascript-template-call-pattern rule path source + spec initial-bindings)] + [else #f])))) + (def (scan-javascript-template-literal-pattern + rule + path + source + pattern) + (scan-javascript-template-literal-pattern-with-bindings rule path source pattern '())) (def (javascript-template-sql-expression-pattern? pattern) (and (string-find-substring pattern "`") (string-find-substring pattern "$SQLSTR") @@ -25951,6 +26491,12 @@ source (rule-pattern rule))) (and (javascript-like-language? language) + (scan-javascript-template-literal-pattern + rule + path + source + (rule-pattern rule))) + (and (javascript-like-language? language) (scan-javascript-cp-pattern-fallbacks rule path source (rule-pattern rule) '())) (and (or (string=? language "python") (string=? language "python2") --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "E1FA896B84952F4") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/scan.ss" . "706A3DCAF7BFED35") ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -17383,6 +17383,560 @@ pattern '())) +(def (javascript-template-literal-end source start) + (let ([len (string-length source)]) + (let loop ([i (+ start 1)] [escaped? #f]) + (cond + [(>= i len) #f] + [escaped? (loop (+ i 1) #f)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) #t)] + [(and (< (+ i 1) len) + (char=? (string-ref source i) #\$) + (char=? (string-ref source (+ i 1)) #\{)) + (let ([close (js-cp-template-expression-close + source + (+ i 1) + len)]) + (and close (loop (+ close 1) #f)))] + [(char=? (string-ref source i) #\`) (+ i 1)] + [else (loop (+ i 1) #f)])))) + +(def (javascript-template-expression-ranges source template-start template-end) + (let loop ([i (+ template-start 1)] [escaped? #f] [acc '()]) + (cond + [(>= i (- template-end 1)) (reverse acc)] + [escaped? (loop (+ i 1) #f acc)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) #t acc)] + [(and (< (+ i 1) (- template-end 1)) + (char=? (string-ref source i) #\$) + (char=? (string-ref source (+ i 1)) #\{)) + (let* ([close (js-cp-template-expression-close + source + (+ i 1) + (- template-end 1))] + [range (and close + (trim-range-horizontal + source + (+ i 2) + close))] + [entry (and range + (list (cons 'dollar-start i) + (cons 'expr-start (car range)) + (cons 'expr-end (cdr range)) + (cons 'close close)))]) + (and close + (loop (+ close 1) + #f + (if entry (cons entry acc) acc))))] + [else (loop (+ i 1) #f acc)]))) + +(def (javascript-template-pattern-metavariable-name text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (and (> len 1) + (char=? (string-ref trimmed 0) #\$) + (not (string=? trimmed "$")) + (not (string=? trimmed "...")) + (normalize-metavariable-name trimmed)))) + +(def (javascript-template-pattern-expression-metavariable + content + expr-open + expr-close) + (let ([expr (string-trim + (substring content (+ expr-open 2) expr-close))]) + (and (> (string-length expr) 0) + (char=? (string-ref expr 0) #\$) + (not (string=? expr "$...")) + (not (string=? expr "...")) + (not (sg-string-prefix? "$..." expr)) + (normalize-metavariable-name expr)))) + +(def (javascript-template-content-spec template-text) + (let* ([len (string-length template-text)] + [content (substring template-text 1 (- len 1))] + [expr-open (string-find-substring content "${")] + [expr-close (and expr-open + (char-index-from + content + #\} + (+ expr-open 2) + (string-length content)))] + [before (and expr-open + (substring content 0 expr-open))] + [after (and expr-close + (substring content + (+ expr-close 1) + (string-length content)))]) + (list + (cons 'requires-expression (and expr-open #t)) + (cons 'expression-metavariable + (and expr-open + expr-close + (javascript-template-pattern-expression-metavariable + content + expr-open + expr-close))) + (cons 'left-metavariable + (and before + (javascript-template-pattern-metavariable-name before))) + (cons 'right-metavariable + (and after + (javascript-template-pattern-metavariable-name after))) + (cons 'wildcard + (or (string-find-substring content "...") + (string-find-substring content "$")))))) + +(def (javascript-template-text-at pattern start end) + (let* ([template (string-trim (substring pattern start end))] + [len (string-length template)]) + (and (>= len 2) + (char=? (string-ref template 0) #\`) + (char=? (string-ref template (- len 1)) #\`) + template))) + +(def (javascript-template-call-pattern-spec trimmed len tick open-paren) + (let* ([close-paren (find-matching-close-paren trimmed open-paren)] + [function (string-trim (substring trimmed 0 open-paren))] + [arg-start (skip-pattern-horizontal + trimmed + (+ open-paren 1) + tick)] + [arg-end (and close-paren + (javascript-skip-whitespace-backward-exclusive + trimmed + (+ open-paren 1) + (- close-paren 1)))] + [template-end (javascript-template-literal-end trimmed tick)] + [template-text (and template-end + (= arg-start tick) + (= arg-end template-end) + (javascript-template-text-at + trimmed + tick + template-end))] + [template-spec (and template-text + (javascript-template-content-spec + template-text))]) + (and close-paren + (= close-paren len) + (not (string=? function "")) + template-spec + (list (cons 'kind 'call) + (cons 'function function) + (cons 'template template-spec))))) + +(def (javascript-template-noncall-pattern-spec trimmed len tick) + (let* ([template-end (javascript-template-literal-end trimmed tick)] + [template-text (and template-end + (= (skip-pattern-horizontal + trimmed + template-end + len) + len) + (javascript-template-text-at + trimmed + tick + template-end))] + [template-spec (and template-text + (javascript-template-content-spec + template-text))]) + (and template-spec + (cond + [(char-index-from trimmed #\= 0 tick) + => (lambda (eq-index) + (let* ([lhs (string-trim + (substring trimmed 0 eq-index))] + [rhs-start (skip-pattern-horizontal + trimmed + (+ eq-index 1) + tick)] + [metavariable + (javascript-template-pattern-metavariable-name + lhs)]) + (and metavariable + (= rhs-start tick) + (list (cons 'kind 'assignment) + (cons 'metavariable metavariable) + (cons 'template template-spec)))))] + [(= tick 0) + (and (alist-ref/default template-spec 'wildcard #f) + (list (cons 'kind 'bare) + (cons 'template template-spec)))] + [else + (let* ([tag (string-trim (substring trimmed 0 tick))] + [tag-metavariable + (javascript-template-pattern-metavariable-name tag)]) + (and tag-metavariable + (list (cons 'kind 'tagged) + (cons 'tag-metavariable tag-metavariable) + (cons 'template template-spec))))])))) + +(def (javascript-template-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)] + [tick (char-index-from trimmed #\` 0 len)]) + (and tick + (let ([open-paren (char-index-from trimmed #\( 0 tick)]) + (if open-paren + (javascript-template-call-pattern-spec + trimmed + len + tick + open-paren) + (javascript-template-noncall-pattern-spec + trimmed + len + tick)))))) + +(def (javascript-template-binding-entry name source start end) + (and name + start + end + (cons name + (make-regex-capture-binding + name + (substring source start end) + source + start + end)))) + +(def (javascript-template-content-bindings + spec + source + template-start