Cover JavaScript constant propagation fixtures
ober
c049e58c9c6639e6564d242ef2853628ed9c1137
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,16 +1,24 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 18:49 MDT +Date: 2026-05-29 20:13 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`bf1fe17 Cover Python statement range fixtures` +`40880ca Close Python pattern constant propagation fixtures` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. Do not treat this handoff as completion. This checkpoint closes -the remaining Python `tests/patterns/python/*.sgrep` frontier: the full Python -pattern sweep is now 164 passed / 0 mismatched. +Semgrep parity. Do not treat this handoff as completion. This handoff records +the clean Python `tests/patterns/python/*.sgrep` frontier and the first promoted +JavaScript `tests/patterns/js/*.sgrep` constant-propagation slice. + +Current headline: + +- Full Python pattern sweep: 164 passed / 0 mismatched. +- Full JavaScript pattern sweep: 54 passed / 78 mismatched, improved from + 41 passed / 91 mismatched by closing the first JS CP slice. +- Broad same-basename upstream sweep: 437 passed / 0 mismatched / 0 Jerboa + errors, with 3 current-Semgrep oracle errors. ## Project Contract @@ -65,7 +73,7 @@ make test Result: ```text -285 tests, 285 passed, 0 failed +286 tests, 286 passed, 0 failed ``` Focused constant/dataflow slice: @@ -124,6 +132,46 @@ git diff --check Result: no output and exit code 0. +Promoted JavaScript constant-propagation oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle +``` + +Result: + +```text +patterns-sweep: 13 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 13 compared +``` + +Full upstream JavaScript pattern sweep: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 54 passed, 78 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +``` + +Broad same-basename upstream sweep: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=20 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared +``` + +The three current-side errors are packaged-Semgrep schema failures for +`anywhere_global`, `anywhere_include`, and `anywhere_metavar`, not Jerboa +mismatches. + ## What Changed The curated `PATTERN_CASE_REGEX` in `Makefile` was expanded from 112 to 116 @@ -140,6 +188,25 @@ The full Python pattern sweep also includes `cp_eval2`; it remains unpromoted in `PATTERN_CASE_REGEX`, but was explicitly regression-checked because the new wildcard string propagation initially overmatched it after `s += argv[1]`. +The latest checkpoint also adds `JS_PATTERN_CASE_REGEX` and a +`patterns-js-oracle` Make target. The promoted JS pattern slice is: + +```text +cp_conditional +cp_dataflow +cp_implicit_conversion +cp_incrdecr +cp_label +cp_throw +cp_undeclared +cp_undeclared1 +infer_const +infer_const_1 +infer_const_regexp +equivalence_constant_propagation2 +equivalence_constant_propagation3 +``` + ## Implementation Map All new scanner logic is in `src/semgrep/scan.ss` and generated into @@ -185,6 +252,29 @@ New scanner fallbacks: Wiring is in the Python fallback chain near the end of `scan.ss`, before the older exact constant-propagation fallbacks. +JavaScript constant-propagation support added in the latest checkpoint: + +- `scan-javascript-cp-pattern-fallbacks` is wired into the JavaScript `pattern` + path before structural matching. +- The JS CP evaluator handles single/double/backtick literals, template + interpolations whose embedded expressions are constant, regexp literals, + numbers, booleans, `null`, `undefined`, `+` concatenation/coercion, and + ternary must-string merges. +- The line-oriented JS environment handles `var`/`let`/`const` declarations, + undeclared assignments, reassignment, increment/decrement invalidation, + augmented assignment invalidation, single-element array destructuring + invalidation, and simple unreachable lines after `throw`/`return` until a + closing block line. +- New JS fallbacks cover wildcard/exact call arguments such as + `console.log("...")`, `require('...')`, `api("...")`, and `new RegExp(/a/)`; + wildcard string subscripts such as `value = obj["..."]`; wildcard/exact + return patterns; `$X == $X` / `$X === $X` with propagated constants; and + direct or propagated bare string-regex patterns such as `"=~/http:///"`. +- The string-regex fallback emits Semgrep-compatible ranges for template + expressions and concatenation prefixes in `infer_const_1`. +- JS CP call/literal scanning ignores line comments so examples in comments, + such as `new RegExp(/a/)`, do not become false positives. + ## Smoke Coverage Added `tests/smoke.ss` extends `scan Python constant propagation pattern fixture @@ -197,8 +287,12 @@ fallbacks` with checks for: - R/L-value propagation from `a = "foo"` into `b = a`, `c[a]`, dict values, and `return a`. -The smoke suite count remains 285 because this work extends an existing test -case rather than adding a new top-level `test-case`. +The latest checkpoint adds a new top-level smoke case, +`scan JavaScript constant propagation pattern fixture fallbacks`, covering +template/concat string calls, conditional string calls, subscript propagation, +throw-unreachable return propagation, and increment invalidation. + +The smoke suite count is now 286. ## Current Frontier @@ -208,20 +302,27 @@ The full Python `.sgrep` pattern fixture sweep is clean: patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` +The full JavaScript `.sgrep` pattern fixture sweep is not clean yet, but the +first JS constant-propagation slice is promoted and green: + +```text +patterns-sweep: 54 passed, 78 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +``` + Do not mark the overall goal complete from this alone. Next high-value work: -1. Decide whether to promote all 164 Python pattern fixtures into +1. Continue the JavaScript pattern frontier. The next concentrated clusters + from the full JS sweep are deep/caching expression matching, import + equivalence variations, JSX/XML range parity, and unordered object/XML + fields. +2. Keep `cp_array_destructure` as a regression guard: the JS equality fallback + must invalidate constants after destructuring reassignment and avoid + matching `imageOrNull == null`. +3. Decide whether to promote all 164 Python pattern fixtures into `PATTERN_CASE_REGEX` or keep `make patterns-oracle` at the curated 116-case runtime. -2. Run a broader upstream same-basename sweep to find the next non-Python or - non-`.sgrep` parity frontier: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh -``` - -3. If the full upstream sweep is too slow, use `MAX_CASES` windows or targeted - `CASE_REGEX` slices and promote verified cases incrementally. +4. Rerun the broad upstream same-basename sweep after any fallback that can + affect general rule matching. ## Operational Notes @@ -241,14 +342,13 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Last Known Broad Guardrail -The earlier handoff recorded a successful broad same-basename upstream sweep: +The latest checkpoint reran the broad same-basename upstream sweep: ```text upstream-sweep: 437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared ``` -That broad upstream sweep was not rerun in this checkpoint. The three recorded -current errors were packaged-Semgrep oracle schema failures for: +The three current errors were packaged-Semgrep oracle schema failures for: ```text anywhere_global @@ -256,5 +356,5 @@ anywhere_include anywhere_metavar ``` -Use the full upstream sweep only when needed; it is slow. The next frontier -should be based on fresh upstream-sweep evidence. +Use the full upstream sweep after broad fallback changes; it is slow but is the +best current guard against rule-level overmatching. --- a/Makefile +++ b/Makefile @@ -5,8 +5,9 @@ SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ +JS_PATTERN_CASE_REGEX ?= ^(cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3)$$ -.PHONY: all build generate test oracle patterns-oracle clean +.PHONY: all build generate test oracle patterns-oracle patterns-js-oracle clean all: build @@ -40,5 +41,14 @@ patterns-oracle: build CASE_REGEX="$(PATTERN_CASE_REGEX)" \ tests/oracle/patterns-sweep.sh +patterns-js-oracle: build + JERBOA_HOME="$(JERBOA_HOME)" \ + JERBOA_TREESITTER_HOME="$(JERBOA_TREESITTER_HOME)" \ + SCHEME="$(SCHEME)" \ + LIBDIRS="$(LIBDIRS)" \ + PATTERN_LANGS=js \ + CASE_REGEX="$(JS_PATTERN_CASE_REGEX)" \ + tests/oracle/patterns-sweep.sh + clean: rm -rf lib --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -11490,6 +11490,1059 @@ (scan-python-return-string-ellipsis-pattern rule path source pattern initial-bindings) (scan-python-list-ellipsis-pattern rule path source pattern initial-bindings))) + (define js-cp-any-string-value + (list 'js-cp-any-string-value)) + (def (js-cp-any-string? value) + (eq? value js-cp-any-string-value)) + (def (js-cp-stringish? value) + (or (string? value) (js-cp-any-string? value))) + (def (js-cp-regexp-value text) + (list 'js-cp-regexp-value text)) + (def (js-cp-regexp? value) + (and (pair? value) (eq? (car value) 'js-cp-regexp-value))) + (def (js-cp-value->string value) + (cond + [(string? value) value] + [(number? value) (number->string value)] + [(eq? value #t) "true"] + [(eq? value #f) "false"] + [(js-cp-any-string? value) js-cp-any-string-value] + [else python-cp-missing-value])) + (def (js-cp-set-binding bindings name value) + (python-cp-set-binding bindings name value)) + (def (js-cp-binding-ref bindings name) + (python-cp-flow-binding-ref bindings name)) + (def (js-cp-remove-binding bindings name) + (python-cp-remove-binding bindings name)) + (def (js-cp-quoted-literal-value text) + (let* ([trimmed (string-trim text)] + [literal (and (> (string-length trimmed) 0) + (parse-code-string-literal-at trimmed 0))]) + (and literal + (= (cdr literal) (string-length trimmed)) + (car literal)))) + (def (js-cp-regexp-literal-value text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (and (>= len 3) + (char=? (string-ref trimmed 0) #\/) + (let loop ([i 1] [escaped? #f] [in-class? #f]) + (cond + [(>= i len) #f] + [escaped? (loop (+ i 1) #f in-class?)] + [(char=? (string-ref trimmed i) #\\) + (loop (+ i 1) #t in-class?)] + [(and (not in-class?) + (char=? (string-ref trimmed i) #\[)) + (loop (+ i 1) #f #t)] + [(and in-class? (char=? (string-ref trimmed i) #\])) + (loop (+ i 1) #f #f)] + [(and (not in-class?) + (char=? (string-ref trimmed i) #\/)) + (let suffix-loop ([j (+ i 1)]) + (cond + [(= j len) + (js-cp-regexp-value (substring trimmed 0 (+ i 1)))] + [(char-alphabetic? (string-ref trimmed j)) + (suffix-loop (+ j 1))] + [else #f]))] + [else (loop (+ i 1) #f in-class?)]))))) + (def (js-cp-template-expression-close + source + open-index + limit) + (let loop ([i (+ open-index 1)] + [depth 1] + [state 'normal] + [escaped? #f]) + (cond + [(>= i limit) #f] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) depth 'double #f)] + [(char=? ch #\') (loop (+ i 1) depth 'single #f)] + [(char=? ch #\`) (loop (+ i 1) depth 'backtick #f)] + [(char=? ch #\{) (loop (+ i 1) (+ depth 1) state #f)] + [(char=? ch #\}) + (if (= depth 1) i (loop (+ i 1) (- depth 1) state #f))] + [else (loop (+ i 1) depth state #f)]))] + [escaped? (loop (+ i 1) depth state #f)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) depth state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) depth 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) depth 'normal #f)] + [(and (eq? state 'backtick) + (char=? (string-ref source i) #\`)) + (loop (+ i 1) depth 'normal #f)] + [else (loop (+ i 1) depth state #f)]))) + (def (js-cp-template-value + text + bindings + source + before-offset) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (and (>= len 2) + (char=? (string-ref trimmed 0) #\`) + (let loop ([i 1] [part-start 1] [parts '()] [any? #f]) + (cond + [(>= i len) python-cp-missing-value] + [(char=? (string-ref trimmed i) #\\) + (loop (min len (+ i 2)) part-start parts any?)] + [(char=? (string-ref trimmed i) #\`) + (and (= i (- len 1)) + (if any? + js-cp-any-string-value + (join-strings + (reverse + (cons + (substring trimmed part-start i) + parts)))))] + [(and (< (+ i 1) len) + (char=? (string-ref trimmed i) #\$) + (char=? (string-ref trimmed (+ i 1)) #\{)) + (let* ([open (+ i 1)] + [close (js-cp-template-expression-close + trimmed + open + len)] + [expr (and close + (substring trimmed (+ open 1) close))] + [value (and expr + (js-cp-expression-value + expr + bindings + source + before-offset))] + [string-value (and value + (js-cp-value->string value))]) + (cond + [(not close) python-cp-missing-value] + [(or (not string-value) + (python-cp-missing? string-value)) + python-cp-missing-value] + [(js-cp-any-string? string-value) + (loop + (+ close 1) + (+ close 1) + (cons (substring trimmed part-start i) parts) + #t)] + [else + (loop + (+ close 1) + (+ close 1) + (cons + string-value + (cons (substring trimmed part-start i) parts)) + any?)]))] + [else (loop (+ i 1) part-start parts any?)]))))) + (def (js-cp-top-level-conditional source) + (let ([len (string-length source)]) + (let loop ([i 0] + [depth 0] + [state 'normal] + [escaped? #f] + [question #f]) + (cond + [(>= i len) #f] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") + (loop (+ i 1) depth 'double #f question)] + [(char=? ch #\') + (loop (+ i 1) depth 'single #f question)] + [(char=? ch #\`) + (loop (+ i 1) depth 'backtick #f question)] + [(or (char=? ch #\() (char=? ch #\[) (char=? ch #\{)) + (loop (+ i 1) (+ depth 1) state #f question)] + [(or (char=? ch #\)) (char=? ch #\]) (char=? ch #\})) + (loop (+ i 1) (max 0 (- depth 1)) state #f question)] + [(and (= depth 0) (char=? ch #\?)) + (loop (+ i 1) depth state #f i)] + [(and question (= depth 0) (char=? ch #\:)) + (cons question i)] + [else (loop (+ i 1) depth state #f question)]))] + [escaped? (loop (+ i 1) depth state #f question)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) depth state #t question)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) depth 'normal #f question)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) depth 'normal #f question)] + [(and (eq? state 'backtick) + (char=? (string-ref source i) #\`)) + (loop (+ i 1) depth 'normal #f question)] + [else (loop (+ i 1) depth state #f question)])))) + (def (js-cp-merge-values left right) + (cond + [(or (python-cp-missing? left) (python-cp-missing? right)) + python-cp-missing-value] + [(equal? left right) left] + [(and (js-cp-stringish? left) (js-cp-stringish? right)) + js-cp-any-string-value] + [else python-cp-missing-value])) + (def (js-cp-expression-value + expr + bindings + source + before-offset) + (let ([trimmed (string-trim expr)]) + (cond + [(string=? trimmed "") python-cp-missing-value] + [(outer-pair? trimmed #\( #\)) + (js-cp-expression-value + (substring trimmed 1 (- (string-length trimmed) 1)) + bindings + source + before-offset)] + [(and (>= (string-length trimmed) 2) + (char=? (string-ref trimmed 0) #\`) + (char=? + (string-ref trimmed (- (string-length trimmed) 1)) + #\`)) + (js-cp-template-value + trimmed + bindings + source + before-offset)] + [(js-cp-quoted-literal-value trimmed) => values] + [(js-cp-regexp-literal-value trimmed) => values] + [(or (string=? trimmed "true") (string=? trimmed "True")) + #t] + [(or (string=? trimmed "false") (string=? trimmed "False")) + #f] + [(string=? trimmed "null") "null"] + [(string=? trimmed "undefined") "undefined"] + [(parse-number-literal trimmed #f) => values] + [(js-cp-top-level-conditional trimmed) => + (lambda (match) + (let* ([question (car match)] + [colon (cdr match)] + [left (js-cp-expression-value + (substring-trim trimmed (+ question 1) colon) + bindings + source + before-offset)] + [right (js-cp-expression-value + (substring-trim + trimmed + (+ colon 1) + (string-length trimmed)) + bindings + source + before-offset)]) + (js-cp-merge-values left right)))] + [(find-top-level-binary-operator trimmed '("+")) => + (lambda (match) + (let* ([index (cdr match)] + [left (js-cp-expression-value + (substring-trim trimmed 0 index) + bindings + source + before-offset)] + [right (js-cp-expression-value + (substring-trim + trimmed + (+ index 1) + (string-length trimmed)) + bindings + source + before-offset)]) + (cond + [(or (python-cp-missing? left) + (python-cp-missing? right)) + python-cp-missing-value] + [(or (js-cp-stringish? left) (js-cp-stringish? right)) + (let ([left-string (js-cp-value->string left)] + [right-string (js-cp-value->string right)]) + (cond + [(or (python-cp-missing? left-string) + (python-cp-missing? right-string)) + python-cp-missing-value] + [(or (js-cp-any-string? left-string) + (js-cp-any-string? right-string)) + js-cp-any-string-value] + [else (string-append left-string right-string)]))] + [(and (number? left) (number? right)) (+ left right)] + [else python-cp-missing-value])))] + [(and (identifier-start-char? (string-ref trimmed 0)) + (let loop ([i 1]) + (or (= i (string-length trimmed)) + (and (identifier-char? (string-ref trimmed i)) + (loop (+ i 1)))))) + (js-cp-binding-ref bindings trimmed)] + [else python-cp-missing-value]))) + (def (js-cp-trim-comment text) + (let ([len (string-length text)]) + (let loop ([i 0] [state 'normal] [escaped? #f]) + (cond + [(>= i len) (string-trim text)] + [(eq? state 'normal) + (let ([ch (string-ref text i)]) + (cond + [(char=? ch #\") (loop (+ i 1) 'double #f)] + [(char=? ch #\') (loop (+ i 1) 'single #f)] + [(char=? ch #\`) (loop (+ i 1) 'backtick #f)] + [(and (< (+ i 1) len) + (char=? ch #\/) + (char=? (string-ref text (+ i 1)) #\/)) + (string-trim (substring text 0 i))] + [else (loop (+ i 1) state #f)]))] + [escaped? (loop (+ i 1) state #f)] + [(char=? (string-ref text i) #\\) (loop (+ i 1) state #t)] + [(and (eq? state 'double) (char=? (string-ref text i) #\")) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'single) (char=? (string-ref text i) #\')) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'backtick) + (char=? (string-ref text i) #\`)) + (loop (+ i 1) 'normal #f)] + [else (loop (+ i 1) state #f)])))) + (def (js-cp-strip-trailing-semicolon text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (if (and (> len 0) + (char=? (string-ref trimmed (- len 1)) #\;)) + (string-trim (substring trimmed 0 (- len 1))) + trimmed))) + (def (js-cp-line-assignment-info source line-start line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (js-cp-trim-comment + (substring source first line-end))] + [line (js-cp-strip-trailing-semicolon line)] + [match (re-search + (re "^(?:var|let|const)?[ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*=[ \\t]*(.+)$") + line + 0)]) + (and match + (not (string-find-substring line "==")) + (let* ([name (re-match-group match 1)] + [rhs (string-trim (re-match-group match 2))] + [rhs-rel (string-find-substring-from line rhs 0)] + [name-rel (string-find-substring-from line name 0)]) + (and rhs-rel + name-rel + (list (cons 'name name) (cons 'rhs rhs) + (cons 'first first) + (cons 'name-start (+ first name-rel)) + (cons 'rhs-start (+ first rhs-rel)) + (cons + 'rhs-end + (+ first rhs-rel (string-length rhs))))))))) + (def (js-cp-line-invalidated-name + source + line-start + line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (js-cp-trim-comment + (substring source first line-end))] + [match (or (re-search + (re "^([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*(?:\\+\\+|--)") + line + 0) + (re-search + (re "^(?:\\+\\+|--)[ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)") + line + 0) + (re-search + (re "^\\[[ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\][ \\t]*=") + line + 0) + (re-search + (re "^([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*[+\\-*/%]=") + line + 0))]) + (and match (re-match-group match 1)))) + (def (js-cp-line-opens-unreachable? + source + line-start + line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (js-cp-trim-comment + (substring source first line-end))]) + (or (sg-string-prefix? "throw " (string-trim line)) + (sg-string-prefix? "return " (string-trim line))))) + (def (js-cp-line-closes-block? source line-start line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (string-trim (substring source first line-end))]) + (and (> (string-length line) 0) + (char=? (string-ref line 0) #\})))) + (def (js-cp-line-comment-before? source index) + (let* ([line-start (line-start-before source index)] + [limit (min index (string-length source))]) + (let loop ([i line-start] [state 'normal] [escaped? #f]) + (cond + [(>= i limit) #f] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) 'double #f)] + [(char=? ch #\') (loop (+ i 1) 'single #f)] + [(char=? ch #\`) (loop (+ i 1) 'backtick #f)] + [(and (< (+ i 1) limit) + (char=? ch #\/) + (char=? (string-ref source (+ i 1)) #\/)) + #t] + [else (loop (+ i 1) state #f)]))] + [escaped? (loop (+ i 1) state #f)] + [(char=? (string-ref source i) #\\) (loop (+ i 1) state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'backtick) + (char=? (string-ref source i) #\`)) + (loop (+ i 1) 'normal #f)] + [else (loop (+ i 1) state #f)])))) + (def (js-cp-env-before source before-offset) + (let ([limit (min before-offset (string-length source))]) + (let loop ([line-start 0] [env '()] [unreachable? #f]) + (if (>= line-start limit) + env + (let* ([line-end (min (line-end-after source line-start) + limit)] + [next (if (< line-end (string-length source)) + (+ line-end 1) + (+ (string-length source) 1))] + [closed? (js-cp-line-closes-block? + source + line-start + line-end)] + [skip? (and unreachable? (not closed?))] + [invalidated (and (not skip?) + (js-cp-line-invalidated-name + source + line-start + line-end))] + [assignment (and (not skip?) + (not invalidated) + (js-cp-line-assignment-info + source + line-start + line-end))] + [next-env (cond + [skip? env] + [invalidated + (js-cp-remove-binding env invalidated)] + [assignment + (let* ([name (alist-ref/default + assignment + 'name + "")] + [rhs (alist-ref/default + assignment + 'rhs + "")] + [value (js-cp-expression-value + rhs + env + source + (alist-ref/default + assignment + 'rhs-start + line-start))]) + (if (python-cp-missing? value) + (js-cp-remove-binding env name) + (js-cp-set-binding + env + name + value)))] + [else env])] + [next-unreachable? (or (and unreachable? + (not closed?)) + (and (not skip?) + (js-cp-line-opens-unreachable? + source + line-start + line-end)))]) + (loop next next-env next-unreachable?)))))) + (def (js-cp-call-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [match (re-search + (re "^(new[ \\t]+RegExp|[A-Za-z_$][A-Za-z0-9_$.]*)[ \\t\\r\\n]*\\(") + trimmed + 0)]) + (and match + (let* ([function-name (re-match-group match 1)] + [open (- (re-match-end match) 1)] + [close (find-matching-close-paren trimmed open)] + [args (and close + (split-top-level-commas + (substring + trimmed + (+ open 1) + (- close 1))))]) + (and close + args + (null? (cdr args)) + (= close (string-length trimmed)) + (let* ([arg (car args)] + [arg-value (js-cp-expression-value + arg + '() + trimmed + 0)] + [wildcard? (or (string=? + (string-trim arg) + "\"...\"") + (string=? + (string-trim arg) + "'...'") + (string=? + (string-trim arg) + "`...`"))]) + (and (or wildcard? + (not (python-cp-missing? arg-value))) + (list + (cons 'function function-name) + (cons 'wildcard wildcard?) + (cons 'expected arg-value))))))))) + (def (js-cp-call-needle function-name) + (if (string=? function-name "new RegExp") + "new RegExp(" + (string-append function-name "("))) + (def (js-cp-value-matches-expected? + value + expected + wildcard?) + (cond + [wildcard? (js-cp-stringish? value)] + [(js-cp-any-string? value) #f] + [else (equal? value expected)])) + (def (scan-javascript-cp-call-pattern rule path source + pattern initial-bindings) + (let ([spec (js-cp-call-pattern-spec pattern)]) + (and spec + (let* ([function-name (alist-ref/default spec 'function "")] + [needle (js-cp-call-needle function-name)] + [expected (alist-ref/default + spec + 'expected + python-cp-missing-value)] + [wildcard? (alist-ref/default spec 'wildcard #f)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from + source + needle + start)]) + (if (not index) + (nonempty-findings (reverse acc)) + (let* ([open (+ index + (- (string-length needle) 1))] + [close (find-matching-close-paren + source + open)] + [args (and close + (split-top-level-commas + (substring + source + (+ open 1) + (- close 1))))] + [arg (and args + (null? (cdr args)) + (car args))] + [value (and arg + (js-cp-expression-value + arg + (js-cp-env-before + source + index) + source + index))] + [finding (and close + value + (js-cp-value-matches-expected? + value + expected + wildcard?) + (not (js-cp-line-comment-before? + source + index)) + (identifier-boundary-before? + source + index) + (finding-for-range-with-bindings rule path source index close + initial-bindings))] + [next (if close + (max (+ index 1) close) + (+ index 1))]) + (loop + next + (if finding (cons finding acc) acc)))))))))) + (def (js-cp-subscript-string-pattern? pattern) + (let* ([trimmed (string-trim pattern)] + [open (string-find-substring trimmed "[")] + [close (and open + (find-matching-close-bracket trimmed open))]) + (and open + close + (let ([inside (string-trim + (substring trimmed (+ open 1) (- close 1)))]) + (or (string=? inside "\"...\"") + (string=? inside "'...'") + (string=? inside "`...`")))))) + (def (js-cp-statement-end source line-start line-end) + (let loop ([i line-start] [state 'normal] [escaped? #f]) + (cond + [(>= i line-end) line-end] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) 'double #f)] + [(char=? ch #\') (loop (+ i 1) 'single #f)] + [(char=? ch #\`) (loop (+ i 1) 'backtick #f)] + [(char=? ch #\;) i] + [else (loop (+ i 1) state #f)]))] + [escaped? (loop (+ i 1) state #f)] + [(char=? (string-ref source i) #\\) (loop (+ i 1) state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'backtick) + (char=? (string-ref source i) #\`)) + (loop (+ i 1) 'normal #f)] + [else (loop (+ i 1) state #f)]))) + (def (scan-javascript-cp-subscript-pattern rule path source pattern initial-bindings) + (and (js-cp-subscript-string-pattern? pattern) + (let ([len (string-length source)]) + (let line-loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [stmt-end (js-cp-statement-end + source + first + line-end)] + [line (substring source first stmt-end)] + [open-rel (string-find-substring line "[")] + [open (and open-rel (+ first open-rel))] + [close (and open + (find-matching-close-bracket + source + open))] + [expr (and close + (<= close stmt-end) + (substring + source + (+ open 1) + (- close 1)))] + [value (and expr + (js-cp-expression-value + expr + (js-cp-env-before source open) + source + open))] + [finding (and value + (js-cp-stringish? value) + (finding-for-range-with-bindings rule path source first stmt-end + initial-bindings))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (line-loop + next + (if finding (cons finding acc) acc)))))))) + (def (js-cp-return-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] [prefix "return "]) + (and (sg-string-prefix? prefix trimmed) + (let* ([expr (substring-trim + trimmed + (string-length prefix) + (string-length trimmed))] + [wildcard? (or (string=? expr "\"...\"") + (string=? expr "'...'") + (string=? expr "`...`"))] + [value (and (not wildcard?) + (js-cp-expression-value + expr + '() + trimmed + 0))]) + (and (or wildcard? (string? value)) + (list + (cons 'wildcard wildcard?) + (cons 'expected value))))))) + (def (scan-javascript-cp-return-pattern rule path source + pattern initial-bindings) + (let ([spec (js-cp-return-pattern-spec pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [stmt-end (js-cp-statement-end + source + first + line-end)] + [line (substring source first stmt-end)] + [expr (and (sg-string-prefix? "return " line) + (string-trim + (substring + line + (string-length "return ") + (string-length line))))] + [value (and expr + (js-cp-expression-value + expr + (js-cp-env-before source first) + source + first))] + [wildcard? (alist-ref/default + spec + 'wildcard + #f)] + [expected (alist-ref/default + spec + 'expected + "")] + [return-end (if (and (< stmt-end line-end) + (char=? + (string-ref + source + stmt-end) + #\;)) + (+ stmt-end 1)