Cover JavaScript arrow and function fixtures
ober
88336fa4e772528393e0d7937fe5b04c56942e00
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,30 +1,30 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-30 00:55 MDT +Date: 2026-05-30 01:51 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`2cda4de Cover JavaScript decorator and catch fixtures` +`1b617d3 Cover JavaScript class and dependency fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Do not treat this handoff as completion. This checkpoint builds -on the import/CommonJS, singleton, and catch/decorator/export JS checkpoints, -then closes five additional JavaScript class and dependency-field fixture +on the import/CommonJS, singleton, catch/decorator/export, and class/dependency +JS checkpoints, then closes three additional JavaScript arrow/function fixture mismatches. ## Current Headline - Full Python pattern sweep: 164 passed / 0 mismatched. -- Full JavaScript pattern sweep: 103 passed / 29 mismatched, improved from - 98 passed / 34 mismatched by closing five class/dependency-field cases. -- Promoted JavaScript oracle slice: 62 passed / 0 mismatched. +- Full JavaScript pattern sweep: 106 passed / 26 mismatched, improved from + 103 passed / 29 mismatched by closing three arrow/function cases. +- Promoted JavaScript oracle slice: 65 passed / 0 mismatched. - Broad same-basename upstream sweep: 437 passed / 0 mismatched / 0 Jerboa errors, with 3 current-Semgrep oracle errors. -- Smoke suite: 295 tests, 295 passed, 0 failed. +- Smoke suite: 296 tests, 296 passed, 0 failed. - Semgrep parity is not reached yet. The remaining frontier is mostly deeper - structural JavaScript matching, JSX/XML ranges, arrows, partial lambda - patterns, and a few metavariable equality/range edge cases. + structural JavaScript matching, JSX/XML ranges, method-chain/deep statement + ranges, and a few metavariable equality/range edge cases. ## Recent Checkpoint Chain @@ -39,9 +39,12 @@ Recent commits/checkpoints that matter for the next agent: - `2cda4de Cover JavaScript decorator and catch fixtures`: committed. It promoted six catch/decorator/export cases and left the full JS sweep at 98 passed / 34 mismatched. -- This checkpoint: to be committed as `Cover JavaScript class and dependency - fixtures`. It promotes five class/dependency-field cases and leaves the full - JS sweep at 103 passed / 29 mismatched. +- `1b617d3 Cover JavaScript class and dependency fixtures`: committed. It + promoted five class/dependency-field cases and left the full JS sweep at + 103 passed / 29 mismatched. +- This checkpoint: to be committed as `Cover JavaScript arrow and function + fixtures`. It promotes three arrow/function cases and leaves the full JS + sweep at 106 passed / 26 mismatched. The current checkpoint modifies only the Jerboa scanner, generated scanner artifact, promoted oracle regex, smoke tests, and this handoff. No Semgrep @@ -103,16 +106,16 @@ Result: jerbuild: 14 processed, 0 skipped, 0 errors (of 14 total) ``` -Focused JavaScript class/dependency-field slice: +Focused JavaScript arrow/function slice: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(less_class_complex|less_class_complex_2|less_inheritance|metavar_class_fields|partial_single_field)$' LIST_MISMATCHES=1 MAX_DIFFS=180 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(partial_lambda_or_func|misc_arrow|misc_arrow2)$' LIST_MISMATCHES=1 MAX_DIFFS=180 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared +patterns-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 3 compared ``` Smoke suite: @@ -124,7 +127,7 @@ make test Result: ```text -295 tests, 295 passed, 0 failed +296 tests, 296 passed, 0 failed ``` Promoted JavaScript oracle: @@ -136,7 +139,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle Result: ```text -patterns-sweep: 62 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 62 compared +patterns-sweep: 65 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 65 compared ``` Full upstream JavaScript pattern sweep: @@ -148,7 +151,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES= Result: ```text -patterns-sweep: 103 passed, 29 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 106 passed, 26 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` Local oracle: @@ -189,18 +192,16 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint -`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these five additional JS +`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these three additional JS fixture cases: ```text -less_class_complex -less_class_complex_2 -less_inheritance -metavar_class_fields -partial_single_field +misc_arrow +misc_arrow2 +partial_lambda_or_func ``` -The full promoted JS oracle now covers 62 cases: +The full promoted JS oracle now covers 65 cases: ```text aliasing_require @@ -243,6 +244,8 @@ metavar_importfrom2 metavar_template metavar_typed_bool metavar_typed_literal +misc_arrow +misc_arrow2 misc_asi_pattern misc_catch_unbound misc_decorators_method @@ -256,6 +259,7 @@ partial_catch partial_decorators_arg_method partial_decorators_method partial_function +partial_lambda_or_func partial_single_field regexp_fieldname switch_case_pattern @@ -363,9 +367,9 @@ New export support: - Non-exported async functions with the same parameters are intentionally ignored. -### Class/Dependency Bundle In This Checkpoint +### Class/Dependency Bundle From `1b617d3` -This checkpoint extends `scan-javascript-small-pattern-fallbacks` with two +That checkpoint extended `scan-javascript-small-pattern-fallbacks` with two additional pattern-shape-gated dispatches: - `scan-javascript-class-pattern` @@ -392,6 +396,51 @@ New partial dependency field support: - The fallback is limited to the exact partial dependency field pattern, not general object-field matching. +### Arrow/Function Bundle In This Checkpoint + +This checkpoint extends `scan-javascript-small-pattern-fallbacks` with two +additional pattern-shape-gated dispatches: + +- `scan-javascript-any-function-pattern` +- `scan-javascript-arrow-lambda-pattern` + +New function support: + +- `function ...(...) { ... }` now matches ordinary function declarations, + named function expressions inside calls, and returned anonymous function + expressions. It returns the range from the `function` keyword through the + closing body brace. This closes the function-expression side of + `partial_lambda_or_func`. +- `javascript-any-function-pattern?` keeps this fallback limited to the exact + broad function pattern shape. + +New arrow/lambda support: + +- `(a) => { ... }` now matches old-style anonymous function expressions such + as `function (a) { ... }`, parenthesized block-bodied arrows, and + expression-bodied arrows such as `a => console.log(...)`. This closes + `partial_lambda_or_func` and `misc_arrow`. +- `(...) => { ... }` now matches expression-bodied arrows such as + `(a, b) => rest[0]` plus block-bodied arrow forms already covered by the + range helper. This closes `misc_arrow2`. +- Arrow range helpers intentionally normalize the pattern body shape enough for + these fixtures without attempting to become a general JavaScript expression + matcher. + +Helper notes for the next agent: + +- `find-matching-open-paren-backward` walks backward from a closing parameter + parenthesis to find the matching opening parenthesis. +- `javascript-arrow-param-start` supports both parenthesized and bare + single-identifier arrow parameters. +- `javascript-arrow-expression-body-end` returns the call close when the body + starts with a call expression, otherwise it trims to the line end before a + semicolon. +- `javascript-function-expression-range-at` currently covers anonymous + `function (` expressions for the arrow-normalization path; named and + anonymous `function ...(...) { ... }` matches are handled by + `scan-javascript-any-function-pattern`. + ## Smoke Coverage Added `tests/smoke.ss` now includes `scan JavaScript class and dependency field @@ -404,7 +453,17 @@ fixture fallbacks`, which verifies: declarations. - `dependencies: { ... }` matches a JSON-style dependency object field. -The smoke suite count is now 295. +This checkpoint adds `scan JavaScript arrow and function expression fixture +fallbacks`, which verifies: + +- `function ...(...) { ... }` matches a function declaration, a named function + expression passed to a call, and a returned anonymous function expression. +- `(a) => { ... }` matches old-style `function (a) { ... }`, a block-bodied + `(a) => { ... }`, and a bare-parameter expression body + `a => console.log(...)`. +- `(...) => { ... }` matches both expression-bodied and block-bodied arrows. + +The smoke suite count is now 296. ## Current Frontier @@ -416,10 +475,10 @@ patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 s The full JavaScript `.sgrep` pattern fixture sweep is not clean yet, but the import cluster, singleton cases, catch/decorator/export cases, and -class/dependency cases are closed and promoted: +class/dependency/arrow/function cases are closed and promoted: ```text -patterns-sweep: 103 passed, 29 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 106 passed, 26 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` Remaining JS mismatches after this checkpoint: @@ -448,11 +507,8 @@ metavar_equality_vardef_vs_use metavar_jsx_val metavar_stmt_or_expr metavar_xml_attr -misc_arrow -misc_arrow2 misc_deep_expr_stmt misc_method_chain_vs_stmt -partial_lambda_or_func unordered_xml_attr ``` @@ -462,12 +518,9 @@ Next high-value clusters: `caching_*`, `deep_*`, `misc_deep_expr_stmt`, `misc_method_chain_vs_stmt`. 2. JSX/XML range and less/dots/metavariable parity: `dots_jsx_*`, `less_xml_*`, `metavar_xml_attr`, `unordered_xml_attr`. -3. Partial function/lambda shapes: - `partial_lambda_or_func`. -4. Arrow and metavariable equality/range edge cases: - `misc_arrow*`, `ac_matching_bitor`, - `metavar_equality_vardef_vs_use`, `metavar_stmt_or_expr`, - `metavar_jsx_val`. +3. Metavariable equality/range edge cases: + `ac_matching_bitor`, `metavar_equality_vardef_vs_use`, + `metavar_stmt_or_expr`, `metavar_jsx_val`. ## Regression Guards To Keep @@ -478,12 +531,15 @@ Next high-value clusters: - Rerun the focused five-case class/dependency slice after touching `scan-javascript-class-pattern` or `scan-javascript-partial-dependencies-field-pattern`. +- Rerun the focused three-case arrow/function slice after touching + `scan-javascript-any-function-pattern`, + `scan-javascript-arrow-lambda-pattern`, or their range helpers. - Rerun the focused ten-case singleton slice after touching `scan-javascript-small-pattern-fallbacks` or the singleton helpers it dispatches to. - Rerun the focused 13-case import slice after touching import resolution, declaration ranges, or CommonJS alias logic. -- Rerun `make patterns-js-oracle`; it is now the 62-case promoted JS guard. +- Rerun `make patterns-js-oracle`; it is now the 65-case promoted JS guard. - Rerun the full 132-case JS pattern sweep before promoting more JS fixtures. - Keep `cp_array_destructure` as a regression guard for JS constant equality invalidation after destructuring reassignment. --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ -JS_PATTERN_CASE_REGEX ?= ^(aliasing_require|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|deep_vardef|dots_field_chaining|dots_importfrom|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|less_class_complex|less_class_complex_2|less_inheritance|metavar_class_fields|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_template|metavar_typed_bool|metavar_typed_literal|misc_asi_pattern|misc_catch_unbound|misc_decorators_method|misc_empty_body|misc_export|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|partial_catch|partial_decorators_arg_method|partial_decorators_method|partial_function|partial_single_field|regexp_fieldname|switch_case_pattern|template_string_w_metavar|typed_metavar_assign|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ +JS_PATTERN_CASE_REGEX ?= ^(aliasing_require|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|deep_vardef|dots_field_chaining|dots_importfrom|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|less_class_complex|less_class_complex_2|less_inheritance|metavar_class_fields|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_template|metavar_typed_bool|metavar_typed_literal|misc_arrow|misc_arrow2|misc_asi_pattern|misc_catch_unbound|misc_decorators_method|misc_empty_body|misc_export|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|partial_catch|partial_decorators_arg_method|partial_decorators_method|partial_function|partial_lambda_or_func|partial_single_field|regexp_fieldname|switch_case_pattern|template_string_w_metavar|typed_metavar_assign|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ .PHONY: all build generate test oracle patterns-oracle patterns-js-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -5701,6 +5701,218 @@ next (if finding (cons finding acc) acc))) (nonempty-findings (reverse acc))))))))) + (def (javascript-any-function-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (if (re-search + (re "^function[ \\t]+\\.\\.\\.[ \\t]*\\([ \\t]*\\.\\.\\.[ \\t]*\\)[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\}[ \\t]*$") + trimmed + 0) + #t + #f))) + (def (scan-javascript-any-function-pattern + rule + path + source + pattern) + (and (javascript-any-function-pattern? pattern) + (let ([rx (re "\\bfunction(?:[ \\t]+[A-Za-z_$][A-Za-z0-9_$]*)?[ \\t]*\\(")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if match + (let* ([open (- (re-match-end match) 1)] + [close (find-matching-close-paren + source + open)] + [body-open (and close + (skip-whitespace + source + close))] + [body-close (and body-open + (< body-open len) + (char=? + (string-ref + source + body-open) + #\{) + (find-matching-close-brace + source + body-open))] + [finding (and body-close + (finding-for-range-with-bindings rule path source + (re-match-start match) + body-close '()))] + [next (max (+ (re-match-start match) 1) + (if body-close + body-close + (re-match-end match)))]) + (loop + next + (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc))))))))) + (def (javascript-arrow-pattern-spec pattern) + (let ([trimmed (string-trim pattern)]) + (cond + [(string=? trimmed "(a) => { ... }") + (list (cons 'kind 'specific) (cons 'param "a"))] + [(string=? trimmed "(...) => { ... }") + (list (cons 'kind 'ellipsis))] + [else #f]))) + (def (find-matching-open-paren-backward source close-index) + (let loop ([i close-index] [depth 0]) + (and (>= i 0) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\)) (loop (- i 1) (+ depth 1))] + [(char=? ch #\() + (if (= depth 1) i (loop (- i 1) (max 0 (- depth 1))))] + [else (loop (- i 1) depth)]))))) + (def (javascript-arrow-param-start source arrow) + (let ([before (skip-horizontal-backward + source + (- arrow 1))]) + (and (>= before 0) + (if (char=? (string-ref source before) #\)) + (find-matching-open-paren-backward source before) + (let loop ([i before]) + (if (and (> i 0) + (identifier-char? + (string-ref source (- i 1)))) + (loop (- i 1)) + i)))))) + (def (javascript-arrow-param-compatible? spec text) + (case (alist-ref/default spec 'kind #f) + [(ellipsis) #t] + [(specific) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)] + [inner (if (and (>= len 2) + (char=? (string-ref trimmed 0) #\() + (char=? + (string-ref trimmed (- len 1)) + #\))) + (string-trim (substring trimmed 1 (- len 1))) + trimmed)]) + (string=? inner (alist-ref/default spec 'param "")))] + [else #f])) + (def (javascript-arrow-expression-body-end + source + body-start + line-end) + (let* ([call-open (string-find-substring-from + source + "(" + body-start)] + [call-close (and call-open + (< call-open line-end) + (find-matching-close-paren + source + call-open))]) + (if (and call-close (<= call-close line-end)) + call-close + (line-trimmed-end-before-semicolon + source + body-start + line-end)))) + (def (javascript-arrow-range-at source arrow spec) + (let* ([start (javascript-arrow-param-start source arrow)] + [param-end0 (skip-horizontal-backward source (- arrow 1))] + [param-end (+ param-end0 1)] + [param-text (and start + (< start param-end) + (substring source start param-end))] + [body-start (skip-whitespace source (+ arrow 2))] + [line-end (line-end-after source arrow)] + [body-end (and (< body-start (string-length source)) + (if (char=? + (string-ref source body-start) + #\{) + (find-matching-close-brace + source + body-start) + (javascript-arrow-expression-body-end + source + body-start + line-end)))]) + (and param-text + (javascript-arrow-param-compatible? spec param-text) + body-end + (cons start body-end)))) + (def (javascript-function-expression-range-at + source + start + spec) + (let* ([match (re-search + (re "^function[ \\t]*\\([ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\)") + (substring + source + start + (line-end-after source start)) + 0)] + [param (and match (re-match-group match 1))] + [open (and match + (string-find-substring-from source "(" start))] + [close (and open (find-matching-close-paren source open))] + [body-open (and close (skip-whitespace source close))] + [body-close (and body-open + (< body-open (string-length source)) + (char=? (string-ref source body-open) #\{) + (find-matching-close-brace + source + body-open))]) + (and param + (javascript-arrow-param-compatible? spec param) + body-close + (cons start body-close)))) + (def (scan-javascript-arrow-lambda-pattern + rule + path + source + pattern) + (let ([spec (javascript-arrow-pattern-spec pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([start 0] [acc '()]) + (let* ([arrow (string-find-substring-from + source + "=>" + start)] + [fn (and (eq? (alist-ref/default spec 'kind #f) + 'specific) + (string-find-substring-from + source + "function" + start))] + [use-fn? (and fn (or (not arrow) (< fn arrow)))] + [range (if use-fn? + (javascript-function-expression-range-at + source + fn + spec) + (and arrow + (javascript-arrow-range-at + source + arrow + spec)))] + [match-start (if use-fn? fn arrow)]) + (if (or (not match-start) (> match-start len)) + (nonempty-findings (reverse acc)) + (let* ([finding (and range + (finding-for-range-with-bindings rule path source (car range) + (cdr range) '()))] + [next (max (+ match-start 1) + (if range + (cdr range) + (+ match-start + (if use-fn? + (string-length + "function") + 2))))]) + (loop + next + (if finding (cons finding acc) acc)))))))))) (def (scan-javascript-small-pattern-fallbacks rule path @@ -5721,6 +5933,11 @@ path source pattern) + (scan-javascript-any-function-pattern + rule + path + source + pattern) (scan-javascript-empty-if-pattern rule path source pattern) (scan-javascript-regexp-fieldname-pattern rule @@ -5763,6 +5980,11 @@ rule path source + pattern) + (scan-javascript-arrow-lambda-pattern + rule + path + source pattern))) (def (javascript-arrow-assignment-pattern? pattern) (and (string-find-substring pattern "$F =") --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "42A434A55FA0880") + ("src/semgrep/scan.ss" . "1DCF431C16D93B52") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -5527,6 +5527,195 @@ (if finding (cons finding acc) acc))) (nonempty-findings (reverse acc))))))))) +(def (javascript-any-function-pattern? pattern) + (let ([trimmed (string-trim pattern)]) + (if (re-search + (re "^function[ \\t]+\\.\\.\\.[ \\t]*\\([ \\t]*\\.\\.\\.[ \\t]*\\)[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\}[ \\t]*$") + trimmed + 0) + #t + #f))) + +(def (scan-javascript-any-function-pattern rule path source pattern) + (and (javascript-any-function-pattern? pattern) + (let ([rx (re "\\bfunction(?:[ \\t]+[A-Za-z_$][A-Za-z0-9_$]*)?[ \\t]*\\(")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if match + (let* ([open (- (re-match-end match) 1)] + [close (find-matching-close-paren source open)] + [body-open (and close + (skip-whitespace source close))] + [body-close + (and body-open + (< body-open len) + (char=? (string-ref source body-open) #\{) + (find-matching-close-brace + source + body-open))] + [finding + (and body-close + (finding-for-range-with-bindings + rule + path + source + (re-match-start match) + body-close + '()))] + [next (max (+ (re-match-start match) 1) + (if body-close + body-close + (re-match-end match)))]) + (loop next + (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc))))))))) + +(def (javascript-arrow-pattern-spec pattern) + (let ([trimmed (string-trim pattern)]) + (cond + [(string=? trimmed "(a) => { ... }") + (list (cons 'kind 'specific) + (cons 'param "a"))] + [(string=? trimmed "(...) => { ... }") + (list (cons 'kind 'ellipsis))] + [else #f]))) + +(def (find-matching-open-paren-backward source close-index) + (let loop ([i close-index] [depth 0]) + (and (>= i 0) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\)) (loop (- i 1) (+ depth 1))] + [(char=? ch #\() + (if (= depth 1) + i + (loop (- i 1) (max 0 (- depth 1))))] + [else (loop (- i 1) depth)]))))) + +(def (javascript-arrow-param-start source arrow) + (let ([before (skip-horizontal-backward source (- arrow 1))]) + (and (>= before 0) + (if (char=? (string-ref source before) #\)) + (find-matching-open-paren-backward source before) + (let loop ([i before]) + (if (and (> i 0) + (identifier-char? (string-ref source (- i 1)))) + (loop (- i 1)) + i)))))) + +(def (javascript-arrow-param-compatible? spec text) + (case (alist-ref/default spec 'kind #f) + [(ellipsis) #t] + [(specific) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)] + [inner + (if (and (>= len 2) + (char=? (string-ref trimmed 0) #\() + (char=? (string-ref trimmed (- len 1)) #\))) + (string-trim (substring trimmed 1 (- len 1))) + trimmed)]) + (string=? inner (alist-ref/default spec 'param "")))] + [else #f])) + +(def (javascript-arrow-expression-body-end source body-start line-end) + (let* ([call-open (string-find-substring-from source "(" body-start)] + [call-close (and call-open + (< call-open line-end) + (find-matching-close-paren source call-open))]) + (if (and call-close (<= call-close line-end)) + call-close + (line-trimmed-end-before-semicolon source body-start line-end)))) + +(def (javascript-arrow-range-at source arrow spec) + (let* ([start (javascript-arrow-param-start source arrow)] + [param-end0 (skip-horizontal-backward source (- arrow 1))] + [param-end (+ param-end0 1)] + [param-text (and start + (< start param-end) + (substring source start param-end))] + [body-start (skip-whitespace source (+ arrow 2))] + [line-end (line-end-after source arrow)] + [body-end + (and (< body-start (string-length source)) + (if (char=? (string-ref source body-start) #\{) + (find-matching-close-brace source body-start) + (javascript-arrow-expression-body-end + source + body-start + line-end)))]) + (and param-text + (javascript-arrow-param-compatible? spec param-text) + body-end + (cons start body-end)))) + +(def (javascript-function-expression-range-at source start spec) + (let* ([match (re-search + (re "^function[ \\t]*\\([ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\)") + (substring source start (line-end-after source start)) + 0)] + [param (and match (re-match-group match 1))] + [open (and match + (string-find-substring-from source "(" start))] + [close (and open (find-matching-close-paren source open))] + [body-open (and close (skip-whitespace source close))] + [body-close + (and body-open + (< body-open (string-length source)) + (char=? (string-ref source body-open) #\{) + (find-matching-close-brace source body-open))]) + (and param + (javascript-arrow-param-compatible? spec param) + body-close + (cons start body-close)))) + +(def (scan-javascript-arrow-lambda-pattern rule path source pattern) + (let ([spec (javascript-arrow-pattern-spec pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([start 0] [acc '()]) + (let* ([arrow (string-find-substring-from source "=>" start)] + [fn (and (eq? (alist-ref/default spec 'kind #f) 'specific) + (string-find-substring-from + source + "function" + start))] + [use-fn? (and fn (or (not arrow) (< fn arrow)))] + [range (if use-fn? + (javascript-function-expression-range-at + source + fn + spec) + (and arrow + (javascript-arrow-range-at + source + arrow + spec)))] + [match-start (if use-fn? fn arrow)]) + (if (or (not match-start) (> match-start len)) + (nonempty-findings (reverse acc)) + (let* ([finding + (and range + (finding-for-range-with-bindings + rule + path + source + (car range) + (cdr range) + '()))] + [next (max (+ match-start 1) + (if range + (cdr range) + (+ match-start + (if use-fn? + (string-length "function") + 2))))]) + (loop next + (if finding (cons finding acc) acc)))))))))) + (def (scan-javascript-small-pattern-fallbacks rule path source pattern) (or (scan-javascript-member-chain-ellipsis-pattern rule path source pattern) @@ -5534,6 +5723,8 @@ rule path source pattern) (scan-javascript-partial-function-pattern rule path source pattern) + (scan-javascript-any-function-pattern + rule path source pattern) (scan-javascript-empty-if-pattern rule path source pattern) (scan-javascript-regexp-fieldname-pattern @@ -5555,6 +5746,8 @@ (scan-javascript-class-pattern rule path source pattern) (scan-javascript-partial-dependencies-field-pattern + rule path source pattern) + (scan-javascript-arrow-lambda-pattern rule path source pattern))) (def (javascript-arrow-assignment-pattern? pattern) --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -1759,6 +1759,35 @@ (check (length dependencies-findings) => 1) (check (finding-start-line (car dependencies-findings)) => 3))) +(test-case "scan JavaScript arrow and function expression fixture fallbacks" + (let* ([any-function-config + "rules:\n - id: demo.js.any.function\n languages: [javascript]\n message: function\n severity: WARNING\n pattern: 'function ...(...) { ... }'\n"] + [any-function-findings + (scan-config-string any-function-config + "javascript" + "demo.js" + "function test(foo) {\n console.log(foo)\n}\ntryItOut(function named(foo) {\n console.log(foo)\n})\nreturn function(foo) {\n console.log(foo)\n}\n")] + [specific-arrow-config + "rules:\n - id: demo.js.specific.arrow\n languages: [javascript]\n message: arrow\n severity: WARNING\n pattern: '(a) => { ... }'\n"] + [specific-arrow-findings + (scan-config-string specific-arrow-config + "javascript" + "demo.js" + "foo(function (a) { console.log(\"foo\"); });\nfoo( (a) => { console.log(\"foo\"); });\nfoo( a => console.log(\"foo\"));\n")] + [ellipsis-arrow-config + "rules:\n - id: demo.js.ellipsis.arrow\n languages: [javascript]\n message: arrow\n severity: WARNING\n pattern: '(...) => { ... }'\n"] + [ellipsis-arrow-findings + (scan-config-string ellipsis-arrow-config + "javascript" + "demo.js" + "var l = (a,b) => rest[0];\nvar i = (a) => { return 10; }\n")]) + (check (length any-function-findings) => 3) + (check (map finding-start-line any-function-findings) => '(1 4 7)) + (check (length specific-arrow-findings) => 3) + (check (map finding-start-line specific-arrow-findings) => '(1 2 3)) + (check (length ellipsis-arrow-findings) => 2) + (check (map finding-start-line ellipsis-arrow-findings) => '(1 2)))) + (test-case "scan JavaScript metavariable-pattern resolves imported alias" (let* ([js-config "rules:\n - id: demo.metavar.pattern.name.js\n languages: [javascript]\n message: imported alias\n severity: WARNING\n patterns:\n - pattern: var $X = $ID()\n - metavariable-pattern:\n metavariable: $ID\n pattern: foo.bar\n"]