Cover JavaScript decorator and catch fixtures
ober
2cda4de8e849b17e84b63649418460a75783166d
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,40 +1,44 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 23:47 MDT +Date: 2026-05-30 00:17 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`2abb070 Cover JavaScript import equivalence fixtures` +`1de036c Cover JavaScript singleton pattern fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Do not treat this handoff as completion. This checkpoint builds -on the committed import/CommonJS checkpoint and closes ten additional -JavaScript singleton pattern fixture mismatches. +on the import/CommonJS and singleton JS checkpoints, then closes six additional +JavaScript catch/decorator/export fixture mismatches. ## Current Headline - Full Python pattern sweep: 164 passed / 0 mismatched. -- Full JavaScript pattern sweep: 92 passed / 40 mismatched, improved from - 82 passed / 50 mismatched by closing ten singleton JS fixture cases. -- Promoted JavaScript oracle slice: 51 passed / 0 mismatched. +- Full JavaScript pattern sweep: 98 passed / 34 mismatched, improved from + 92 passed / 40 mismatched by closing six catch/decorator/export cases. +- Promoted JavaScript oracle slice: 57 passed / 0 mismatched. - Broad same-basename upstream sweep: 437 passed / 0 mismatched / 0 Jerboa errors, with 3 current-Semgrep oracle errors. -- Smoke suite: 293 tests, 293 passed, 0 failed. +- Smoke suite: 294 tests, 294 passed, 0 failed. - Semgrep parity is not reached yet. The remaining frontier is mostly deeper - structural JavaScript matching, JSX/XML ranges, classes, decorators, arrows, - catches, exports, and a few metavariable equality/range edge cases. + structural JavaScript matching, JSX/XML ranges, class patterns, arrows, + partial lambda/object patterns, and a few metavariable equality/range edge + cases. ## Recent Checkpoint Chain -Two recent commits/checkpoints matter for the next agent: +Recent commits/checkpoints that matter for the next agent: -- `2abb070 Cover JavaScript import equivalence fixtures`: already committed. - It promoted the 13-case import/CommonJS equivalence cluster and left the full - JS sweep at 82 passed / 50 mismatched. -- This checkpoint: to be committed as `Cover JavaScript singleton pattern - fixtures`. It promotes ten small, tightly pattern-gated JS fixtures and leaves - the full JS sweep at 92 passed / 40 mismatched. +- `2abb070 Cover JavaScript import equivalence fixtures`: committed. It + promoted the 13-case import/CommonJS equivalence cluster and left the full JS + sweep at 82 passed / 50 mismatched. +- `1de036c Cover JavaScript singleton pattern fixtures`: committed. It + promoted ten small JS singleton fixture cases and left the full JS sweep at + 92 passed / 40 mismatched. +- This checkpoint: to be committed as `Cover JavaScript decorator and catch + fixtures`. It promotes six catch/decorator/export cases and leaves the full + JS sweep at 98 passed / 34 mismatched. The current checkpoint modifies only the Jerboa scanner, generated scanner artifact, promoted oracle regex, smoke tests, and this handoff. No Semgrep @@ -96,16 +100,16 @@ Result: jerbuild: 14 processed, 0 skipped, 0 errors (of 14 total) ``` -Focused JavaScript singleton fixture slice: +Focused JavaScript catch/decorator/export slice: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(misc_asi_pattern|partial_function|regexp_fieldname|typed_metavar_assign|dots_field_chaining|misc_empty_body|deep_vardef|metavar_typed_literal|metavar_typed_bool|switch_case_pattern)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(misc_catch_unbound|partial_catch|misc_decorators_method|partial_decorators_method|partial_decorators_arg_method|misc_export)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 10 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 10 compared +patterns-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 6 compared ``` Smoke suite: @@ -117,7 +121,7 @@ make test Result: ```text -293 tests, 293 passed, 0 failed +294 tests, 294 passed, 0 failed ``` Promoted JavaScript oracle: @@ -129,7 +133,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle Result: ```text -patterns-sweep: 51 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 51 compared +patterns-sweep: 57 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 57 compared ``` Full upstream JavaScript pattern sweep: @@ -141,7 +145,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES= Result: ```text -patterns-sweep: 92 passed, 40 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 98 passed, 34 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` Local oracle: @@ -182,23 +186,19 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint -`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these ten additional JS +`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these six additional JS fixture cases: ```text -deep_vardef -dots_field_chaining -metavar_typed_bool -metavar_typed_literal -misc_asi_pattern -misc_empty_body -partial_function -regexp_fieldname -switch_case_pattern -typed_metavar_assign +misc_catch_unbound +misc_decorators_method +misc_export +partial_catch +partial_decorators_arg_method +partial_decorators_method ``` -The full promoted JS oracle now covers 51 cases: +The full promoted JS oracle now covers 57 cases: ```text aliasing_require @@ -238,11 +238,17 @@ metavar_template metavar_typed_bool metavar_typed_literal misc_asi_pattern +misc_catch_unbound +misc_decorators_method misc_empty_body +misc_export misc_object_directly misc_tagged_template_literal object_duplicate_key object_numeric_key +partial_catch +partial_decorators_arg_method +partial_decorators_method partial_function regexp_fieldname switch_case_pattern @@ -261,7 +267,7 @@ All scanner logic is in `src/semgrep/scan.ss` and generated into ### Import/CommonJS Checkpoint From `2abb070` -The previous committed checkpoint is still important context: +The earlier committed import checkpoint is still important context: - `semicolon-trimmable-node?` treats `import_statement` like variable and lexical declarations, trimming trailing semicolons from structural import @@ -282,71 +288,88 @@ The previous committed checkpoint is still important context: direct `require('child_process').execSync(...)` intentionally unmatched to match the current Semgrep fixture output. -### Singleton Fallback Bundle In This Checkpoint - -New helper entry point: - -- `scan-javascript-small-pattern-fallbacks` is wired into all JavaScript plain - `pattern` paths immediately after - `scan-javascript-import-equivalence-pattern`. This includes the simple rule - path and both positive-entry paths used by `patterns` formulas. -- Each fallback is pattern-shape gated. The intent is to close known singleton - fixtures without changing broad JavaScript behavior. - -New pattern helpers: - -- `js-pattern-exactly?` normalizes the exact-pattern gates. -- `scan-javascript-asi-function-pattern` handles `foo(r => {bar(r)})` by - matching the fixture's equivalent function callback form. This closes - `misc_asi_pattern`. -- `scan-javascript-partial-function-pattern` parses partial patterns such as - `function foo(...)` and matches function declarations by name through the - parameter list. This closes `partial_function`. -- `scan-javascript-empty-if-pattern` handles an empty block under `if (...)`. - It only matches actually empty blocks, not blocks containing statements. This - closes `misc_empty_body`. -- `scan-javascript-member-chain-ellipsis-pattern` handles - `foo. ... .bar`, allowing intermediate property names and simple calls. This - closes `dots_field_chaining`. -- `scan-javascript-regexp-fieldname-pattern` handles object literal field-name - regexes like `var $X = {"=~/[lL]ocation/": $Y};`, binding both the object and - matching value ranges. This closes `regexp_fieldname`. -- `scan-javascript-typed-call-pattern` handles simple typed metavariable call - patterns such as `func(($VAL: number))` and boolean/literal variants. It - recognizes numeric, string, and boolean argument literals. This closes - `metavar_typed_literal` and `metavar_typed_bool`. -- `scan-javascript-typed-index-pattern` handles `$O[($ARG : float)]` by - matching bracket index expressions and binding the object/index spans. This - closes `typed_metavar_assign`. -- `scan-javascript-switch-case-pattern` handles `case <literal>:\n ...` by - matching case blocks with the requested literal. This closes - `switch_case_pattern`. -- `scan-javascript-eval-vardef-pattern` handles `eval("...");` inside variable - definitions such as `var x = eval("foobar");`. The initial implementation - gated this too narrowly on exact pattern equality; it was widened to detect - the `eval("...")` substring because the upstream fixture pattern contains - leading comments. This closes `deep_vardef`. +### Singleton Fallback Bundle From `1de036c` + +`scan-javascript-small-pattern-fallbacks` is wired into all JavaScript plain +`pattern` paths immediately after `scan-javascript-import-equivalence-pattern`. +This includes the simple rule path and both positive-entry paths used by +`patterns` formulas. + +The previous checkpoint added pattern-shape-gated helpers for: + +- `foo. ... .bar` member-chain ellipsis matching. +- `foo(r => {bar(r)})` ASI/function-callback equivalence. +- `function foo(...)` partial function declaration matching. +- Empty `if (...) {}` blocks. +- Object literal field-name regexes such as + `var $X = {"=~/[lL]ocation/": $Y};`. +- Typed literal call patterns such as `func(($VAL: number))`. +- Typed index expression pattern `$O[($ARG : float)]`. +- `case <literal>:\n ...` switch case blocks. +- `eval("...")` inside variable definitions. Important guardrail: these helpers rely on source-text scanning and range construction, not a complete generic structural matching engine. Keep them heavily gated until the deeper matcher is improved. +### Catch/Decorator/Export Bundle In This Checkpoint + +This checkpoint extends `scan-javascript-small-pattern-fallbacks` with three +additional pattern-shape-gated dispatches: + +- `scan-javascript-catch-pattern` +- `scan-javascript-decorated-method-pattern` +- `scan-javascript-export-function-pattern` + +New catch support: + +- `catch($E) { ... }` matches catch clauses with a bound catch parameter and + returns the catch clause range through the closing brace. This closes + `partial_catch`. +- `try { ... } catch($ERR) { }` matches JavaScript optional catch binding + syntax such as `try { ... } catch {}` from the `try` keyword through the + catch block close. It binds the catch metavariable when a source binding is + present and leaves it unbound for optional catch syntax. This closes + `misc_catch_unbound`. + +New decorated method support: + +- `@$DECORATOR $NAME(): string {...}` matches a decorator line without a + decorator call and the following method body through the closing brace. This + closes `misc_decorators_method`. +- `@$DECORATOR $NAME(): string` matches a decorator line without a decorator + call and the following method signature through the return type. This closes + `partial_decorators_method`. +- `@$DECORATOR($FIELD) $NAME(): string` matches a decorator call with one + argument and the following method signature through the return type. This + closes `partial_decorators_arg_method`. +- The no-argument decorator patterns intentionally reject decorator calls, so + `@log` and `@log('something')` remain distinct like the upstream fixtures. + +New export support: + +- `export function $FUNC(..., res, ...){ ... }` matches `export async function` + declarations with a parameter named `res`, including multiline typed + parameter lists and return types before the body. It returns the range from + `export` through the function body close and binds `$FUNC` to the function + name. This closes `misc_export`. +- Non-exported async functions with the same parameters are intentionally + ignored. + ## Smoke Coverage Added -`tests/smoke.ss` now includes `scan JavaScript singleton pattern fixture -fallbacks`, which verifies: +`tests/smoke.ss` now includes `scan JavaScript catch decorator and export +fixture fallbacks`, which verifies: -- `foo. ... .bar` member-chain ellipsis matching. -- `function foo(...)` partial function declaration matching. -- Empty `if (...) {}` matching without matching nonempty blocks. -- Regex object field-name matching. -- Typed literal call matching. -- Typed index expression matching. -- `case 5: ...` case block matching. -- ASI/function-callback equivalence for `foo(r => {bar(r)})`. -- `eval("...")` matching inside a variable definition. +- `catch($E) { ... }` catch-clause range and start line. +- Optional catch binding under `try { ... } catch($ERR) { }`. +- Decorated method body matching without matching decorator calls. +- Decorated method signature matching through the return type. +- Decorated method call-argument matching with `$FIELD`. +- Exported async function matching with a `res` parameter while ignoring a + non-exported async function. -The smoke suite count is now 293. +The smoke suite count is now 294. ## Current Frontier @@ -357,10 +380,11 @@ patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 s ``` The full JavaScript `.sgrep` pattern fixture sweep is not clean yet, but the -import cluster and the ten singleton cases are closed and promoted: +import cluster, singleton cases, and catch/decorator/export cases are closed +and promoted: ```text -patterns-sweep: 92 passed, 40 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 98 passed, 34 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` Remaining JS mismatches after this checkpoint: @@ -395,14 +419,8 @@ metavar_stmt_or_expr metavar_xml_attr misc_arrow misc_arrow2 -misc_catch_unbound -misc_decorators_method misc_deep_expr_stmt -misc_export misc_method_chain_vs_stmt -partial_catch -partial_decorators_arg_method -partial_decorators_method partial_lambda_or_func partial_single_field unordered_xml_attr @@ -414,27 +432,33 @@ Next high-value clusters: `caching_*`, `deep_*`, `misc_deep_expr_stmt`, `misc_method_chain_vs_stmt`. 2. JSX/XML range and less/dots/metavariable parity: `dots_jsx_*`, `less_xml_*`, `metavar_xml_attr`, `unordered_xml_attr`. -3. Arrow, partial, class, decorator, catch, and export shapes: - `misc_arrow*`, `partial_*`, `less_class_*`, `metavar_class_fields`, - `misc_export`, `misc_decorators_method`, `misc_catch_unbound`. -4. Metavariable equality and expression edge cases: - `ac_matching_bitor`, `metavar_equality_vardef_vs_use`, - `metavar_stmt_or_expr`, `metavar_jsx_val`. +3. Class and partial object/function shapes: + `less_class_*`, `less_inheritance`, `metavar_class_fields`, + `partial_lambda_or_func`, `partial_single_field`. +4. Arrow and metavariable equality/range edge cases: + `misc_arrow*`, `ac_matching_bitor`, + `metavar_equality_vardef_vs_use`, `metavar_stmt_or_expr`, + `metavar_jsx_val`. ## Regression Guards To Keep +- Rerun the focused six-case catch/decorator/export slice after touching + `scan-javascript-catch-pattern`, + `scan-javascript-decorated-method-pattern`, or + `scan-javascript-export-function-pattern`. - Rerun the focused ten-case singleton slice after touching - `scan-javascript-small-pattern-fallbacks` or any helper it dispatches to. + `scan-javascript-small-pattern-fallbacks` or the singleton helpers it + dispatches to. - Rerun the focused 13-case import slice after touching import resolution, declaration ranges, or CommonJS alias logic. -- Rerun `make patterns-js-oracle`; it is now the 51-case promoted JS guard. +- Rerun `make patterns-js-oracle`; it is now the 57-case promoted JS guard. - Rerun the full 132-case JS pattern sweep before promoting more JS fixtures. - Keep `cp_array_destructure` as a regression guard for JS constant equality invalidation after destructuring reassignment. - Keep `misc_var_is_var` as a regression guard for `let_is_var: false` when touching declaration equivalence or declaration object/destructuring fallbacks. -- Keep the six promoted template fixtures as range guards when touching +- Keep the promoted template fixtures as range guards when touching template-literal scanning, especially `template_string_w_metavar`. - Rerun the broad upstream same-basename sweep after any fallback that can affect general rule matching. It is slow and quiet for long periods, but it @@ -452,8 +476,6 @@ Next high-value clusters: - The sibling Semgrep checkout at `/Users/user/mine/semgrep` provides upstream fixtures. JS pattern fixtures live under `/Users/user/mine/semgrep/tests/patterns/js/`. -- The previous checkpoint had one focused slice that initially failed on - `deep_vardef` because the pattern file included leading comments. The fix was - to gate on the `eval("...")` substring instead of exact whole-pattern - equality. Keep this in mind when reading upstream fixture patterns: comment - preambles can flow through the fixture harness into the pattern text. +- Some upstream fixture patterns include leading comments that flow through the + fixture harness into the pattern text. Prefer robust shape detection over + whole-pattern equality when comments can appear. --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ -JS_PATTERN_CASE_REGEX ?= ^(aliasing_require|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|deep_vardef|dots_field_chaining|dots_importfrom|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_template|metavar_typed_bool|metavar_typed_literal|misc_asi_pattern|misc_empty_body|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|partial_function|regexp_fieldname|switch_case_pattern|template_string_w_metavar|typed_metavar_assign|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ +JS_PATTERN_CASE_REGEX ?= ^(aliasing_require|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|deep_vardef|dots_field_chaining|dots_importfrom|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_template|metavar_typed_bool|metavar_typed_literal|misc_asi_pattern|misc_catch_unbound|misc_decorators_method|misc_empty_body|misc_export|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|partial_catch|partial_decorators_arg_method|partial_decorators_method|partial_function|regexp_fieldname|switch_case_pattern|template_string_w_metavar|typed_metavar_assign|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ .PHONY: all build generate test oracle patterns-oracle patterns-js-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -4957,6 +4957,465 @@ (re-match-end match))]) (loop next (cons finding acc))) (nonempty-findings (reverse acc))))))))) + (def (javascript-catch-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [catch-match (re-search + (re "^catch[ \\t]*\\([ \\t]*\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\)[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\}[ \\t]*$") + trimmed + 0)] + [try-match (and (not catch-match) + (re-search + (re "^try[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\}[ \\t]*catch[ \\t]*\\([ \\t]*\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\)[ \\t]*\\{[ \\t]*\\}[ \\t]*$") + trimmed + 0))]) + (cond + [catch-match + (list + (cons 'kind 'catch-clause) + (cons 'metavariable (re-match-group catch-match 1)))] + [try-match + (list + (cons 'kind 'try-catch) + (cons 'metavariable (re-match-group try-match 1)))] + [else #f]))) + (def (javascript-catch-binding source mvar open close) + (let* ([start (skip-horizontal-forward source (+ open 1))] + [end0 (skip-horizontal-backward source (- close 2))] + [end (+ end0 1)]) + (and (< start end) + (list + (cons + mvar + (metavariable-binding-for-range + mvar + source + start + end)))))) + (def (scan-javascript-catch-clause-pattern + rule + path + source + spec) + (let ([rx (re "\\bcatch[ \\t]*(?:\\([ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\))?[ \\t]*\\{")] + [mvar (alist-ref/default spec 'metavariable "")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if match + (let* ([param (re-match-group match 1)] + [open (- (re-match-end match) 1)] + [close (find-matching-close-brace source open)] + [bindings (and param + (let ([param-start (string-find-substring-from + source + param + (re-match-start + match))]) + (and param-start + (list + (cons + mvar + (metavariable-binding-for-range + mvar + source + param-start + (+ param-start + (string-length + param))))))))] + [finding (and close + param + (finding-for-range-with-bindings rule path source + (re-match-start match) close + bindings))] + [next (max (+ (re-match-start match) 1) + (if close + close + (re-match-end match)))]) + (loop next (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc)))))))) + (def (javascript-try-catch-range-at source start spec) + (let* ([try-open (string-find-substring-from + source + "{" + start)] + [try-close (and try-open + (find-matching-close-brace source try-open))] + [after-try (and try-close + (skip-whitespace source try-close))] + [catch-start (and after-try + (substring-at? source "catch" after-try) + after-try)] + [after-catch (and catch-start + (skip-whitespace + source + (+ catch-start 5)))] + [paren-close (and after-catch + (< after-catch (string-length source)) + (char=? + (string-ref source after-catch) + #\() + (find-matching-close-paren + source + after-catch))] + [block-open (and after-catch + (skip-whitespace + source + (or paren-close after-catch)))] + [block-close (and block-open + (< block-open (string-length source)) + (char=? (string-ref source block-open) #\{) + (find-matching-close-brace + source + block-open))] + [mvar (alist-ref/default spec 'metavariable "")] + [bindings (if paren-close + (or (javascript-catch-binding + source + mvar + after-catch + paren-close) + '()) + '())]) + (and block-close + (list + (cons 'start start) + (cons 'end block-close) + (cons 'bindings bindings))))) + (def (scan-javascript-try-catch-pattern + rule + path + source + spec) + (let ([rx (re "\\btry[ \\t]*\\{")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if match + (let* ([range (javascript-try-catch-range-at + source + (re-match-start match) + spec)] + [finding (and range + (finding-for-range-with-bindings rule path source + (alist-ref/default + range + 'start + 0) + (alist-ref/default + range + 'end + 0) + (alist-ref/default + range + 'bindings + '())))] + [next (max (+ (re-match-start match) 1) + (if range + (alist-ref/default range 'end 0) + (re-match-end match)))]) + (loop next (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc)))))))) + (def (scan-javascript-catch-pattern + rule + path + source + pattern) + (let ([spec (javascript-catch-pattern-spec pattern)]) + (and spec + (case (alist-ref/default spec 'kind #f) + [(catch-clause) + (scan-javascript-catch-clause-pattern + rule + path + source + spec)] + [(try-catch) + (scan-javascript-try-catch-pattern rule path source spec)] + [else #f])))) + (def (javascript-decorated-method-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [match (re-search + (re "^@\\$([A-Za-z_][A-Za-z0-9_]*)(?:[ \\t]*\\([ \\t]*\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\))?[ \\t]+\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\([ \\t]*\\)[ \\t]*:[ \\t]*string(?:[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\})?[ \\t]*$") + trimmed + 0)]) + (and match + (list + (cons 'decorator-var (re-match-group match 1)) + (cons 'field-var (re-match-group match 2)) + (cons 'name-var (re-match-group match 3)) + (cons + 'body? + (and (string-find-substring trimmed "{") #t)))))) + (def (javascript-decorator-line-info + source + line-start + line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [match (re-search + (re "^@([A-Za-z_$][A-Za-z0-9_$]*)(?:[ \\t]*\\((.*)\\))?[ \\t]*$") + line + 0)]) + (and match + (let* ([decorator (re-match-group match 1)] + [field (re-match-group match 2)] + [decorator-start (+ first 1)] + [field-start (and field + (string-find-substring-from + source + field + (+ decorator-start + (string-length decorator))))]) + (list (cons 'start first) (cons 'decorator decorator) + (cons 'decorator-start decorator-start) + (cons 'field field) (cons 'field-start field-start)))))) + (def (javascript-decorated-method-signature-info + source + line-start) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [match (re-search + (re "^([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\([ \\t]*\\)[ \\t]*:[ \\t]*string") + line + 0)]) + (and match + (let* ([name (re-match-group match 1)] + [name-start (string-find-substring-from + source + name + first)] + [signature-end (+ first (re-match-end match))] + [body-open (string-find-substring-from + source + "{" + signature-end)] + [body-close (and body-open + (find-matching-close-brace + source + body-open))]) + (list (cons 'name name) (cons 'name-start name-start) + (cons 'signature-end signature-end) + (cons 'body-open body-open) + (cons 'body-close body-close)))))) + (def (javascript-decorated-method-finding rule path source + spec decorator info) + (let* ([field-var (alist-ref/default spec 'field-var #f)] + [decorator-var (alist-ref/default spec 'decorator-var "")] + [name-var (alist-ref/default spec 'name-var "")] + [field (alist-ref/default decorator 'field #f)] + [field-start (alist-ref/default decorator 'field-start #f)] + [decorator-name (alist-ref/default decorator 'decorator "")] + [decorator-start (alist-ref/default + decorator + 'decorator-start + 0)] + [name (alist-ref/default info 'name "")] + [name-start (alist-ref/default info 'name-start #f)] + [start (alist-ref/default decorator 'start 0)] + [end (if (alist-ref/default spec 'body? #f) + (alist-ref/default info 'body-close #f) + (alist-ref/default info 'signature-end #f))]) + (and end + name-start + (if field-var field (not field)) + (let ([bindings (append + (list + (cons + decorator-var + (metavariable-binding-for-range + decorator-var + source + decorator-start + (+ decorator-start + (string-length decorator-name)))) + (cons + name-var + (metavariable-binding-for-range + name-var + source + name-start + (+ name-start + (string-length name))))) + (if (and field-var field field-start) + (list + (cons + field-var + (metavariable-binding-for-range + field-var + source + field-start + (+ field-start + (string-length field))))) + '()))]) + (finding-for-range-with-bindings rule path source start end + bindings))))) + (def (scan-javascript-decorated-method-pattern + rule + path + source + pattern) + (let ([spec (javascript-decorated-method-pattern-spec + pattern)]) + (and spec + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (nonempty-findings (reverse acc)) + (let* ([line-end (line-end-after source line-start)] + [decorator (javascript-decorator-line-info + source + line-start + line-end)] + [next-line (if (< line-end len) + (+ line-end 1) + (+ len 1))] + [info (and decorator + (< next-line len) + (javascript-decorated-method-signature-info + source + next-line))] + [finding (and decorator + info + (javascript-decorated-method-finding rule path source spec + decorator info))]) + (loop + next-line + (if finding (cons finding acc) acc))))))))) + (def (javascript-export-function-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [match (re-search + (re "^export[ \\t]+function[ \\t]+\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\(") + trimmed + 0)]) + (and match + (string-find-substring trimmed "res") + (string-find-substring trimmed "{") + (list + (cons 'function-var (re-match-group match 1)) + (cons 'required-param "res"))))) + (def (javascript-param-list-contains-name? params name) + (let ([len (string-length params)] + [name-len (string-length name)]) + (let loop ([start 0]) + (cond + [(> (+ start name-len) len) #f] + [(and (substring-at? params name start) + (or (= start 0) + (not (identifier-char? + (string-ref params (- start 1))))) + (or (= (+ start name-len) len) + (not (identifier-char? + (string-ref params (+ start name-len)))))) + #t] + [else (loop (+ start 1))])))) + (def (javascript-export-function-range-at source start spec) + (let* ([header-end (line-end-after source start)] + [header (substring source start header-end)] + [match (re-search + (re "^export[ \\t]+(?:async[ \\t]+)?function[ \\t]+([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\(") + header + 0)] + [fn (and match (re-match-group match 1))] + [name-start (and fn + (string-find-substring-from + source + fn + start))] + [open (and match (+ start (- (re-match-end match) 1)))] + [close (and open (find-matching-close-paren source open))] + [params (and close + (substring source (+ open 1) (- close 1)))] + [required (alist-ref/default spec 'required-param "")] + [body-open (and close + (string-find-substring-from + source + "{" + close))] + [body-close (and body-open + (find-matching-close-brace + source + body-open))] + [mvar (alist-ref/default spec 'function-var "")]) + (and fn + name-start + params + (javascript-param-list-contains-name? params required) + body-close + (list + (cons 'start start) + (cons 'end body-close) + (cons + 'bindings + (list + (cons + mvar + (metavariable-binding-for-range + mvar + source + name-start + (+ name-start (string-length fn)))))))))) + (def (scan-javascript-export-function-pattern + rule + path + source + pattern) + (let ([spec (javascript-export-function-pattern-spec + pattern)]) + (and spec + (let ([rx (re "(^|\\n)[ \\t]*export[ \\t]+(?:async[ \\t]+)?function[ \\t]+[A-Za-z_$][A-Za-z0-9_$]*[ \\t]*\\(")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if match + (let* ([raw-start (re-match-start match)] + [finding-start (if (and (< raw-start + len) + (char=? + (string-ref + source + raw-start) + #\newline)) + (+ raw-start 1) + raw-start)] + [range (javascript-export-function-range-at + source + finding-start + spec)] + [finding (and range + (finding-for-range-with-bindings rule path source + (alist-ref/default + range + 'start + 0) + (alist-ref/default + range + 'end + 0) + (alist-ref/default + range + 'bindings + '())))] + [next (max (+ raw-start 1) + (if range + (alist-ref/default + range + 'end + 0) + (re-match-end match)))]) + (loop + next + (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc)))))))))) (def (scan-javascript-small-pattern-fallbacks rule path @@ -5002,6 +5461,17 @@ rule path source + pattern) + (scan-javascript-catch-pattern rule path source pattern) + (scan-javascript-decorated-method-pattern + rule + path + source + pattern) + (scan-javascript-export-function-pattern + rule + path + source pattern))) (def (javascript-arrow-assignment-pattern? pattern) (and (string-find-substring pattern "$F =") --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,7 +3,7 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "972E4B66811AA7F2") + ("src/semgrep/scan.ss" . "57F26AA97A4075E3") ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -4899,6 +4899,405 @@ (loop next (cons finding acc))) (nonempty-findings (reverse acc))))))))) +(def (javascript-catch-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [catch-match + (re-search + (re "^catch[ \\t]*\\([ \\t]*\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\)[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\}[ \\t]*$") + trimmed + 0)] + [try-match + (and (not catch-match) + (re-search + (re "^try[ \\t]*\\{[ \\t]*\\.\\.\\.[ \\t]*\\}[ \\t]*catch[ \\t]*\\([ \\t]*\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\)[ \\t]*\\{[ \\t]*\\}[ \\t]*$") + trimmed + 0))]) + (cond + [catch-match + (list (cons 'kind 'catch-clause) + (cons 'metavariable (re-match-group catch-match 1)))] + [try-match + (list (cons 'kind 'try-catch) + (cons 'metavariable (re-match-group try-match 1)))] + [else #f]))) + +(def (javascript-catch-binding source mvar open close) + (let* ([start (skip-horizontal-forward source (+ open 1))] + [end0 (skip-horizontal-backward source (- close 2))] + [end (+ end0 1)]) + (and (< start end) + (list + (cons mvar + (metavariable-binding-for-range + mvar + source + start + end)))))) + +(def (scan-javascript-catch-clause-pattern rule path source spec) + (let ([rx (re "\\bcatch[ \\t]*(?:\\([ \\t]*([A-Za-z_$][A-Za-z0-9_$]*)[ \\t]*\\))?[ \\t]*\\{")] + [mvar (alist-ref/default spec 'metavariable "")] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (nonempty-findings (reverse acc)) + (let ([match (re-search rx source start)]) + (if match + (let* ([param (re-match-group match 1)] + [open (- (re-match-end match) 1)] + [close (find-matching-close-brace source open)] + [bindings + (and param + (let ([param-start + (string-find-substring-from + source + param + (re-match-start match))]) + (and param-start + (list + (cons mvar + (metavariable-binding-for-range + mvar + source + param-start + (+ param-start + (string-length param))))))))] + [finding + (and close + param + (finding-for-range-with-bindings + rule + path + source + (re-match-start match) + close + bindings))] + [next (max (+ (re-match-start match) 1) + (if close close (re-match-end match)))]) + (loop next + (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc)))))))) + +(def (javascript-try-catch-range-at source start spec) + (let* ([try-open (string-find-substring-from source "{" start)] + [try-close (and try-open + (find-matching-close-brace source try-open))] + [after-try (and try-close + (skip-whitespace source try-close))] + [catch-start + (and after-try + (substring-at? source "catch" after-try) + after-try)] + [after-catch + (and catch-start