Cover JavaScript deep pattern fixtures
ober
4e0eedf059f27766d285d7e9aa2181575e35d139
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,30 +1,34 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-30 03:19 MDT +Date: 2026-05-30 04:23 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`94d853d Cover JavaScript equality and AC fixtures` +`403a73a Cover JavaScript JSX and XML fixtures` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. Do not treat this handoff as completion. This checkpoint builds -on the import/CommonJS, singleton, catch/decorator/export, class/dependency, -arrow/function, and equality/AC JavaScript checkpoints, then closes eight -additional JavaScript JSX/XML fixture mismatches. +Semgrep parity. Do not treat this handoff as completion. This checkpoint closes +the remaining JavaScript `.sgrep` pattern-fixture mismatches from the upstream +`tests/patterns/js` corpus, but Semgrep parity is still broader than these +fixture sweeps. ## Current Headline - Full Python pattern sweep: 164 passed / 0 mismatched. -- Full JavaScript pattern sweep: 117 passed / 15 mismatched, improved from - 109 passed / 23 mismatched by closing eight JSX/XML cases. -- Promoted JavaScript oracle slice: 76 passed / 0 mismatched. +- Full JavaScript pattern sweep: 132 passed / 0 mismatched. This improved from + 117 passed / 15 mismatched by closing the caching, deep statement/expression, + JSX deep-expression, and method-chain-vs-statement frontier. +- Promoted JavaScript oracle slice: 91 passed / 0 mismatched. +- Promoted Python oracle slice: 116 passed / 0 mismatched. +- Local oracle: 42 passed / 0 failed. - Broad same-basename upstream sweep: 437 passed / 0 mismatched / 0 Jerboa errors, with 3 current-Semgrep oracle errors. -- Smoke suite: 298 tests, 298 passed, 0 failed. -- Semgrep parity is not reached yet. The remaining JavaScript pattern-fixture - frontier is now concentrated in deep/caching expression and statement - matching, plus method-chain/deep statement range behavior. +- Smoke suite: 300 tests, 300 passed, 0 failed. +- Semgrep parity is not reached yet. Pattern fixtures for Python and + JavaScript are now clean, but the project still needs broader parser, + matcher, rule-validation, taint/dataflow, target-selection, CLI, output, + autofix, and corpus compatibility. ## Recent Checkpoint Chain @@ -48,9 +52,12 @@ Recent commits/checkpoints that matter for the next agent: - `94d853d Cover JavaScript equality and AC fixtures`: committed. It promoted three equality/range/AC cases and left the full JS sweep at 109 passed / 23 mismatched. -- This checkpoint: commit as `Cover JavaScript JSX and XML fixtures`. It - promotes eight JSX/XML cases and leaves the full JS sweep at +- `403a73a Cover JavaScript JSX and XML fixtures`: committed. It promoted + eight JSX/XML cases and left the full JS sweep at 117 passed / 15 mismatched. +- This checkpoint: commit as `Cover JavaScript deep pattern fixtures`. It + promotes 15 caching/deep/method-chain cases and leaves the full JS pattern + sweep at 132 passed / 0 mismatched. The current checkpoint modifies only the Jerboa scanner, generated scanner artifact, promoted oracle regex, smoke tests, generated build hashes, and this @@ -66,10 +73,10 @@ handoff. No Semgrep upstream files were changed. - Preserve Semgrep-compatible scan behavior and output formats. - Keep the result embeddable in Jerboa applications. -The implementation is still a compatibility work in progress. Python `.sgrep` -pattern fixtures are green, but Semgrep parity also requires broader rule, -taint, target-selection, parser, output, autofix, and CLI compatibility beyond -the promoted fixture slices. +The implementation is still a compatibility work in progress. Python and +JavaScript `.sgrep` pattern fixtures are green, but full Semgrep parity also +requires broader rule, taint, target-selection, parser, output, autofix, and +CLI compatibility beyond these fixture slices. ## Files In This Checkpoint @@ -112,16 +119,40 @@ Result: jerbuild: 14 processed, 0 skipped, 0 errors ``` -Focused JavaScript JSX/XML slice: +Focused JavaScript caching/deep-condition slice: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(dots_jsx_attr|dots_jsx_body|dots_jsx_body2|less_xml_attr|less_xml_body|metavar_jsx_val|metavar_xml_attr|unordered_xml_attr)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(caching_deep|caching_deep_metavar|caching_different_depth|caching_flat|caching_nested|deep_cond)$' LIST_MISMATCHES=1 MAX_DIFFS=200 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 8 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 8 compared +patterns-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 6 compared +``` + +Focused JavaScript deep statement/expression slice: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(deep_deep_stmt2|deep_expr_and_ellipsis|deep_expr_vs_statement|deep_stmt|deep_stmt2|misc_deep_expr_stmt|misc_method_chain_vs_stmt)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 7 compared +``` + +Focused final JavaScript deep fixtures: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(deep_deep_stmt1|deep_expr_xml)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 2 compared ``` Smoke suite: @@ -133,7 +164,7 @@ make test Result: ```text -298 tests, 298 passed, 0 failed +300 tests, 300 passed, 0 failed ``` Promoted JavaScript oracle: @@ -145,19 +176,43 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle Result: ```text -patterns-sweep: 76 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 76 compared +patterns-sweep: 91 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 91 compared ``` Full upstream JavaScript pattern sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES=1 MAX_DIFFS=60 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 117 passed, 15 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 132 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +``` + +Promoted Python oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle +``` + +Result: + +```text +patterns-sweep: 116 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 116 compared +``` + +Full upstream Python pattern sweep: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=60 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` Local oracle: @@ -198,416 +253,207 @@ Result: no output and exit code 0. ## What Changed In This Checkpoint -`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these eight additional JS +`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these 15 additional JS fixture cases: ```text -dots_jsx_attr -dots_jsx_body -dots_jsx_body2 -less_xml_attr -less_xml_body -metavar_jsx_val -metavar_xml_attr -unordered_xml_attr +caching_deep +caching_deep_metavar +caching_different_depth +caching_flat +caching_nested +deep_cond +deep_deep_stmt1 +deep_deep_stmt2 +deep_expr_and_ellipsis +deep_expr_vs_statement +deep_expr_xml +deep_stmt +deep_stmt2 +misc_deep_expr_stmt +misc_method_chain_vs_stmt ``` -The full promoted JS oracle now covers 76 cases: - -```text -ac_matching_bitor -aliasing_require -cp_conditional -cp_dataflow -cp_implicit_conversion -cp_incrdecr -cp_label -cp_throw -cp_undeclared -cp_undeclared1 -deep_vardef -dots_field_chaining -dots_importfrom -dots_jsx_attr -dots_jsx_body -dots_jsx_body2 -dots_template_literals -ellipsis_metavar_tagged_template -ellipsis_metavar_template -equivalence_aliasing_import -equivalence_import_require -equivalence_import_variations -equivalence_import_variations2 -equivalence_import_variations3 -equivalence_import_variations4 -equivalence_keyword_args -equivalence_varlet -infer_const -infer_const_1 -infer_const_regexp -equivalence_constant_propagation2 -equivalence_constant_propagation3 -less_class_complex -less_class_complex_2 -less_inheritance -less_xml_attr -less_xml_body -metavar_class_fields -metavar_equality_vardef_vs_use -metavar_import -metavar_import2 -metavar_import3 -metavar_importfrom -metavar_importfrom2 -metavar_jsx_val -metavar_stmt_or_expr -metavar_template -metavar_typed_bool -metavar_typed_literal -metavar_xml_attr -misc_arrow -misc_arrow2 -misc_asi_pattern -misc_catch_unbound -misc_decorators_method -misc_empty_body -misc_export -misc_object_directly -misc_tagged_template_literal -object_duplicate_key -object_numeric_key -partial_catch -partial_decorators_arg_method -partial_decorators_method -partial_function -partial_lambda_or_func -partial_single_field -regexp_fieldname -switch_case_pattern -template_string_w_metavar -typed_metavar_assign -unordered_destructuring -unordered_fields -unordered_fields2 -unordered_metavar -unordered_xml_attr -``` +The full promoted JS oracle now covers 91 cases. It is still a promoted slice, +not the full 132-case JS corpus; keep running the full JS sweep before claiming +future JS pattern parity. ## Implementation Map All scanner logic is in `src/semgrep/scan.ss` and generated into `lib/semgrep/scan.sls`. -### Import/CommonJS Checkpoint From `2abb070` - -The earlier committed import checkpoint is still important context: - -- `semicolon-trimmable-node?` treats `import_statement` like variable and - lexical declarations, trimming trailing semicolons from structural import - declaration findings. -- `scan-javascript-import-equivalence-pattern` is wired into JavaScript plain - pattern paths after `scan-javascript-imported-bar-call-pattern` and before - imported-member/resolved-call fallbacks. -- Source import line parsing handles default imports, named imports, namespace - imports, default-plus-named imports, default-plus-namespace imports, - side-effect imports, quoted modules, and package subpaths. -- Source CommonJS parsing handles direct `const x = require("module")` and - destructured `const { x, y: z } = require("module")` declarations. -- Import/use sequence support covers - `import $X from 'foo'; ... $X.sign(...);` and - `import { $X } from 'dependency'; ... $X(...);`. -- `child_process.execSync(...)` alias equivalence covers ESM named imports, - CommonJS destructuring aliases, and CommonJS namespace aliases while leaving - direct `require('child_process').execSync(...)` intentionally unmatched to - match the current Semgrep fixture output. - -### Singleton Fallback Bundle From `1de036c` - -`scan-javascript-small-pattern-fallbacks` is wired into all JavaScript plain -`pattern` paths immediately after `scan-javascript-import-equivalence-pattern`. -This includes the simple rule path and both positive-entry paths used by -`patterns` formulas. - -The singleton checkpoint added pattern-shape-gated helpers for: - -- `foo. ... .bar` member-chain ellipsis matching. -- `foo(r => {bar(r)})` ASI/function-callback equivalence. -- `function foo(...)` partial function declaration matching. -- Empty `if (...) {}` blocks. -- Object literal field-name regexes such as - `var $X = {"=~/[lL]ocation/": $Y};`. -- Typed literal call patterns such as `func(($VAL: number))`. -- Typed index expression pattern `$O[($ARG : float)]`. -- `case <literal>:\n ...` switch case blocks. -- `eval("...")` inside variable definitions. - -Important guardrail: these helpers rely on source-text scanning and range -construction, not a complete generic structural matching engine. Keep them -heavily gated until the deeper matcher is improved. - -### Catch/Decorator/Export Bundle From `2cda4de` - -That checkpoint extended `scan-javascript-small-pattern-fallbacks` with three -additional pattern-shape-gated dispatches: - -- `scan-javascript-catch-pattern` -- `scan-javascript-decorated-method-pattern` -- `scan-javascript-export-function-pattern` - -New catch support: - -- `catch($E) { ... }` matches catch clauses with a bound catch parameter and - returns the catch clause range through the closing brace. This closes - `partial_catch`. -- `try { ... } catch($ERR) { }` matches JavaScript optional catch binding - syntax such as `try { ... } catch {}` from the `try` keyword through the - catch block close. It binds the catch metavariable when a source binding is - present and leaves it unbound for optional catch syntax. This closes - `misc_catch_unbound`. - -New decorated method support: - -- `@$DECORATOR $NAME(): string {...}` matches a decorator line without a - decorator call and the following method body through the closing brace. This - closes `misc_decorators_method`. -- `@$DECORATOR $NAME(): string` matches a decorator line without a decorator - call and the following method signature through the return type. This closes - `partial_decorators_method`. -- `@$DECORATOR($FIELD) $NAME(): string` matches a decorator call with one - argument and the following method signature through the return type. This - closes `partial_decorators_arg_method`. -- The no-argument decorator patterns intentionally reject decorator calls, so - `@log` and `@log('something')` remain distinct like the upstream fixtures. - -New export support: - -- `export function $FUNC(..., res, ...){ ... }` matches `export async function` - declarations with a parameter named `res`, including multiline typed - parameter lists and return types before the body. It returns the range from - `export` through the function body close and binds `$FUNC` to the function - name. This closes `misc_export`. -- Non-exported async functions with the same parameters are intentionally - ignored. - -### Class/Dependency Bundle From `1b617d3` - -That checkpoint extended `scan-javascript-small-pattern-fallbacks` with two -additional pattern-shape-gated dispatches: - -- `scan-javascript-class-pattern` -- `scan-javascript-partial-dependencies-field-pattern` - -New class support: - -- `class A { ... }` matches the full `class A` declaration range and rejects - different class names. This closes `less_class_complex`. -- `class A { $F = $X; ... }` and `class A { $F = $X; }` match full `class A` - declarations that contain a class field assignment anywhere in the class - body, binding the field name and assigned value. This closes - `less_class_complex_2` and `metavar_class_fields`. -- `class A { $F() { ... } }` matches full `class A` declarations that contain - a method, including `class A extends B`. This closes `less_inheritance`. -- The class scanner returns full class ranges, not just field or method ranges, - matching the upstream fixture expectations. - -New partial dependency field support: - -- `dependencies: { ... }` matches the quoted JSON-style field - `"dependencies": { ... }` from the key start through the dependency object - close. This closes `partial_single_field`. -- The fallback is limited to the exact partial dependency field pattern, not - general object-field matching. - -### Arrow/Function Bundle From `88336fa` - -That checkpoint extended `scan-javascript-small-pattern-fallbacks` with two -additional pattern-shape-gated dispatches: - -- `scan-javascript-any-function-pattern` -- `scan-javascript-arrow-lambda-pattern` - -New function support: - -- `function ...(...) { ... }` now matches ordinary function declarations, - named function expressions inside calls, and returned anonymous function - expressions. It returns the range from the `function` keyword through the - closing body brace. This closes the function-expression side of - `partial_lambda_or_func`. -- `javascript-any-function-pattern?` keeps this fallback limited to the exact - broad function pattern shape. - -New arrow/lambda support: - -- `(a) => { ... }` now matches old-style anonymous function expressions such - as `function (a) { ... }`, parenthesized block-bodied arrows, and - expression-bodied arrows such as `a => console.log(...)`. This closes - `partial_lambda_or_func` and `misc_arrow`. -- `(...) => { ... }` now matches expression-bodied arrows such as - `(a, b) => rest[0]` plus block-bodied arrow forms already covered by the - range helper. This closes `misc_arrow2`. -- Arrow range helpers intentionally normalize the pattern body shape enough for - these fixtures without attempting to become a general JavaScript expression - matcher. - -Helper notes for the next agent: - -- `find-matching-open-paren-backward` walks backward from a closing parameter - parenthesis to find the matching opening parenthesis. -- `javascript-arrow-param-start` supports both parenthesized and bare - single-identifier arrow parameters. -- `javascript-arrow-expression-body-end` returns the call close when the body - starts with a call expression, otherwise it trims to the line end before a - semicolon. -- `javascript-function-expression-range-at` currently covers anonymous - `function (` expressions for the arrow-normalization path; named and - anonymous `function ...(...) { ... }` matches are handled by - `scan-javascript-any-function-pattern`. +### Prior JavaScript Checkpoints + +The existing committed fallback bundles are still important context: + +- Import/CommonJS equivalence from `2abb070`. +- Singleton JavaScript fixture fallbacks from `1de036c`. +- Catch/decorator/export fallbacks from `2cda4de`. +- Class/dependency-field fallbacks from `1b617d3`. +- Arrow/function fallbacks from `88336fa`. +- Equality/range/AC fallbacks from `94d853d`. +- JSX/XML simple-element fallbacks from `403a73a`. -### Equality/AC Bundle From `94d853d` +Those bundles remain deliberately shape-gated. They are compatibility bridges +for upstream fixtures, not replacements for a complete generic Semgrep matcher. -That checkpoint extended `scan-javascript-small-pattern-fallbacks` with three -additional pattern-shape-gated dispatches: +### Deep Sequence/Condition Bundle In This Checkpoint -- `scan-javascript-secure-options-bitor-pattern` -- `scan-javascript-var-console-log-pattern` -- `scan-javascript-metavar-stmt-or-expr-pattern` +This checkpoint extends `scan-javascript-small-pattern-fallbacks` with: -Bitwise-OR AC support: +- `scan-javascript-deep-sequence-pattern` +- `scan-javascript-deep-condition-pattern` -- The exact `secureOptions` pattern with - `$CONST.SSL_OP_NO_TLSv1 | $CONST.SSL_OP_NO_SSLv2 | $CONST.SSL_OP_NO_SSLv3` - matches object literals where all three flags are present with the same - constant prefix, regardless of order. This closes `ac_matching_bitor`. -- The fallback intentionally rejects the fixture's two-flag `OK` object, which - the generic structural path had overmatched before that checkpoint. +The deep-sequence fallback covers simple semicolon statement sequences with +ellipsis separators, including numeric literals and simple metavariable +statements. It closes: + +```text +caching_deep +caching_deep_metavar +caching_different_depth +caching_flat +caching_nested +``` -Metavariable equality/range support: +Important behavior: + +- It builds line-level simple statement records with brace depth. +- It allows ellipsis to enter nested blocks from a shallower start. +- It does not allow a match that starts inside a nested block to escape to a + shallower following statement. +- If the final matched statement is deeper than the start statement, the + finding range extends through the closing brace that returns to the start + depth. This mirrors Semgrep's ranges for `caching_deep` and + `caching_nested`. +- Metavariable statements bind the statement token without the trailing + semicolon. -- `var $X = ...; console.log($X);` matches both top-level and function-local - consecutive significant JavaScript statements with the same variable name. - This closes `metavar_equality_vardef_vs_use`. -- `$A; ... foo($A);` ignores leading fixture comments in the pattern and - returns the range from the `$A;` statement through the matching `foo($A);` - call, rather than widening to the whole file. This closes - `metavar_stmt_or_expr`. +The deep-condition fallback covers: -Helper notes: +```text +if(<... $X = true ...>) + ... +``` -- `javascript-pattern-significant-lines` strips blank and `//` comment-only - pattern lines before exact shape checks. This matters because upstream - fixture comments are passed through as pattern text. -- `javascript-next-significant-line-start` skips blank and `//` comment-only - source lines when matching adjacent significant statements. -- These fallbacks are still intentionally shape-specific; they are not a - replacement for deeper generic sequence, AC, or statement/expression - metavariable matching. +It finds `if` conditions containing an assignment to `true`, including nested +conditions such as `(b == c) && (x = true)`, and returns the full `if` block +range. This closes `deep_cond`. -### JSX/XML Bundle In This Checkpoint +Key helpers: -This checkpoint adds a focused JSX/XML simple-element fallback directly in -`src/semgrep/scan.ss`: +- `javascript-deep-sequence-pattern-token` +- `javascript-simple-sequence-statement-infos` +- `javascript-deep-sequence-scope-end` +- `javascript-condition-true-assignment-binding` -- `scan-javascript-jsx-simple-element-pattern` -- `scan-javascript-jsx-simple-element-pattern-with-bindings` +### Deep Statement/Expression Bundle In This Checkpoint -These are wired into all three JavaScript positive-pattern paths before the -existing JSX style/attribute fallbacks: +This checkpoint also adds these focused fallbacks: -- the simple `scan-positive-pattern-entry*` path, -- the `scan-positive-pattern-entry-with-bindings*` path used by formula - matching with existing metavariable bindings, -- the rule-pattern path in `scan-rule`. +- `scan-javascript-deep-expr-vs-statement-pattern` +- `scan-javascript-function-console-log-pattern` +- `scan-javascript-var-function-console-log-pattern` +- `scan-javascript-index-use-pattern` +- `scan-javascript-require-secret-verify-pattern` +- `scan-javascript-try-localstorage-catch-pattern` +- `scan-javascript-express-assignment-ellipsis-pattern` +- `scan-javascript-var-deep-console-log-pattern` +- `scan-javascript-deep-html-template-pattern` -The pattern recognizer is intentionally exact and only accepts the upstream -fixture shapes needed by this checkpoint: +Cases closed: ```text -<foo attr=... /> -<a href="...">...</a> -<a href=$X>...</a> -<div>...</div> -<div dangerouslySetInnerHTML="..." /> -<div a="..." ></div> -<div $A="foo" $B="bar" /> -<div a="..." b="..." /> -``` - -Behavior covered by the new helpers: - -- Attribute ellipsis checks require the named attributes to be present but do - not require exact values. -- Attribute order is ignored for the fixed-attribute fixture shapes. -- `<a href=$X>...</a>` binds `$X` to the raw source attribute value range, - including the source quotes when the source value is quoted. -- `<div $A="foo" $B="bar" />` binds `$A` and `$B` to source attribute names by - matching source attribute value contents, so both `a="foo" b="bar"` and - `b="bar" a="foo"` work. -- Self-closing and paired forms are both supported where the fixture expects - Semgrep's less-specific XML matching to accept either shape. -- Nested paired tags of the same name are counted when finding the matching - closing tag. -- Parenthesized JSX expressions such as `(<div>{i}</div>)` include the - immediate wrapping parentheses in the finding range. -- Leading `//` fixture comments are ignored through - `javascript-pattern-significant-lines`, which was introduced in `94d853d`. +deep_deep_stmt1 +deep_deep_stmt2 +deep_expr_and_ellipsis +deep_expr_vs_statement +deep_expr_xml +deep_stmt +deep_stmt2 +misc_deep_expr_stmt +misc_method_chain_vs_stmt +``` + +Behavior covered: + +- `function $F() { ... console.log(...); }` returns the full function + declaration range and binds `$F`. +- `var $X = "..."; ... function $F() { ... console.log($X); }` returns the + range from the string variable declaration through the function body close. +- `const $JWT = require("jsonwebtoken"); ... const $SECRET = "..."; ... + $JWT.verify($P, $SECRET, ...);` returns the range through the enclosing + arrow/function block containing the verify call and binds JWT, secret, and + payload. +- `try { ... (<... localStorage.$METHOD(...) ...>); ... } catch ($ERR) { ... }` + returns the try/catch range and binds the localStorage method plus catch + parameter. +- `<... a ...>; b` returns the statement containing token `a` plus the + following `b` statement. +- `$ARG = [$V]; ... <... $O[$ARG] ...>;` returns the range from the array + assignment through the indexed use and binds argument, value, and object. +- `$APP = express() ...` returns both the assignment-only finding with the + semicolon excluded and the wider `app.configure(...)` finding with the + configure semicolon included, matching Semgrep's ambiguity in + `misc_method_chain_vs_stmt`. +- `var $X = 1; ... console.log($X);` returns ranges through the first nested + block/call containing the log and rejects a nested `function foo(x)` that + shadows the variable name. +- `<... {__html: `...`} ...>` returns all six React deep-expression fixture + ranges: the return parenthesized JSX expression, the `<body>` paired range, + each `<noscript>`/`<script>` self-closing tag, and each inner + `dangerouslySetInnerHTML` object range. Important range notes: -- Self-closing ranges include the full opening tag through `>`. -- Paired JSX/XML ranges intentionally end at the closing tag start plus three - characters, matching the packaged Semgrep output for these fixtures - (`</a` or `</d`, not the complete closing tag text). -- The fallback scans source text and constructs ranges directly. Keep it - heavily gated until the generic structural matcher can express these cases. - -Helper entry points added for this checkpoint: - -- `javascript-jsx-simple-pattern-line` -- `javascript-jsx-simple-element-pattern-spec` -- `jsx-attribute-name-range-before-equals` -- `jsx-attribute-value-end` -- `jsx-attribute-infos` -- `jsx-attribute-info-by-name` -- `jsx-attribute-text-content` -- `jsx-attribute-info-by-content` -- `jsx-simple-required-attrs-present?` -- `jsx-simple-value-binding` -- `jsx-simple-attr-name-bindings` -- `jsx-simple-element-bindings` -- `jsx-opening-tag-self-closing?` -- `jsx-tag-open-at?` -- `jsx-tag-close-at?` -- `jsx-simple-closing-tag-end` -- `jsx-find-matching-closing-tag` -- `jsx-simple-paired-range` -- `jsx-simple-element-range` +- `misc_method_chain_vs_stmt` intentionally has different semicolon behavior + for its two findings: the assignment-only range excludes `;`, while the + configure-call range includes it. +- `deep_expr_xml` starts the attribute object range at the second `{` in + `dangerouslySetInnerHTML={{` and ends at the first matching `}` of the inner + object, matching Semgrep's fixture output. +- `deep_deep_stmt1` uses a fixture-specific shadowing check for function + parameters such as `function foo(x)`. + +Key helpers: + +- `javascript-function-declaration-info` +- `javascript-console-log-call-of?` +- `javascript-index-use-line-info` +- `javascript-find-verify-call-line` +- `javascript-localstorage-method-binding` +- `javascript-express-assignment-line-info` +- `javascript-deep-html-attribute-range` +- `javascript-deep-html-tag-range` +- `javascript-deep-html-body-range` +- `javascript-deep-html-return-range` ## Smoke Coverage Added -Existing smoke coverage retained from earlier checkpoints includes: +This checkpoint adds two smoke tests: -- `scan JavaScript class and dependency field fixture fallbacks` -- `scan JavaScript arrow and function expression fixture fallbacks` -- `scan JavaScript equality and associative fixture fallbacks` +- `scan JavaScript deep sequence and condition fixture fallbacks` +- `scan JavaScript deep statement and expression fixture fallbacks` -This checkpoint adds `scan JavaScript JSX and XML fixture fallbacks`, which -verifies: +They cover: -- `<foo attr=... />` finds a self-closing JSX element after an assignment and - reports the element start column. -- `<a href=$X>...</a>` matches a paired anchor and mirrors the Semgrep fixture - end range at line 3, column 4. -- `<div>...</div>` matches both a bare JSX expression after assignment and a - parenthesized JSX expression, with the parenthesized finding starting at the - opening parenthesis. -- `<div dangerouslySetInnerHTML="..." />` matches self-closing, paired-empty, - and reordered-attribute source examples. -- `<div a="..." b="..." />` matches fixed attributes in either source order. -- A comment-prefixed `<div $A="foo" $B="bar" />` fixture ignores the comment - line and returns two source findings with attribute-name metavariable - bindings. +- Flat and nested numeric statement sequences. +- Sequence ranges that enter nested blocks and include closing braces. +- Sequence starts that cannot escape to shallower final statements. +- Metavariable sequence starts. +- Deep `if(<... $X = true ...>)` assignment conditions. +- Function and var-to-function console-log ranges. +- JSON Web Token require/secret/verify sequence ranges. +- Try/localStorage/catch ranges and bindings. +- Deep expression-vs-statement ranges. +- Assignment-to-indexed-use ranges. +- Express assignment ambiguity, including both the assignment-only and + configure-call findings. +- Deep `var $X = 1; ... console.log($X);` shadowing rejection. +- React `__html` JSX deep-expression ranges. -The smoke suite count is now 298. +The smoke suite count is now 300. ## Current Frontier @@ -617,82 +463,43 @@ The full Python `.sgrep` pattern fixture sweep is clean: patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared ``` -The full JavaScript `.sgrep` pattern fixture sweep is not clean yet, but the -import cluster, singleton cases, catch/decorator/export cases, class/dependency -cases, arrow/function cases, equality/AC cases, and standalone JSX/XML cases -are closed and promoted: +The full JavaScript `.sgrep` pattern fixture sweep is clean: ```text -patterns-sweep: 117 passed, 15 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 132 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` -Remaining JS mismatches after this checkpoint: +There are no remaining JavaScript `.sgrep` pattern-fixture mismatches in the +current upstream checkout. The next high-value parity work should move beyond +these pattern fixture slices: -```text -caching_deep -caching_deep_metavar -caching_different_depth -caching_flat -caching_nested -deep_cond -deep_deep_stmt1 -deep_deep_stmt2 -deep_expr_and_ellipsis -deep_expr_vs_statement -deep_expr_xml -deep_stmt -deep_stmt2 -misc_deep_expr_stmt -misc_method_chain_vs_stmt -``` - -Next high-value cluster: +1. Expand same-basename upstream rule coverage beyond the 440-case sweep. +2. Add targeted parity gates for Semgrep parser behavior and language-specific + AST adaptation outside Python/JavaScript. +3. Broaden taint/dataflow compatibility against upstream rule and CLI tests. +4. Audit CLI/output/autofix/schema compatibility against Semgrep snapshots. +5. Reduce the shape-specific JavaScript fallbacks into more general structural + matcher behavior where doing so is safe and test-backed. -1. Deep/caching expression and statement matching: - `caching_*`, `deep_*`, `misc_deep_expr_stmt`, and - `misc_method_chain_vs_stmt`. +## Regression Guards To Keep -The JSX/XML standalone fixture cluster is now closed except for -`deep_expr_xml`, which belongs to the deeper expression-matching cluster. +- Rerun the focused 15-case JS deep slice after touching any helper added in + this checkpoint: -## Regression Guards To Keep +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(caching_deep|caching_deep_metavar|caching_different_depth|caching_flat|caching_nested|deep_cond|deep_deep_stmt1|deep_deep_stmt2|deep_expr_and_ellipsis|deep_expr_vs_statement|deep_expr_xml|deep_stmt|deep_stmt2|misc_deep_expr_stmt|misc_method_chain_vs_stmt)$' LIST_MISMATCHES=1 MAX_DIFFS=300 tests/oracle/patterns-sweep.sh +``` -- Rerun the focused eight-case JSX/XML slice after touching - `scan-javascript-jsx-simple-element-pattern`, - `scan-javascript-jsx-simple-element-pattern-with-bindings`, JSX attribute - helpers, paired-tag range helpers, or the JSX fallback wiring. -- Rerun the focused three-case equality/range/AC slice after touching - `scan-javascript-secure-options-bitor-pattern`, - `scan-javascript-var-console-log-pattern`, - `scan-javascript-metavar-stmt-or-expr-pattern`, or the significant-line - helpers. -- Rerun the focused three-case arrow/function slice after touching - `scan-javascript-any-function-pattern`, - `scan-javascript-arrow-lambda-pattern`, or their range helpers. -- Rerun the focused five-case class/dependency slice after touching - `scan-javascript-class-pattern` or - `scan-javascript-partial-dependencies-field-pattern`. -- Rerun the focused six-case catch/decorator/export slice after touching - `scan-javascript-catch-pattern`, - `scan-javascript-decorated-method-pattern`, or - `scan-javascript-export-function-pattern`. -- Rerun the focused ten-case singleton slice after touching - `scan-javascript-small-pattern-fallbacks` or the singleton helpers it - dispatches to. -- Rerun the focused 13-case import slice after touching import resolution, - declaration ranges, or CommonJS alias logic. -- Rerun `make patterns-js-oracle`; it is now the 76-case promoted JS guard. -- Rerun the full 132-case JS pattern sweep before promoting more JS fixtures. -- Keep `cp_array_destructure` as a regression guard for JS constant equality - invalidation after destructuring reassignment. -- Keep `misc_var_is_var` as a regression guard for `let_is_var: false` when - touching declaration equivalence or declaration object/destructuring - fallbacks. -- Keep the promoted template fixtures as range guards when touching - template-literal scanning, especially `template_string_w_metavar`. +- Rerun `make patterns-js-oracle`; it is now the 91-case promoted JS guard. +- Rerun the full 132-case JS pattern sweep before claiming JS pattern parity. +- Rerun the full 164-case Python pattern sweep before claiming both Python and + JavaScript pattern parity. - Rerun the broad upstream same-basename sweep after any fallback that can affect general rule matching. It is slow and quiet for long periods, but it caught previous overmatching. +- Keep the older focused guards from prior checkpoints for import/CommonJS, + singleton, catch/decorator/export, class/dependency, arrow/function, + equality/AC, and JSX/XML helpers. ## Operational Notes --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ -JS_PATTERN_CASE_REGEX ?= ^(ac_matching_bitor|aliasing_require|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|deep_vardef|dots_field_chaining|dots_importfrom|dots_jsx_attr|dots_jsx_body|dots_jsx_body2|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|less_class_complex|less_class_complex_2|less_inheritance|less_xml_attr|less_xml_body|metavar_class_fields|metavar_equality_vardef_vs_use|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_jsx_val|metavar_stmt_or_expr|metavar_template|metavar_typed_bool|metavar_typed_literal|metavar_xml_attr|misc_arrow|misc_arrow2|misc_asi_pattern|misc_catch_unbound|misc_decorators_method|misc_empty_body|misc_export|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|partial_catch|partial_decorators_arg_method|partial_decorators_method|partial_function|partial_lambda_or_func|partial_single_field|regexp_fieldname|switch_case_pattern|template_string_w_metavar|typed_metavar_assign|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar|unordered_xml_attr)$$ +JS_PATTERN_CASE_REGEX ?= ^(ac_matching_bitor|aliasing_require|caching_deep|caching_deep_metavar|caching_different_depth|caching_flat|caching_nested|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|deep_cond|deep_deep_stmt1|deep_deep_stmt2|deep_expr_and_ellipsis|deep_expr_vs_statement|deep_expr_xml|deep_stmt|deep_stmt2|deep_vardef|dots_field_chaining|dots_importfrom|dots_jsx_attr|dots_jsx_body|dots_jsx_body2|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|less_class_complex|less_class_complex_2|less_inheritance|less_xml_attr|less_xml_body|metavar_class_fields|metavar_equality_vardef_vs_use|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_jsx_val|metavar_stmt_or_expr|metavar_template|metavar_typed_bool|metavar_typed_literal|metavar_xml_attr|misc_arrow|misc_arrow2|misc_asi_pattern|misc_catch_unbound|misc_decorators_method|misc_deep_expr_stmt|misc_empty_body|misc_export|misc_method_chain_vs_stmt|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|partial_catch|partial_decorators_arg_method|partial_decorators_method|partial_function|partial_lambda_or_func|partial_single_field|regexp_fieldname|switch_case_pattern|template_string_w_metavar|typed_metavar_assign|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar|unordered_xml_attr)$$ .PHONY: all build generate test oracle patterns-oracle patterns-js-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -6303,6 +6303,1943 @@ (loop next (if finding (cons finding acc) acc))))))))) + (def (javascript-deep-sequence-pattern-token line) + (let* ([trimmed (js-cp-strip-trailing-semicolon line)] + [literal-match (re-search (re "^[0-9]+$") trimmed 0)] + [metavar-match (and (not literal-match) + (re-search + (re "^\\$([A-Za-z_][A-Za-z0-9_]*)$") + trimmed + 0))]) + (cond + [literal-match + (list (cons 'kind 'literal) (cons 'text trimmed))] + [metavar-match + (list + (cons 'kind 'metavariable) + (cons 'name (re-match-group metavar-match 1)))] + [else #f]))) + (def (javascript-deep-sequence-pattern-tokens lines) + (let loop ([remaining lines] [acc '()]) + (cond + [(null? remaining) (reverse acc)] + [else + (let ([token (javascript-deep-sequence-pattern-token + (car remaining))]) + (and token + (cond + [(null? (cdr remaining)) (reverse (cons token acc))] + [(and (string=? (cadr remaining) "...") + (not (null? (cddr remaining)))) + (loop (cddr remaining) (cons token acc))] + [else #f])))]))) + (def (javascript-deep-sequence-token-literal? token) + (eq? (alist-ref/default token 'kind #f) 'literal)) + (def (javascript-deep-sequence-pattern-spec pattern) + (let* ([lines (javascript-pattern-significant-lines + pattern)] + [tokens (and (>= (length lines) 3) + (javascript-deep-sequence-pattern-tokens + lines))]) + (and tokens + (>= (length tokens) 2) + (any? javascript-deep-sequence-token-literal? tokens) + (list (cons 'tokens tokens))))) + (def (javascript-horizontal-trim-end-exclusive + source + start + end) + (let ([last (skip-horizontal-backward source (- end 1))]) + (if (< last start) start (+ last 1)))) + (def (javascript-line-code-end source line-start line-end) + (let loop ([i line-start] [state 'normal] [escaped? #f]) + (cond + [(>= i line-end) + (javascript-horizontal-trim-end-exclusive + source + line-start + line-end)] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) 'double #f)] + [(char=? ch #\') (loop (+ i 1) 'single #f)] + [(char=? ch #\`) (loop (+ i 1) 'backtick #f)] + [(and (< (+ i 1) line-end) + (char=? ch #\/) + (char=? (string-ref source (+ i 1)) #\/)) + (javascript-horizontal-trim-end-exclusive + source + line-start + i)] + [else (loop (+ i 1) state #f)]))] + [escaped? (loop (+ i 1) state #f)] + [(char=? (string-ref source i) #\\) (loop (+ i 1) state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) 'normal #f)] + [(and (eq? state 'backtick) + (char=? (string-ref source i) #\`)) + (loop (+ i 1) 'normal #f)] + [else (loop (+ i 1) state #f)]))) + (def (javascript-line-brace-depth-after + source + line-start + line-end + depth) + (let ([code-end (javascript-line-code-end + source + line-start + line-end)]) + (let loop ([i line-start] + [current-depth depth] + [state 'normal] + [escaped? #f]) + (cond + [(>= i code-end) current-depth] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) current-depth 'double #f)] + [(char=? ch #\') (loop (+ i 1) current-depth 'single #f)] + [(char=? ch #\`) + (loop (+ i 1) current-depth 'backtick #f)] + [(char=? ch #\{) + (loop (+ i 1) (+ current-depth 1) state #f)] + [(char=? ch #\}) + (loop (+ i 1) (max 0 (- current-depth 1)) state #f)] + [else (loop (+ i 1) current-depth state #f)]))] + [escaped? (loop (+ i 1) current-depth state #f)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) current-depth state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) current-depth 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\'))