Cover JavaScript import equivalence fixtures
ober
2abb07085c8d54d05984ee340419bbcf34369351
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,25 +1,26 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 22:35 MDT +Date: 2026-05-29 23:18 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`0109749 Cover JavaScript object field fixtures` +`bb63b69 Cover JavaScript template literal fixtures` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. Do not treat this handoff as completion. This handoff records -the clean Python `tests/patterns/python/*.sgrep` frontier and the promoted -JavaScript `tests/patterns/js/*.sgrep` -constant-propagation/equivalence/object-field/template-literal slice. +Semgrep parity. Do not treat this handoff as completion. This checkpoint closes +the JavaScript import/CommonJS equivalence fixture cluster and promotes it into +the JS oracle target. -Current headline: +## Current Headline - Full Python pattern sweep: 164 passed / 0 mismatched. -- Full JavaScript pattern sweep: 69 passed / 63 mismatched, improved from - 63 passed / 69 mismatched by closing the JS template-literal slice. +- Full JavaScript pattern sweep: 82 passed / 50 mismatched, improved from + 69 passed / 63 mismatched by closing the 13-case import/equivalence cluster. +- Promoted JavaScript oracle slice: 41 passed / 0 mismatched. - Broad same-basename upstream sweep: 437 passed / 0 mismatched / 0 Jerboa errors, with 3 current-Semgrep oracle errors. +- Smoke suite: 292 tests, 292 passed, 0 failed. ## Project Contract @@ -31,10 +32,10 @@ Current headline: - Preserve Semgrep-compatible scan behavior and output formats. - Keep the result embeddable in Jerboa applications. -The current implementation is still a compatibility work in progress. The full -Python `.sgrep` pattern fixture slice is now green, but Semgrep parity also -requires broader rule, taint, target-selection, parser, output, autofix, and CLI -compatibility beyond this slice. +The implementation is still a compatibility work in progress. Python `.sgrep` +pattern fixtures are green, but Semgrep parity also requires broader rule, +taint, target-selection, parser, output, autofix, and CLI compatibility beyond +the promoted fixture slices. ## Files In This Checkpoint @@ -49,9 +50,9 @@ src/.jerbuild-hashes tests/smoke.ss ``` -`lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by `make build` -and `make test`, but they are tracked in this repository and should be -committed with the matching source changes. +`lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by +`make build`/`make test`, but they are tracked and should be committed with the +matching source changes. Expected post-commit state: @@ -65,100 +66,43 @@ should show a clean `main` branch. All commands below were run from `/Users/user/mine/jerboa-semgrep`. -Smoke suite: - -```sh -make test -``` - -Result: - -```text -289 tests, 289 passed, 0 failed -``` - -Focused constant/dataflow slice: +Build: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(constprop_dataflow|cp_exception|cp_label|cp_rlval|cp_eval2)$' LIST_MISMATCHES=1 MAX_DIFFS=10000 tests/oracle/patterns-sweep.sh +make build ``` Result: ```text -patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared +jerbuild: 14 processed, 0 skipped, 0 errors (of 14 total) ``` -Promoted curated upstream Python pattern oracle: +Focused JavaScript import/CommonJS equivalence slice: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_aliasing_import|aliasing_require|dots_importfrom|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2)$' LIST_MISMATCHES=1 MAX_DIFFS=100 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 116 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 116 compared -``` - -Full upstream Python pattern sweep: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh -``` - -Result: - -```text -patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 164 compared -``` - -Local oracle: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle -``` - -Result: - -```text -oracle: 42 passed, 0 failed -``` - -Whitespace check: - -```sh -git diff --check -``` - -Result: no output and exit code 0. - -Focused JavaScript object/destructuring slice: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(misc_object_directly|object_duplicate_key|object_numeric_key|unordered_fields|unordered_fields2|unordered_metavar|unordered_destructuring)$' LIST_MISMATCHES=1 MAX_DIFFS=100 tests/oracle/patterns-sweep.sh +patterns-sweep: 13 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 13 compared ``` -Result: - -```text -patterns-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 7 compared -``` - -Focused broad-regression fix: +Smoke suite: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=20 CASE_REGEX='^misc_var_is_var$' tests/oracle/upstream-sweep.sh +make test ``` Result: ```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared +292 tests, 292 passed, 0 failed ``` -Promoted JavaScript constant-propagation/equivalence/object-field/template -oracle: +Promoted JavaScript oracle: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle @@ -167,37 +111,37 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-js-oracle Result: ```text -patterns-sweep: 28 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 28 compared +patterns-sweep: 41 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 41 compared ``` Full upstream JavaScript pattern sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/patterns-sweep.sh ``` Result: ```text -patterns-sweep: 69 passed, 63 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +patterns-sweep: 82 passed, 50 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared ``` -Focused JavaScript template-literal slice: +Local oracle: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=js CASE_REGEX='^(dots_template_literals|ellipsis_metavar_template|ellipsis_metavar_tagged_template|metavar_template|template_string_w_metavar|misc_tagged_template_literal)$' LIST_MISMATCHES=1 MAX_DIFFS=100 tests/oracle/patterns-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle ``` Result: ```text -patterns-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 6 compared +oracle: 42 passed, 0 failed ``` Broad same-basename upstream sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=20 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep tests/oracle/upstream-sweep.sh ``` Result: @@ -210,26 +154,39 @@ The three current-side errors are packaged-Semgrep schema failures for `anywhere_global`, `anywhere_include`, and `anywhere_metavar`, not Jerboa mismatches. +Whitespace check: + +```sh +git diff --check +``` + +Result: no output and exit code 0. + ## What Changed -The curated `PATTERN_CASE_REGEX` in `Makefile` was expanded from 112 to 116 -cases by promoting the last four Python pattern mismatches: +`JS_PATTERN_CASE_REGEX` in `Makefile` now promotes these 13 additional JS +fixture cases: ```text -constprop_dataflow -cp_exception -cp_label -cp_rlval +aliasing_require +dots_importfrom +equivalence_aliasing_import +equivalence_import_require +equivalence_import_variations +equivalence_import_variations2 +equivalence_import_variations3 +equivalence_import_variations4 +metavar_import +metavar_import2 +metavar_import3 +metavar_importfrom +metavar_importfrom2 ``` -The full Python pattern sweep also includes `cp_eval2`; it remains unpromoted -in `PATTERN_CASE_REGEX`, but was explicitly regression-checked because the new -wildcard string propagation initially overmatched it after `s += argv[1]`. - -`JS_PATTERN_CASE_REGEX` and the `patterns-js-oracle` Make target now cover the -promoted JS pattern slice below: +The full promoted JS oracle now covers 41 cases: ```text +aliasing_require cp_conditional cp_dataflow cp_implicit_conversion @@ -238,16 +195,28 @@ cp_label cp_throw cp_undeclared cp_undeclared1 +dots_importfrom dots_template_literals ellipsis_metavar_tagged_template ellipsis_metavar_template +equivalence_aliasing_import +equivalence_import_require +equivalence_import_variations +equivalence_import_variations2 +equivalence_import_variations3 +equivalence_import_variations4 +equivalence_keyword_args +equivalence_varlet infer_const infer_const_1 infer_const_regexp equivalence_constant_propagation2 equivalence_constant_propagation3 -equivalence_keyword_args -equivalence_varlet +metavar_import +metavar_import2 +metavar_import3 +metavar_importfrom +metavar_importfrom2 metavar_template misc_object_directly misc_tagged_template_literal @@ -260,165 +229,84 @@ unordered_fields2 unordered_metavar ``` -`scan-javascript-varlet-equivalence-pattern` and the new declaration object -fallbacks now respect `options: let_is_var: false`; the broad guardrail -initially exposed an overmatch in `tests/rules/misc_var_is_var.js`, where -`var $X = 1;` must not match `let z = 1`. - ## Implementation Map -All new scanner logic is in `src/semgrep/scan.ss` and generated into +All scanner logic is in `src/semgrep/scan.ss` and generated into `lib/semgrep/scan.sls`. -New constant/dataflow support: - -- Added a small block-aware Python must-constant environment: - `python-cp-flow-analyze-range`, `python-cp-flow-expression-value`, and - supporting environment merge helpers. -- The environment keeps exact constants when all paths agree, keeps an - `any-string` sentinel when all paths are known strings but not the same - literal, and drops bindings when a value is not a must constant. -- It skips unrelated function/class bodies while still allowing globals to flow - into the current function, which is needed for `cp_rlval`. -- It handles `if`/`else` must merges: - `if c: a = "a" else: a = "b"` is enough for `foo("...")`, but a branch-only - assignment without `else` is not a must constant after the branch. -- It handles `while` conservatively by invalidating names assigned in the loop - body, preventing string constants from being used inside or after loop - mutation. -- It handles minimal `try`/`except`/`else`/`finally` flow for `cp_exception`: - direct `raise` takes the exception path, `try: pass` takes `else`, calls named - `any_function_call_may_raise()` are treated as may-raise, and `finally` - overwrites the merged state. -- It invalidates augmented assignments such as `s += argv[1]`; this was needed - to preserve the negative `cp_eval2` fixture. - -New scanner fallbacks: - -- `scan-python-cp-wildcard-string-call-pattern` - uses the block-aware environment for wildcard string-call patterns such as - `foo("...")` and `eval("...")`. -- `scan-python-cp-self-equality-pattern` - handles `$X == $X` by resolving both sides through the block-aware constant - environment. -- `scan-python-cp-bare-string-pattern` - handles ASCII bare string-literal patterns such as `"foo"` with direct literal - matches, continuation/comment content ranges, and propagated expression uses. - It intentionally declines non-ASCII patterns so Unicode literals continue - through the structural path that preserves Semgrep byte offsets. - -Wiring is in the Python fallback chain near the end of `scan.ss`, before the -older exact constant-propagation fallbacks. - -JavaScript constant-propagation support already in the current frontier: - -- `scan-javascript-cp-pattern-fallbacks` is wired into the JavaScript `pattern` - path before structural matching. -- The JS CP evaluator handles single/double/backtick literals, template - interpolations whose embedded expressions are constant, regexp literals, - numbers, booleans, `null`, `undefined`, `+` concatenation/coercion, and - ternary must-string merges. -- The line-oriented JS environment handles `var`/`let`/`const` declarations, - undeclared assignments, reassignment, increment/decrement invalidation, - augmented assignment invalidation, single-element array destructuring - invalidation, and simple unreachable lines after `throw`/`return` until a - closing block line. -- New JS fallbacks cover wildcard/exact call arguments such as - `console.log("...")`, `require('...')`, `api("...")`, and `new RegExp(/a/)`; - wildcard string subscripts such as `value = obj["..."]`; wildcard/exact - return patterns; `$X == $X` / `$X === $X` with propagated constants; and - direct or propagated bare string-regex patterns such as `"=~/http:///"`. -- The string-regex fallback emits Semgrep-compatible ranges for template - expressions and concatenation prefixes in `infer_const_1`. -- JS CP call/literal scanning ignores line comments so examples in comments, - such as `new RegExp(/a/)`, do not become false positives. - -JavaScript equivalence support already in the current frontier: - -- `scan-javascript-varlet-equivalence-pattern` matches `var $X = ...` patterns - against `var`, `let`, and `const` declarations with Semgrep-compatible - declaration ranges, unless the rule explicitly sets `let_is_var: false`. -- `scan-javascript-object-call-equivalence-pattern` matches object argument - fields independent of order and allows extra fields, covering - `foo({kwd2:$X, kwd1:$Y})` against `foo({kwd1:1, kwd2:2, kwd3:3})`. - -JavaScript object/destructuring support added in this checkpoint: - -- `scan-javascript-object-pattern-fallbacks` is wired into the JavaScript - `pattern` path after the object-call equivalence fallback and before JS - constant propagation. -- `javascript-field-records` parses comma-separated object/destructuring - members while respecting nested brackets/braces/parens and quoted strings. -- Object key normalization treats quoted and unquoted keys such as `"bar"` and - `bar` as equal, and treats numeric keys such as `0x1`, `1`, and `1.0` as - equal when their numeric value matches. -- Direct object literal matching covers required fields with extra fields - allowed, including `{ fld1: $X }` against `{ fld2: 4, fld1: 5 }`. -- Duplicate-key matching covers the upstream `$X: $A ... $X: $B` pattern for - string and numeric key equivalence. -- Declaration object matching covers `var $F = { ... }` with unordered - required fields, extra fields allowed, fixed values, and metavariable field - names such as `$FLD: 2`. -- Destructuring declaration matching covers `var { one, two } = foo;` against - declarations where the required names appear in any order with additional - names allowed. -- Declaration-style object/destructuring fallbacks use the same - `let_is_var: false` option check as var/let equivalence so rule-level - semantics are preserved. - -JavaScript template-literal support added in this checkpoint: - -- `scan-javascript-template-literal-pattern` is wired into the JavaScript - `pattern` path after object fallbacks and before JS constant propagation. -- The fallback parses simple backtick pattern forms and scans matching source - template literals while skipping `${...}` expression bodies correctly enough - for the upstream fixture shapes. -- Assignment patterns such as ``$X = `...${...}...` `` match assignment ranges - from the bound left-hand side through Semgrep's template range endpoint. -- Bare template patterns such as `` `$LEFT${...}$RIGHT` `` and - `` `$...LEFT${...}$...RIGHT` `` match template literals with interpolations, - including the Semgrep-compatible range that excludes the closing backtick. -- Tagged template patterns such as ``$T`...` `` match from the tag start through - the Semgrep-compatible template endpoint and bind the tag metavariable. -- Interpolation metavariable patterns such as `` `...${$X}...` `` bind the - expression content and follow Semgrep's narrower endpoint before the closing - interpolation brace. -- Parenthesized template-call patterns such as - ``prisma.$executeRaw(`...`)`` match calls whose single argument is a template - literal, while intentionally not matching tagged template calls. +General range fix: + +- `semicolon-trimmable-node?` now treats `import_statement` like variable and + lexical declarations, trimming a trailing semicolon from structural import + declaration findings. This fixes the one-byte range mismatches for existing + structural matches such as `dots_importfrom`, `metavar_import`, and + `metavar_importfrom2`. + +New JavaScript import/equivalence fallback: + +- `scan-javascript-import-equivalence-pattern` is wired into both JavaScript + plain pattern paths, after `scan-javascript-imported-bar-call-pattern` and + before imported-member/resolved-call fallbacks. +- The fallback is pattern-shape gated; it only handles import declarations, + import/use sequences, and `child_process.execSync(...)`. +- Source import line parsing handles default imports, named imports, + namespace imports, default-plus-named imports, default-plus-namespace imports, + side-effect imports, quoted modules, and package subpaths such as + `module-name/path`. +- Source CommonJS parsing handles direct `const x = require("module")` and + destructured `const { x, y: z } = require("module")` declarations. +- Module pattern specs support literal modules, `"..."`, unquoted module + metavariables such as `$X`, and quoted module metavariables such as `"$X"`. +- Named import specs support unordered literal requirements, extra source + imported names, aliases, and metavariable names such as `$X1`. +- Side-effect pattern `import "module-name";` now matches all Semgrep fixture + variants in `equivalence_import_variations`, including static import forms + and the two `require("module-name")` declarations. +- Named import patterns such as `import { export1 } from "module-name";` and + `import {x, y} from "module-name";` match required names in any order, with + aliases and extras allowed. +- Namespace pattern `import * as $X from "module-name";` matches both + `import * as name ...` and `import defaultExport, * as name ...`. +- Default import patterns such as `import $X from 'foo';` and module + metavariable patterns such as `import jwt_decode from $X;` now produce the + Semgrep-compatible import declaration ranges. + +Import/use sequence support: + +- `import $X from 'foo'; ... $X.sign(...);` matches from the import declaration + start through the method call endpoint, matching `metavar_import2`. +- `import { $X } from 'dependency'; ... $X(...);` binds the called imported + name and matches from the import declaration start through the expression + statement endpoint, matching `metavar_import3`. + +`child_process.execSync(...)` alias equivalence: + +- Named ESM import aliases are resolved: + `import { execSync } from 'child_process'; execSync("ls")`. +- CommonJS destructuring aliases are resolved: + `const { execSync } = require(...)` and + `const { execSync: es } = require(...)`. +- CommonJS namespace aliases are resolved: + `const cp = require('child_process'); cp.execSync("ls")`. +- Direct alias call matching explicitly rejects property access so a previous + direct `execSync` alias does not also match the `execSync` segment inside + `cp.execSync(...)`. +- The direct `require('child_process').execSync("ls")` fixture line remains + intentionally unmatched, matching upstream Semgrep's current fixture output. ## Smoke Coverage Added -`tests/smoke.ss` extends `scan Python constant propagation pattern fixture -fallbacks` with checks for: - -- Branch-sensitive wildcard string propagation. -- Loop mutation invalidation for wildcard string calls. -- `try`/`except`/`else` constant equality. -- Continuation/comment string content ranges for `"foo"`. -- R/L-value propagation from `a = "foo"` into `b = a`, `c[a]`, dict values, - and `return a`. - -The current smoke suite includes a top-level smoke case, -`scan JavaScript constant propagation pattern fixture fallbacks`, covering -template/concat string calls, conditional string calls, subscript propagation, -throw-unreachable return propagation, and increment invalidation. - -The current smoke suite includes a top-level smoke case, -`scan JavaScript equivalence pattern fixture fallbacks`, covering var/let/const -declaration equivalence and object-call field-order equivalence. +`tests/smoke.ss` now includes: -This checkpoint adds a new top-level smoke case, -`scan JavaScript object field pattern fixture fallbacks`, covering direct object -field matching, numeric key normalization, duplicate keys, unordered object -declarations, and unordered destructuring declarations. +- `scan JavaScript import declaration equivalence`: verifies side-effect import + patterns match default import, named subpath import, and direct CommonJS + require declarations. +- `scan JavaScript import sequence metavariable`: verifies the default import + plus `$X.sign(...)` sequence range. +- `scan JavaScript child_process import aliases`: verifies ESM named import, + CommonJS destructuring alias, and CommonJS namespace alias calls. -This checkpoint adds a new top-level smoke case, -`scan JavaScript template literal pattern fixture fallbacks`, covering -assignment-to-template patterns, bare template metavariables, interpolation -metavariables, tagged templates, and parenthesized template-call patterns. - -The smoke suite count is now 289. +The smoke suite count is now 292. ## Current Frontier @@ -429,65 +317,104 @@ patterns-sweep: 164 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 s ``` The full JavaScript `.sgrep` pattern fixture sweep is not clean yet, but the -promoted JS constant-propagation/equivalence/object-field/template-literal slice -is green: +import cluster is closed and promoted: + +```text +patterns-sweep: 82 passed, 50 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +``` + +Remaining JS mismatches after this checkpoint: ```text -patterns-sweep: 69 passed, 63 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 132 compared +ac_matching_bitor +caching_deep +caching_deep_metavar +caching_different_depth +caching_flat +caching_nested +deep_cond +deep_deep_stmt1 +deep_deep_stmt2 +deep_expr_and_ellipsis +deep_expr_vs_statement +deep_expr_xml +deep_stmt +deep_stmt2 +deep_vardef +dots_field_chaining +dots_jsx_attr +dots_jsx_body +dots_jsx_body2 +less_class_complex +less_class_complex_2 +less_inheritance +less_xml_attr +less_xml_body +metavar_class_fields +metavar_equality_vardef_vs_use +metavar_jsx_val +metavar_stmt_or_expr +metavar_typed_bool +metavar_typed_literal +metavar_xml_attr +misc_arrow +misc_arrow2 +misc_asi_pattern +misc_catch_unbound +misc_decorators_method +misc_deep_expr_stmt +misc_empty_body +misc_export +misc_method_chain_vs_stmt +partial_catch +partial_decorators_arg_method +partial_decorators_method +partial_function +partial_lambda_or_func +partial_single_field +regexp_fieldname +switch_case_pattern +typed_metavar_assign +unordered_xml_attr ``` -Do not mark the overall goal complete from this alone. Next high-value work: - -1. Continue the JavaScript pattern frontier. The next concentrated clusters - from the full JS sweep are deep/caching expression matching, import - equivalence/import metavariable variations, JSX/XML dots/less/metavariable - range parity, and arrow/class/partial pattern shapes. -2. Keep `cp_array_destructure` as a regression guard: the JS equality fallback - must invalidate constants after destructuring reassignment and avoid - matching `imageOrNull == null`. -3. Keep `misc_var_is_var` as a regression guard for `let_is_var: false` when - touching declaration equivalence or declaration object/destructuring - fallbacks. -4. Keep the six promoted template fixtures as a range regression guard when - touching template-literal scanning, especially `template_string_w_metavar`, - whose Semgrep range ends before the closing interpolation brace. -5. Decide whether to promote all 164 Python pattern fixtures into - `PATTERN_CASE_REGEX` or keep `make patterns-oracle` at the curated 116-case - runtime. -6. Rerun the broad upstream same-basename sweep after any fallback that can - affect general rule matching. +Next high-value clusters: + +1. Deep/caching expression and statement matching: + `caching_*`, `deep_*`, `misc_deep_expr_stmt`, `misc_method_chain_vs_stmt`. +2. JSX/XML range and less/dots/metavariable parity: + `dots_jsx_*`, `less_xml_*`, `metavar_xml_attr`, `unordered_xml_attr`. +3. Arrow, partial, class, decorator, and export shapes: + `misc_arrow*`, `partial_*`, `less_class_*`, `metavar_class_fields`, + `misc_export`, `misc_decorators_method`. +4. Typed metavariable and field-name regex support: + `metavar_typed_*`, `typed_metavar_assign`, `regexp_fieldname`. + +## Regression Guards To Keep + +- Rerun the focused 13-case import slice after touching import resolution, + declaration ranges, or CommonJS alias logic. +- Rerun `make patterns-js-oracle`; it is now the 41-case promoted JS guard. +- Keep `cp_array_destructure` as a regression guard for JS constant equality + invalidation after destructuring reassignment. +- Keep `misc_var_is_var` as a regression guard for `let_is_var: false` when + touching declaration equivalence or declaration object/destructuring + fallbacks. +- Keep the six promoted template fixtures as range guards when touching + template-literal scanning, especially `template_string_w_metavar`. +- Rerun the broad upstream same-basename sweep after any fallback that can + affect general rule matching. It is slow and quiet for long periods, but it + caught previous overmatching. ## Operational Notes - Use `rg` first for source discovery. - Prefer `apply_patch` for manual edits. - Do not revert unrelated user changes in a dirty tree. -- `make build` can pass even when a scanner fallback has a runtime invalid - definition-context error. Always run the focused oracle for newly touched - paths. -- The upstream pattern sweep is quiet for long periods. A silent run usually - means no mismatches have printed yet. +- `make build` can pass even when a scanner fallback has a runtime definition + issue. Always run a focused oracle for newly touched paths. - `SEMGREP_CURRENT=/Users/user/.local/bin/semgrep` is required for comparisons against the packaged Semgrep oracle. -- The sibling Semgrep checkout at `/Users/user/mine/semgrep` provides the - upstream fixtures. Python pattern fixtures live under - `/Users/user/mine/semgrep/tests/patterns/python/`. - -## Last Known Broad Guardrail - -The latest checkpoint reran the broad same-basename upstream sweep: - -```text -upstream-sweep: 437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared -``` - -The three current errors were packaged-Semgrep oracle schema failures for: - -```text -anywhere_global -anywhere_include -anywhere_metavar -``` - -Use the full upstream sweep after broad fallback changes; it is slow but is the -best current guard against rule-level overmatching. +- The sibling Semgrep checkout at `/Users/user/mine/semgrep` provides upstream + fixtures. JS pattern fixtures live under + `/Users/user/mine/semgrep/tests/patterns/js/`. --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib PATTERN_CASE_REGEX ?= ^(ac_matching_dots|ac_matching_dots1|ac_matching_explosion|ac_matching_explosion1|ac_matching_explosion2|ac_matching_explosion3|ac_matching_free|ac_matching_free1|ac_matching_if|ac_matching_if1|ac_matching_mvars|ac_matching_mvars1|ac_matching_mvars2|aliasing_attribute|assoc_matching_bug|assoc_matching_dots|assoc_matching_dots1|assoc_matching_explosion|assoc_matching_explosion1|assoc_matching_explosion2|assoc_matching_explosion3|assoc_matching_free|assoc_matching_free1|assoc_matching_mvars|assoc_matching_mvars1|constprop_dataflow|constprop_used_twice|cp_concat|cp_exception|cp_label|cp_label1|cp_method_call|cp_python_mult_string|cp_python_mult_string1|cp_rlval|cp_string_mvar|cp_strings|cp_with|cp_yield|deep_cond|deep_expr_operator|deep_expr_vs_statement|deep_exprstmt|deep_fake_dataflow|deep_try|dict_ellipsis|dots_attribute|dots_expr_plus|dots_expr_plus_string|dots_fstring|dots_fstring_with_match_stmt|dots_inherit|dots_list|dots_nested_stmts|dots_stmts|equivalence_constant_propagation|equivalence_f_string|equivalence_f_string_2|equivalence_f_string_3|equivalence_f_string_4|equivalence_interpolated_str|equivalence_interpolated_str2|equivalence_keyword_args|equivalence_naming_import|import_metavar|import_negatives|import_negatives2|imports|index_tuple|less_attributes|less_catch|less_catch_multiple|less_inherits|less_typehint|less_typehint2|matching_if_expr|metavar_anno|metavar_anno_fqn|metavar_class_def|metavar_dict|metavar_equality_param_vs_use|metavar_equality_var|metavar_func_def|metavar_stmt|metavar_typed|misc_attributes|misc_block_import|misc_class_regression|misc_dots_stmts|misc_except_matching|misc_faketok2|misc_if1|misc_metavar_vs_fstring|misc_naming_bug2|misc_paren|misc_range_statements|misc_regression1|misc_regression2|misc_stmts1|misc_tuple2|misc_with_parens|multi_import|multi_qualified_wildcard|pip614-extended-decorator-grammer1|regexp_string_backref|set_vs_dict|set_vs_dict2|standalone_decorator|standalone_decorator_single_arg|static_method|stmts_to_fields|tuple_subscript_ellipsis|tuple_subscript_ellipsis_with_match_stmt|unordered_metavar2|untyped_vs_typed|wildcard_qualified)$$ -JS_PATTERN_CASE_REGEX ?= ^(cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|metavar_template|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|template_string_w_metavar|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ +JS_PATTERN_CASE_REGEX ?= ^(aliasing_require|cp_conditional|cp_dataflow|cp_implicit_conversion|cp_incrdecr|cp_label|cp_throw|cp_undeclared|cp_undeclared1|dots_importfrom|dots_template_literals|ellipsis_metavar_tagged_template|ellipsis_metavar_template|equivalence_aliasing_import|equivalence_import_require|equivalence_import_variations|equivalence_import_variations2|equivalence_import_variations3|equivalence_import_variations4|equivalence_keyword_args|equivalence_varlet|infer_const|infer_const_1|infer_const_regexp|equivalence_constant_propagation2|equivalence_constant_propagation3|metavar_import|metavar_import2|metavar_import3|metavar_importfrom|metavar_importfrom2|metavar_template|misc_object_directly|misc_tagged_template_literal|object_duplicate_key|object_numeric_key|template_string_w_metavar|unordered_destructuring|unordered_fields|unordered_fields2|unordered_metavar)$$ .PHONY: all build generate test oracle patterns-oracle patterns-js-oracle clean --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -415,7 +415,8 @@ (loop (cdr xs))))))))) (def (semicolon-trimmable-node? node) (or (string=? (node-type node) "variable_declaration") - (string=? (node-type node) "lexical_declaration"))) + (string=? (node-type node) "lexical_declaration") + (string=? (node-type node) "import_statement"))) (def (trim-node-finding-end source node start end) (if (and (semicolon-trimmable-node? node) (> end start) @@ -3395,6 +3396,1054 @@ (finding-from-match rule path source match) acc))) (reverse acc)))))))) + (def (js-identifier-name? text) + (let ([len (string-length text)]) + (and (> len 0) + (identifier-start-char? (string-ref text 0)) + (let loop ([i 1]) + (or (= i len) + (and (identifier-char? (string-ref text i)) + (loop (+ i 1)))))))) + (def (js-strip-trailing-semicolon text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (if (and (> len 0) + (char=? (string-ref trimmed (- len 1)) #\;)) + (string-trim (substring trimmed 0 (- len 1))) + trimmed))) + (def (js-quoted-text-inner text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (and (>= len 2) + (let ([first (string-ref trimmed 0)] + [last (string-ref trimmed (- len 1))]) + (and (or (char=? first #\") (char=? first #\')) + (char=? first last) + (substring trimmed 1 (- len 1))))))) + (def (js-metavariable-name text) + (let ([trimmed (string-trim text)]) + (and (> (string-length trimmed) 1) + (char=? (string-ref trimmed 0) #\$) + (substring trimmed 1 (string-length trimmed))))) + (def (js-import-module-pattern-spec text) + (let* ([trimmed (js-strip-trailing-semicolon text)] + [inner (js-quoted-text-inner trimmed)] + [unquoted-name (js-metavariable-name trimmed)]) + (cond + [inner + (let ([quoted-name (js-metavariable-name inner)]) + (cond + [(string=? inner "...") (list (cons 'kind 'ellipsis))] + [quoted-name + (list + (cons 'kind 'quoted-metavariable) + (cons 'name quoted-name))] + [else (list (cons 'kind 'literal) (cons 'module inner))]))] + [unquoted-name + (list + (cons 'kind 'metavariable) + (cons 'name unquoted-name))] + [else #f]))) + (def (js-module-name-matches? expected actual) + (or (string=? expected actual) + (sg-string-prefix? (string-append expected "/") actual))) + (def (js-module-pattern-matches? spec actual) + (case (alist-ref/default spec 'kind #f) + [(ellipsis metavariable quoted-metavariable) #t] + [(literal) + (js-module-name-matches? + (alist-ref/default spec 'module "") + actual)] + [else #f])) + (def (js-pattern-name-spec text) + (let* ([trimmed (string-trim text)] + [name (js-metavariable-name trimmed)]) + (if name + (list (cons 'kind 'metavariable) (cons 'name name)) + (list (cons 'kind 'literal) (cons 'name trimmed))))) + (def (js-pattern-name-spec-metavariable? spec) + (eq? (alist-ref/default spec 'kind #f) 'metavariable)) + (def (js-pattern-name-spec-literal-name spec) + (and (eq? (alist-ref/default spec 'kind #f) 'literal) + (alist-ref/default spec 'name ""))) + (def (js-pattern-name-spec-binding spec source item) + (and (js-pattern-name-spec-metavariable? spec) + (let ([name (alist-ref/default spec 'name "")] + [start (alist-ref/default item 'start #f)] + [end (alist-ref/default item 'end #f)]) + (and start + end + (cons + name + (metavariable-binding-for-range + name + source + start + end)))))) + (def (js-parse-pattern-name-specs items-text) + (let loop ([xs (split-on-char items-text #\,)] [acc '()]) + (if (null? xs) + (reverse acc) + (let-values ([(imported alias) (split-alias (car xs))]) + (loop + (cdr xs) + (cons (js-pattern-name-spec imported) acc)))))) + (def (js-source-named-item imported local start end) + (list + (cons 'imported imported) + (cons 'local local) + (cons 'start start) + (cons 'end end))) + (def (js-source-named-item-for-local + source + item + search-start) + (let-values ([(imported alias) (split-alias item)]) + (let* ([local (or alias imported)] + [local-start (and (not (string=? local "")) + (string-find-substring-from + source + local + search-start))] + [local-end (and local-start + (+ local-start (string-length local)))]) + (and (not (string=? imported "")) + (not (string=? local "")) + local-start + local-end + (js-source-named-item + imported + local + local-start + local-end))))) + (def (js-source-named-items source items-text items-start) + (let loop ([xs (split-on-char items-text #\,)] + [search-start items-start] + [acc '()]) + (if (null? xs) + (reverse acc) + (let* ([item (js-source-named-item-for-local + source + (car xs) + search-start)] + [next (if item + (alist-ref/default item 'end search-start) + search-start)]) + (loop (cdr xs) next (if item (cons item acc) acc)))))) + (def (js-source-destructure-item source item search-start) + (let* ([trimmed (string-trim item)] + [colon (string-find-substring trimmed ":")] + [imported (if colon + (string-trim (substring trimmed 0 colon)) + trimmed)] + [local (if colon + (string-trim + (substring + trimmed + (+ colon 1) + (string-length trimmed))) + imported)] + [local-start (and (not (string=? local "")) + (string-find-substring-from + source + local + search-start))] + [local-end (and local-start + (+ local-start (string-length local)))]) + (and (not (string=? imported "")) + (not (string=? local "")) + local-start + local-end + (js-source-named-item + imported + local + local-start + local-end)))) + (def (js-source-destructure-items + source + items-text + items-start) + (let loop ([xs (split-on-char items-text #\,)] + [search-start items-start] + [acc '()]) + (if (null? xs) + (reverse acc) + (let* ([item (js-source-destructure-item + source + (car xs) + search-start)] + [next (if item + (alist-ref/default item 'end search-start) + search-start)]) + (loop (cdr xs) next (if item (cons item acc) acc)))))) + (def (js-find-named-block text) + (let ([open (string-find-substring text "{")]) + (and open + (let ([close (string-find-substring-from + text + "}" + (+ open 1))]) + (and close (cons open close)))))) + (def (js-import-module-from-text text) + (let ([match (re-search + (re "from[ \\t]+['\"]([^'\"]+)['\"][ \\t]*$") + text + 0)]) + (and match (re-match-group match 1)))) + (def (js-import-module-from-side-effect text) + (let ([match (re-search + (re "^import[ \\t]+['\"]([^'\"]+)['\"][ \\t]*$") + text + 0)]) + (and match (re-match-group match 1)))) + (def (js-import-namespace-name text) + (let ([match (re-search + (re "(^|,|import[ \\t]+)[ \\t]*\\*[ \\t]+as[ \\t]+([A-Za-z_$][A-Za-z0-9_$]*)") + text + 0)]) + (and match (re-match-group match 2)))) + (def (js-import-default-name text) + (let* ([tail (and (sg-string-prefix? "import " text) + (substring text 7 (string-length text)))] + [from-index (and tail + (string-find-substring tail " from "))] + [before-from (and from-index (substring tail 0 from-index))] + [comma (and before-from + (string-find-substring before-from ","))] + [head (and before-from + (string-trim + (if comma + (substring before-from 0 comma) + before-from)))]) + (and head + (not (string=? head "")) + (not (sg-string-prefix? "{" head)) + (not (sg-string-prefix? "*" head)) + (js-identifier-name? head) + head))) + (def (js-source-import-info source line-start line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [end (line-trimmed-end-before-semicolon + source + first + line-end)] + [text (substring source first end)] + [side-module (js-import-module-from-side-effect text)] + [from-module (js-import-module-from-text text)] + [module (or side-module from-module)]) + (and module + (sg-string-prefix? "import " text) + (let* ([named-block (js-find-named-block text)] + [named-items (if named-block + (js-source-named-items + source + (substring + text + (+ (car named-block) 1) + (cdr named-block)) + (+ first (car named-block) 1)) + '())] + [default (js-import-default-name text)] + [default-start (and default + (string-find-substring-from + source + default + first))] + [namespace (js-import-namespace-name text)] + [namespace-start (and namespace + (string-find-substring-from + source + namespace + first))]) + (list (cons 'kind 'import) (cons 'start first) (cons 'end end) + (cons 'module module) (cons 'default default) + (cons 'default-start default-start) + (cons 'namespace namespace) + (cons 'namespace-start namespace-start) + (cons 'named named-items)))))) + (def (js-source-require-info source line-start line-end) + (let* ([first (line-first-nonspace + source + line-start + line-end)] + [end (line-trimmed-end-before-semicolon + source + first + line-end)] + [text (substring source first end)] + [module-match (re-search + (re "require[ \\t]*\\([ \\t]*['\"]([^'\"]+)['\"][ \\t]*\\)")