Advance semgrep parity fallbacks
ober
18bfc3de6f49a5d852f917e5391b31016a0366ef
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,20 +1,49 @@ -# Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity Work +# Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity Date: 2026-05-28 Workspace: `/Users/user/mine/jerboa-semgrep` -Upstream Semgrep checkout: `/Users/user/mine/semgrep` -Current Semgrep oracle binary: `/Users/user/.local/bin/semgrep` +Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` +Oracle Semgrep binary used in this session: `/Users/user/.local/bin/semgrep` +Primary source file: `src/semgrep/scan.ss` +Generated scan library: `lib/semgrep/scan.sls` -This document captures the exact state after the latest Codex session. The user -asked to continue toward parity with Semgrep, then interrupted and asked for a -commit plus this handoff. The project directory was not a git repository when -the commit request arrived, so Codex will initialize a new repository and make -an initial commit containing the current workspace after this document is -written. +The user asked to continue until this Jerboa implementation reaches parity with +Semgrep. They then asked to commit all work and create this handoff for Opus +4.8. Treat this repository as the active handoff state and continue from the +latest commit. -## Current Verification State +## Repository State -The latest verification commands run from `/Users/user/mine/jerboa-semgrep`: +This directory was not a git repository when the commit request arrived. Codex +initialized git and created the first commit: + +```text +5a6e332 Initial jerboa semgrep parity work +``` + +After that first commit, additional parity work was done in the same session. +Those changes are intended to be included in the commit that contains this +handoff. + +Files changed by the latest parity work: + +```text +src/semgrep/scan.ss +lib/semgrep/scan.sls +src/.jerbuild-hashes +tests/smoke.ss +HANDOFF_OPUS_4_8.md +``` + +`lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by `make test` +via Jerboa's build step. They were intentionally kept in the commit because the +repository already tracks generated library output. + +## Verification Snapshot + +All commands below were run from `/Users/user/mine/jerboa-semgrep`. + +Smoke tests: ```sh make test @@ -23,9 +52,11 @@ make test Result: ```text -162 tests, 162 passed, 0 failed +163 tests, 163 passed, 0 failed ``` +Local oracle suite: + ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle ``` @@ -36,6 +67,32 @@ Result: oracle: 42 passed, 0 failed ``` +Focused resolved/strict sweep: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(resolved_metavariables|relevant_rule_badutf8|strict_metavar_name_binding|no_strict_metavar_name_binding)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 4 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 4 compared +``` + +Focused symbolic-propagation sweep: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion|sym_prop_lambda|sym_prop_lhs_exp|sym_prop_merge|sym_prop_no_merge2|sym_prop_non_constant_exp|sym_prop_open_redirect|sym_prop_python_with|sym_prop_python_with1|sym_prop_react|sym_prop_record|sym_prop_redundancy|sym_prop_string_eq)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 11 passed, 5 mismatched, 0 jerboa errors, 0 current errors, 16 compared +``` + +First 160 upstream rules: + ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` @@ -43,29 +100,17 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 M Result: ```text -upstream-sweep: 129 passed, 29 mismatched, 0 jerboa errors, 2 current errors, 160 compared +upstream-sweep: 141 passed, 17 mismatched, 0 jerboa errors, 2 current errors, 160 compared ``` Current first-160 mismatch list: ```text -resolved_metavariables -sym_prop_chain sym_prop_decorator -sym_prop_exp sym_prop_explosion -sym_prop_lambda -sym_prop_lhs_exp -sym_prop_merge -sym_prop_no_merge2 -sym_prop_non_constant_exp sym_prop_open_redirect sym_prop_python_with sym_prop_python_with1 -sym_prop_react -sym_prop_record -sym_prop_redundancy -sym_prop_string_eq taint-general-source-sink taint_array taint_assign_cleans_shape @@ -80,452 +125,377 @@ taint_best_fit_sink taint_best_fit_sink10 ``` -The first-160 sweep improved from `127 passed, 31 mismatched` to `129 passed, -29 mismatched`. The isolated `relevant_rule_badutf8` and -`strict_metavar_name_binding` cases are no longer blockers. The remaining -isolated non-symbolic case is `resolved_metavariables`. - -Focused isolated check: +The first-160 count improved during this session from earlier snapshots of +127/31, then 129/29, then 130/28, to the current 141/17. The major eliminated +cluster was `resolved_metavariables` plus most basic `sym_prop_*` cases. -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(relevant_rule_badutf8|resolved_metavariables|strict_metavar_name_binding|no_strict_metavar_name_binding)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh -``` +## Important Current Deltas -Result: +The latest uncommitted-at-start delta before the final commit was approximately: ```text -upstream-sweep: 3 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 4 compared +lib/semgrep/scan.sls | 570 ++++++++++++++++++++++++++++++++++++++++++++++++++- +src/.jerbuild-hashes | 6 +- +src/semgrep/scan.ss | 551 ++++++++++++++++++++++++++++++++++++++++++++++++- +tests/smoke.ss | 12 ++ ``` -Remaining focused diff shape for `resolved_metavariables`: +The work is intentionally concentrated in `src/semgrep/scan.ss`. Avoid broad +refactors unless a parity fix truly needs them; this codebase currently uses +targeted fallbacks heavily. -- Expected but missing: `var res3 = baz()` at line 12. -- Correctly present: `res1`, `res2`, `res4`, `res5`. -- Extra false positives: `res6`, `res7`, `res8` at lines 27, 30, 33. +## Completed: Rule Option Lookup -## Important Files +`rule-option-enabled?` used to check only the literal key passed by callers. +Parsed YAML options are symbol keys such as `symbolic_propagation`, while the +scanner often queried string keys such as `"symbolic_propagation"`. -Core implementation: - -- `src/semgrep/scan.ss` -- `src/semgrep/match/structural.ss` -- `src/semgrep/rule.ss` -- `src/semgrep/rule/parse-rule.ss` -- `src/semgrep/parse/parse-target.ss` -- `src/semgrep/lang.ss` +Added: -Generated Chez Scheme output: +```scheme +(def (rule-option-entry rule key) + ...) +``` -- `lib/semgrep/*.sls` -- `lib/semgrep/*/*.sls` +`rule-option-enabled?` now accepts either string or symbol representations. This +matters for: -Tests and oracle drivers: +```text +symbolic_propagation +strict_mvar_name_binding +``` -- `tests/smoke.ss` -- `tests/oracle/run.sh` -- `tests/oracle/upstream-sweep.sh` -- `tests/oracle/cases/*` +This change is small but important. Without it, symbolic propagation was mostly +not running from parsed upstream YAML rules. -Planning/status: +## Completed: Resolved Metavariables -- `what.md` -- `HANDOFF_OPUS_4_8.md` +The upstream `resolved_metavariables` case now matches Semgrep in the focused +sweep. -## High-Level Architecture +Implemented an internal resolved-decomposition marker: -This is a Jerboa Scheme Semgrep-compatible scanner. It is not a full Semgrep -port yet. The current approach combines: +```text +internal-resolved-decomposition-name "__sg_resolved_decomposition" +resolved-decomposition-extra +finding-has-resolved-decomposition? +``` -- YAML rule parsing in `src/semgrep/rule/parse-rule.ss`. -- Rule record storage in `src/semgrep/rule.ss`. -- Tree-sitter parsing via `jerboa-treesitter` in `src/semgrep/parse/parse-target.ss`. -- Structural matching in `src/semgrep/match/structural.ss`. -- Scan orchestration, Semgrep formula handling, focus, metavariable filters, - selected taint support, and many compatibility fallbacks in - `src/semgrep/scan.ss`. -- JSON/SARIF/text output modules under `src/semgrep/output/`. -- CLI glue in `src/semgrep/cli.ss`. +The marker is hidden from public JSON metavariables by `internal-binding-entry?` +but preserved in finding extras as `resolved-decomposition-vars` so later +filters can distinguish artificial split bindings. -The matcher is broad enough for many Python/JavaScript/TypeScript/JSON search -rules but still uses targeted fallbacks for Semgrep behaviors that tree-sitter -plus a simple structural matcher do not naturally express. +Behavioral changes: -## Work Completed Before This Handoff +- `metavariable-comparison-satisfied?` rejects candidates carrying the internal + resolved-decomposition marker. This prevents false positives where an + artificial decomposition would make two metavariables compare equal. +- `javascript-resolved-call-pattern-spec` rejects repeated same artificial + metavariable forms such as `$PACKAGE.$PACKAGE()`. +- `dotted-name-pattern?` was loosened so simple names such as `foo` are accepted + by resolved-name `metavariable-pattern` checks. +- Added smoke coverage in `tests/smoke.ss`: `scan JavaScript resolved + metavariables`. -The prior part of the session already had: +The focused resolved/strict command now passes: -- `make test` green at 162 tests. -- Local oracle green at 42 cases. -- First-120 upstream sweep green except 2 current-Semgrep errors: - `118 passed, 0 mismatched, 0 Jerboa errors, 2 current errors, 120 compared`. -- First-160 sweep at `127 passed, 31 mismatched`. +```text +upstream-sweep: 4 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 4 compared +``` -Compatibility features already in place include: +## Completed: Symbolic Propagation Layer -- Regex-only rules without requiring AST parsing. -- Pattern, pattern-either, patterns, pattern-not, pattern-inside, - pattern-not-inside, pattern-regex. -- Rule focus-metavariable and focus clauses. -- Metavariable regex, pattern, type, name, analysis, and comparison support. -- Nested pattern formulas with scoped binding propagation. -- Basic Semgrep-style taint mode with sources, sinks, sanitizers, propagators, - labels, requires, exactness, side effects, and some implicit assignment - propagation. -- Python bare `$X` pattern fallback. -- Python exact `pattern: ...` fallback. -- Python constant f-string call fallback. -- JavaScript/TypeScript object record pattern fallbacks. -- JSX `style={...}` and object-style `style={{...}}` fallbacks. -- JavaScript arrow assignment fallback. -- Numeric comparison fix so numeric `==` and `!=` use numeric equality. -- Persistent inner bindings from selected metavariable-pattern results when a - later focus-metavariable needs them. +A first symbolic-propagation layer was added in `src/semgrep/scan.ss` after +`scan-javascript-record-pattern`. -## Latest Changes Since The 127/31 Sweep +New helper groups: -The latest isolated-fix attempt added or changed the following behavior. +```text +nonempty-findings +find-matching-close-paren +symbolic-identifier-start-char? +symbolic-call-head-char? +symbolic-call-head-start +make-symbolic-candidate +symbolic-call-candidates +symbolic-condition-candidates +symbolic-assignment-lhs-candidates +symbolic-expression-candidates +make-symbolic-assignment +symbolic-python-like-language? +symbolic-assignment-line-offset +symbolic-simple-assignment-pattern +symbolic-simple-assignments-before +symbolic-js-object-assignments-before +symbolic-assignments-before +symbolic-assignment-eligible? +python-symbolic-while-between? +symbolic-assignment-blocked? +symbolic-last-assignment-before +symbolic-identifier-occurrences +replace-range +symbolic-import-replacements +symbolic-expression-variants +symbolic-expression-matches-pattern? +finding-strictly-contains? +drop-containing-findings +scan-symbolic-propagation-pattern +``` + +The layer is currently intentionally conservative: + +- It only runs when `symbolic_propagation: true`. +- It supports Python/Python2/Python3/Py and JavaScript-like languages. +- It gathers candidate call expressions, Python `if` conditions, and assignment + left-hand sides. +- It expands simple assignments before the candidate. +- It expands JavaScript object literals assigned with `var`, `let`, or `const`. +- It expands import aliases through `name-import-map`. +- It chooses the latest assignment to a name; if the latest assignment is + ineligible, it does not fall back to an older assignment. +- Python assignments with deeper indentation than the candidate are ineligible. +- Python propagation is blocked across `while` lines at or above candidate + indentation. +- It drops wider symbolic findings when a narrower symbolic finding is present. + +Important matching detail: + +`symbolic-expression-matches-pattern?` now uses exact trimmed text equality or +`structural-text-root-matches?`. The root-only structural check was added to +avoid false positives where a pattern only matches a nested expression inside an +expanded candidate. This specifically fixed `sym_prop_redundancy`, where +`f(x)` expanded to `f(g(5))` but Semgrep reports only the direct `g(5)` RHS. + +## Completed: Empty Fallback Short-Circuit Fix + +`scan-python-loop-else-constant-pattern` used to return an empty list when it +found no matches. In Scheme, `()` is truthy, so the surrounding `or` chain +stopped before reaching symbolic propagation and structural matching. + +It now returns `#f` on no findings. This was necessary for +`sym_prop_redundancy` because the pattern `g(5)` looked like a numeric call +pattern to that fallback, but the fallback found no eligible loop/else match. + +There may be other old fallbacks with the same bug. When a later fallback or +structural matcher mysteriously does not run, check whether an earlier fallback +returns `()` rather than `#f`. + +## Symbolic Cases Now Passing + +The following upstream symbolic-propagation cases passed in the latest focused +sweep: -### Rule Options Are Now Parsed +```text +sym_prop_chain +sym_prop_exp +sym_prop_lambda +sym_prop_lhs_exp +sym_prop_merge +sym_prop_no_merge2 +sym_prop_non_constant_exp +sym_prop_react +sym_prop_record +sym_prop_redundancy +sym_prop_string_eq +``` -Files: +Useful fixed-case notes: -- `src/semgrep/rule.ss` -- `src/semgrep/rule/parse-rule.ss` -- generated `lib/semgrep/rule.sls` -- generated `lib/semgrep/rule/parse-rule.sls` +- `sym_prop_lhs_exp`: fixed by scanning assignment left-hand-side candidates, + so patterns like `foo.Bar.$X` can match `x.alice` and `y.alice` after + propagation. +- `sym_prop_redundancy`: fixed by allowing direct RHS `g(5)` to reach symbolic + or structural matching and by requiring root structural matches for symbolic + candidate variants. +- `sym_prop_merge`: fixed by latest-assignment behavior plus Python + indentation/while barriers. +- `sym_prop_lambda` and `sym_prop_react`: fixed by nested call scanning and + dropping wider containing findings. -The `rule` record now has an `options` field and accessor: +## Remaining Symbolic Mismatches -```scheme -rule-options -``` +### sym_prop_decorator -`parse-rule` stores the YAML `options:` mapping via: +Still missing: -```scheme -(alist-ref/default rule "options" '()) +```text +line 10 col 33-38, text False ``` -`src/semgrep/scan.ss` has: +Fixture: -```scheme -(def (rule-option-enabled? rule key) ...) +```text +../semgrep/tests/rules/sym_prop_decorator.py +../semgrep/tests/rules/sym_prop_decorator.yaml ``` -This was added to support `options.strict_mvar_name_binding`. - -### Strict JavaScript Imported-Bar Fallback Now Honors strict_mvar_name_binding - -File: - -- `src/semgrep/scan.ss` - -The existing fallback for: +Shape: -```yaml -pattern: $X.bar(...) -``` - -with: +```python +talisman = Talisman(app) -```js -import { bar } from 'foo' -bar('test') +@talisman(session_cookie_secure=False, session_cookie_http_only=False) +def index(): + ... ``` -is valid when strict metavariable name binding is off. Semgrep rejects it when: +Rule shape: ```yaml -options: - strict_mvar_name_binding: true -``` - -`scan-javascript-imported-bar-call-pattern` now checks: - -```scheme -(not (rule-option-enabled? rule "strict_mvar_name_binding")) +pattern: | + @flask_talisman.Talisman(...)(..., session_cookie_secure=$FALSE, ...) +metavariable-regex: + metavariable: $FALSE + regex: ^(False)$ +focus-metavariable: $FALSE ``` -This fixed upstream `strict_metavar_name_binding` while preserving -`no_strict_metavar_name_binding`. - -### Bad UTF-8 / CRLF Range Normalization - -File: +Likely next work: -- `src/semgrep/scan.ss` +- Add decorator candidates such as `@talisman(...)`. +- Expand `talisman` from `talisman = Talisman(app)`. +- Apply import alias expansion so `Talisman` becomes + `flask_talisman.Talisman`. +- Preserve/bind `$FALSE`; the current symbolic fallback creates findings with + empty bindings, so it cannot satisfy `metavariable-regex` plus focus on its + own. -Upstream fixture: +### sym_prop_explosion -- `/Users/user/mine/semgrep/tests/rules/relevant_rule_badutf8.js` -- raw bytes include invalid byte `0x97` and CRLF line endings. +Still missing: -Semgrep reports the match `foo` at byte offsets `98..101`. Tree-sitter reports -offsets after decoding the invalid byte as U+FFFD, which is 3 UTF-8 bytes. That -made Jerboa report `100..103` and a line-5 end. - -Added helpers: - -```scheme -char-utf8-byte-length -semgrep-source-byte-length -source-index->semgrep-offset -tree-byte-offset->source-index +```text +line 22 col 7-43 ``` -`finding-from-node` now: - -1. Converts tree-sitter byte offsets to source character indexes. -2. Converts those indexes to Semgrep-style byte offsets where U+FFFD counts as - one original byte. -3. Computes line/column from source character indexes, not tree-sitter columns. +Fixture: -This fixed `relevant_rule_badutf8`. +```text +../semgrep/tests/rules/sym_prop_explosion.js +../semgrep/tests/rules/sym_prop_explosion.yaml +``` -Risk and follow-up: +Likely next work: -- The conversion is currently applied in `finding-from-node`. -- Many manual regex/fallback findings still use string offsets directly. -- If non-ASCII or invalid-byte fixtures start failing outside structural node - findings, centralize offset mapping behind a small source-location abstraction - instead of copying this logic into each fallback. +- JavaScript assignment collection currently handles declarations such as + `var x = ...`, `let x = ...`, and `const x = ...`. +- This fixture uses reassignment shapes like `x = ff(...)`. +- Add JS reassignment history carefully, with the same "before candidate" + cutoff used by existing expansion to avoid self-recursive explosions. +- Keep the depth limit. The current expansion depth is 5. -### JavaScript Resolved Import Call Fallback +### sym_prop_python_with -File: +Still missing: -- `src/semgrep/scan.ss` +```text +line 4 col 7-18, text s.execute() +``` -Added: +Fixture: -```scheme -javascript-resolved-call-pattern-spec -scan-javascript-resolved-call-pattern* -scan-javascript-resolved-call-pattern -scan-javascript-resolved-call-pattern-with-bindings +```text +../semgrep/tests/rules/sym_prop_python_with.py +../semgrep/tests/rules/sym_prop_python_with.yaml ``` -This fallback targets upstream `resolved_metavariables`, especially source like: +Shape: -```js -import {bar as baz} from "foo"; -var res1 = baz() -var res2 = baz() -var res3 = baz() -var res4 = baz() -var res5 = baz() +```python +with Session() as s: + s.execute() ``` -and rule alternatives like: +Rule shape: ```yaml -pattern: var res1 = foo.bar() -pattern: var res2 = $ID() -pattern: var res3 = $PACKAGE.$ID() -pattern: var res4 = $PACKAGE.$ID() +pattern: Session().execute() +options: + symbolic_propagation: true ``` -It also wires the fallback into both: +Likely next work: -- `scan-positive-pattern-entry*` -- `scan-positive-pattern-entry-with-bindings*` +- Treat `with <expr> as <name>:` as a symbolic assignment active in the body. +- For this fixture, record `s -> Session()` before scanning body candidates. +- The current assignment scanner only sees regular `name = expr` lines. -### Resolved Name Candidates Include Dotted Prefixes +### sym_prop_python_with1 -File: +Still missing: -- `src/semgrep/scan.ss` +```text +line 11 col 24-76, focused SQL expression +``` -Added: +Fixture: -```scheme -dotted-prefix-candidates -candidates-with-dotted-prefixes -binding-name-candidates-with-prefixes +```text +../semgrep/tests/rules/sym_prop_python_with1.py +../semgrep/tests/rules/sym_prop_python_with1.yaml ``` -`metavariable-pattern-resolved-name-satisfied?` now uses -`binding-name-candidates-with-prefixes` instead of only exact candidates. +This is taint-mode shaped. The source builds: -`metavariable-regex-candidate-texts` now appends JavaScript and Python resolved -name candidates, including dotted prefixes. This lets a regex like `^foo` -match a binding whose source text is `baz` when `baz` resolves to `foo.bar`. +```python +Session = scoped_session(sessionmaker(bind=engine)) +with Session(engine) as s: + result = s.execute('SELECT ...' + source) +``` -This is why `resolved_metavariables` now gets `res4`. +The sink pattern is effectively: -Risk and follow-up: +```yaml +sqlalchemy.orm.scoped_session(...)(...).execute($SQL, ...) +``` -- This is intentionally a compatibility shortcut. -- It may be too broad for some future metavariable-regex cases because it - appends resolved candidates without knowing the candidate language. If later - Python or JS cases regress, pass the language into `metavariable-regex-apply` - instead of trying both import maps. +Likely next work: -## Current `resolved_metavariables` Diagnosis +- Implement `with` alias propagation as above. +- Confirm symbolic propagation is used while collecting taint sinks. Search + around `scan-taint-rule`, sink spec construction, and calls to + `scan-positive-pattern-entry*` / `scan-positive-pattern-entry-with-bindings*`. +- Preserve `$SQL` binding and focus range. Empty-binding symbolic findings will + not be sufficient for this case. -Rule: +### sym_prop_open_redirect -```sh -nl -ba ../semgrep/tests/rules/resolved_metavariables.yaml -nl -ba ../semgrep/tests/rules/resolved_metavariables.js -``` - -Fixture summary: - -```js -import {bar as baz} from "foo"; - -var res1 = baz() // should match literal resolved foo.bar() -var res2 = baz() // should match $ID + metavariable-pattern foo.bar -var res3 = baz() // should match $PACKAGE.$ID + mvar-pattern $PACKAGE: foo -var res4 = baz() // should match $PACKAGE.$ID + mvar-regex $PACKAGE: ^foo -var res5 = baz() // should match focused $PACKAGE - -var res6 = baz() // should NOT match comparison str($PACKAGE) == str($ID) -var res7 = baz() // should NOT match comparison $PACKAGE == $ID -var res8 = baz() // should NOT match repeated $PACKAGE.$PACKAGE() -``` - -Current Jerboa behavior after latest changes: - -- `res1`: matches. -- `res2`: matches. -- `res3`: missing. -- `res4`: matches. -- `res5`: matches. -- `res6`: false positive. -- `res7`: false positive. -- `res8`: false positive. - -Likely reason `res3` is missing: - -- `metavariable-pattern-resolved-name-satisfied?` only uses - `metavariable-pattern-dotted-name`. -- `metavariable-pattern-dotted-name` currently requires the expected pattern to - contain a dot via `dotted-name-pattern?`. -- The `res3` filter is: - - ```yaml - metavariable-pattern: - metavariable: $PACKAGE - pattern: | - foo - ``` - - `foo` is not dotted, so the resolved-name shortcut does not apply. It then - falls back to structural matching the source text `baz` against pattern `foo`, - which fails. - -Likely reason `res6`, `res7`, and `res8` are false positives: - -- The JavaScript resolved-call fallback decomposes a single imported alias - `baz` into both `$PACKAGE` and `$ID` by binding both to the same source range - and text. -- The comparison clauses then see equal source text and pass. -- Semgrep treats these artificial resolved-name decompositions differently for - comparisons and repeated metavariables. Upstream comments say these cases - exist to ensure artificial names are pretty-printed sensibly and not used as - ordinary single-token metavariable content. - -Suggested next fix: - -1. Extend the resolved-name shortcut to allow simple identifier expected names - such as `foo`, but only inside `metavariable-pattern` resolved-name checks. - Do not globally loosen `dotted-name-pattern?` if other code assumes dotted - names. -2. Mark artificial resolved decompositions in bindings or finding extra. A small - option is to add a private binding name such as - `__sg_artificial_resolved_decomposition` or a private extra key. -3. Make `metavariable-comparison-satisfied?` reject comparisons that reference - artificial decomposition bindings, at least when all referenced values are - coming from the same source token. -4. Make repeated metavariable equality reject artificial decomposition from the - same imported alias for patterns such as `$PACKAGE.$PACKAGE()`. -5. Rerun: - - ```sh - SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^resolved_metavariables$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh - make test - SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle - SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh - ``` - -## Current First-160 Roadmap - -Recommended order: - -1. Finish `resolved_metavariables`. -2. Tackle the symbolic propagation group as a group. -3. Tackle the taint group as a group. - -Do not jump directly into the taint cluster before understanding the symbolic -propagation cases. Several taint mismatches likely need the same underlying -constant/name/property propagation machinery. - -### Symbolic Propagation Cluster - -Current mismatches: +Still missing eight expected findings: ```text -sym_prop_chain -sym_prop_decorator -sym_prop_exp -sym_prop_explosion -sym_prop_lambda -sym_prop_lhs_exp -sym_prop_merge -sym_prop_no_merge2 -sym_prop_non_constant_exp -sym_prop_open_redirect -sym_prop_python_with -sym_prop_python_with1 -sym_prop_react -sym_prop_record -sym_prop_redundancy -sym_prop_string_eq +line 11 redirect(url) and inner url/request-derived range +line 23 redirect(url) and inner url/request-derived range +line 29 redirect(url) and inner url/request-derived range +line 36 redirect(url) and inner url/request-derived range ``` -Use focused commands like: +Fixture: -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +```text +../semgrep/tests/rules/sym_prop_open_redirect.py +../semgrep/tests/rules/sym_prop_open_redirect.yaml ``` -Read upstream fixtures: +There is already a targeted Django redirect fallback in `scan.ss`: -```sh -rg --files ../semgrep/tests/rules | rg 'sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion' -nl -ba ../semgrep/tests/rules/<case>.yaml -nl -ba ../semgrep/tests/rules/<case>.<ext> +```text +scan-python-django-redirect-pattern +python-redirect-finding-for-assignment ``` -Likely needed abstraction: - -- A real, per-language symbolic environment for local constant/name/property - propagation before matching metavariable filters. -- It should model at least: - - Simple assignment aliases. - - Object/record field aliases. - - Python `with` aliases. - - Decorator expressions. - - Lambda and expression aliases. - - String equality / constant propagation. - - Merge/no-merge semantics where Semgrep intentionally does or does not - combine paths. - -Current helpers that may be relevant: +Likely next work: -- `constant-bindings-before` -- `binding-resolved-for-comparison` -- `comparison-bindings-with-constants` -- `name-import-map` -- `binding-name-candidates` -- `binding-name-candidates-with-prefixes` -- JavaScript object/record fallbacks in `scan-javascript-record-pattern`. +- The fallback may not fire for this full upstream formula shape, or it may only + fire for simpler nested smoke cases. +- Compare the exact parsed rule shape with the existing smoke test + `scan nested metavariable-pattern Django redirect`. +- Decide whether to make the existing redirect fallback formula-aware or to add + a targeted bridge for the upstream rule's nested `metavariable-pattern` + structure. -### Taint Cluster +## Taint Mismatch Cluster -Current mismatches: +After symbolic work, the first-160 mismatch list is now dominated by taint: ```text taint-general-source-sink @@ -542,163 +512,113 @@ taint_best_fit_sink taint_best_fit_sink10 ``` -Use focused commands like: +Do not start broad taint rewrites blindly. First reproduce one focused diff: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_basic|taint_array|taint_assign_record)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_basic$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -Core taint implementation is in `src/semgrep/scan.ss` around: - -- `scan-taint-source-matches` -- `scan-taint-specs` -- `scan-taint-propagators` -- `source-state-reaches-finding?` -- `source-state-reaches-sink-spec-shape?` -- `sanitizer-blocks?` -- `expand-taint-sources` -- `propagator-applies?` -- `propagated-source` -- `scan-taint-rule` - -The smoke suite already has many taint regression tests. Add focused smoke -tests for any semantic change before broadening taint behavior. - -## Commands To Use - -Build only: +Then inspect the upstream YAML/source pair: ```sh -make build +nl -ba ../semgrep/tests/rules/taint_basic.yaml +nl -ba ../semgrep/tests/rules/taint_basic.py ``` -Smoke tests: +The taint implementation already has a substantial set of smoke tests and +oracle cases passing. Preserve those while filling specific parity gaps. -```sh -make test -``` +## Useful Commands -Local oracle: +Standard verification: ```sh +make test SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle ``` -First 120 upstream gate: +Focused upstream sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=120 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^CASE_NAME$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -First 160 upstream gate: +First-160 progress check: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -Focused upstream case: +Symbolic cluster: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^case_name$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion|sym_prop_lambda|sym_prop_lhs_exp|sym_prop_merge|sym_prop_no_merge2|sym_prop_non_constant_exp|sym_prop_open_redirect|sym_prop_python_with|sym_prop_python_with1|sym_prop_react|sym_prop_record|sym_prop_redundancy|sym_prop_string_eq)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -Focused upstream group: +Resolved/strict guardrail: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(case1|case2|case3)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(resolved_metavariables|relevant_rule_badutf8|strict_metavar_name_binding|no_strict_metavar_name_binding)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -Find upstream fixture paths: +Direct Jerboa CLI invocation for one rule: ```sh -rg --files ../semgrep/tests/rules | rg '<case-name>' +JERBOA_TREE_SITTER_LIB=/Users/user/mine/jerboa-treesitter \ +DYLD_LIBRARY_PATH=/Users/user/mine/jerboa-treesitter \ +LD_LIBRARY_PATH=/Users/user/mine/jerboa-treesitter \ +/Users/user/mine/jerboa/.chez/bin/scheme \ + --libdirs "lib:/Users/user/mine/jerboa-treesitter/lib:/Users/user/mine/jerboa/lib" \ + --script bin/semgrep.ss \ + scan --json --config ../semgrep/tests/rules/CASE.yaml ../semgrep/tests/rules/CASE.py ``` -Read fixtures: - -```sh -nl -ba ../semgrep/tests/rules/<case>.yaml -nl -ba ../semgrep/tests/rules/<case>.<ext> -``` +Use the right extension for non-Python cases (`.js`, `.ts`, `.tsx`, `.json`, +`.generic`, etc.). ## Coding Notes -- Prefer small targeted changes with focused oracle checks. -- Keep `make test` green after each semantic change. -- Run `SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle` before broad - sweeps. - Use `apply_patch` for manual edits. -- Generated `lib/` files are produced by `make build`; include them if this - repository is committing generated output. -- The existing code often returns `#f` for an inapplicable fallback and a list - for an applicable fallback. Be careful with Scheme truthiness: `'()` is true. - A fallback that returns an empty list inside an `or` chain will suppress later - fallbacks. Several existing fallbacks already account for this; new fallbacks - should return `#f` when not applicable and avoid returning `'()` unless it is - the intended final answer. -- Be cautious with broad resolved-name shortcuts. They can make one Semgrep - fixture pass while introducing artificial binding behavior that later filters - treat as ordinary source text. -- For user-visible findings, Semgrep-compatible ranges are byte offsets. The - current code has a mix of tree-sitter byte offsets and Scheme string indexes. - The latest bad-UTF fix only addresses structural node findings. - -## Known Fragile Areas