Clear default-parameter taint frontier
ober
370470da517ec33460d4fd0b7c08a6c0421ec6ab
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,16 +1,14 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 00:51 MDT local workspace time +Date: 2026-05-29 01:22 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Primary source file: `src/semgrep/scan.ss` -Generated library output: `lib/semgrep/scan.sls` +Current pre-handoff HEAD: `a1201fb Clear typed request taint frontier` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. Continue from this state. Do not restart discovery from -scratch; the current mismatch frontier and the exact commands below are the -working map. +Semgrep parity. Continue from this state. Do not restart broad discovery from +scratch; use the current frontier and commands below as the working map. ## Project Goal @@ -26,31 +24,9 @@ This is a compatibility-preserving Semgrep implementation in Jerboa, not a similar scanner. High-risk parity areas remain structural matching, taint/dataflow, path and target semantics, autofix, and output schemas. -## Current Commit Lineage +## Files In This Checkpoint -Recent checkpoints before this handoff commit: - -```text -8b8005b Clear Express SQL taint frontier -38bfcf1 Clear destructuring taint frontier -19f45f7 Advance lambda and source taint parity -7cfb751 Clear lambda taint label reach -31db293 Clear taint label branch frontier -34227c0 Clear final global taint frontier -268a616 Clear field-sensitive taint frontier -24796db Advance field-sensitive taint parity -971b714 Advance taint exception and field parity -29dd690 Clear taint exact source frontier -7beb039 Advance taint control parity -e429d34 Advance taint safe function parity -70a1319 Advance taint best-fit parity handoff -87b6beb Clear first upstream taint parity window -424af58 Advance semgrep parity and handoff -18bfc3d Advance semgrep parity fallbacks -5a6e332 Initial jerboa semgrep parity work -``` - -The commit containing this document should include: +This handoff commit should include: ```text HANDOFF_OPUS_4_8.md @@ -61,7 +37,7 @@ tests/smoke.ss ``` `lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by -`make test`; they are tracked and should be committed with source changes. +`make test`; they are tracked and must be committed with source changes. ## Verified State @@ -76,7 +52,7 @@ make test Result: ```text -196 tests, 196 passed, 0 failed +200 tests, 200 passed, 0 failed ``` Local oracle: @@ -91,73 +67,31 @@ Result: oracle: 42 passed, 0 failed ``` -Focused request-host, route-template, destructuring, and assignment-only taint -guardrail for this checkpoint: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_object_funcdef|taint_object_funcdef_1|taint_nested_record_pattern|taint_no_builtin_props|taint_no_builtin_props1|taint_object_destructure)$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh -``` - -Result: - -```text -upstream-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 6 compared -``` - -This confirms `taint_object_funcdef`, `taint_object_funcdef_1`, -`taint_nested_record_pattern`, `taint_no_builtin_props`, -`taint_no_builtin_props1`, and -`taint_object_destructure` are clean in the current tree. - -Focused request-host and callback-token cases cleared in this checkpoint: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_object_funcdef_1|taint_propagator1)$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh -``` - -Result: - -```text -upstream-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 2 compared -``` - -Focused upstream guardrail containing the field-sensitive cases fixed so far: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_field_sensitive1|taint_field_sensitive2|taint_field_sensitive3|taint_field_sensitive4|taint_field_sensitive6|taint_field_sensitive7|taint_field_sensitive8)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Result: - -```text -upstream-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 7 compared -``` - -Nearby regression guardrail: +Focused default-parameter family: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_field_sensitive2|taint_field_sensitive1|taint_exact_sources)$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_param_default[0-9]*$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared +upstream-sweep: 4 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 4 compared ``` -Focused label guardrail cleared in the previous checkpoint: +Focused spread-record-small guardrail: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_labels2|taint_labels3|taint_typestate|taint_source_requires|taint_source_requires_reopen)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_spread_record_small$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` -Current full upstream sweep: +Full upstream sweep: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh @@ -166,724 +100,125 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 218 passed, 21 mismatched, 0 jerboa errors, 2 current errors, 241 compared +upstream-sweep: 223 passed, 16 mismatched, 0 jerboa errors, 2 current errors, 241 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner -errors. Use `MAX_CASES=220` when a first-window guardrail is needed; this -command intentionally ran the full sorted 241-case upstream rule sweep. +errors. Use `MAX_CASES=220` when a shorter first-window guardrail is enough; +the command above intentionally ran the full sorted 241-case sweep. ## What Changed In This Checkpoint -This checkpoint clears `taint_object_funcdef_1` and `taint_propagator1`, -building on `8b8005b` which cleared `taint_object_funcdef`. - -Implementation changes in `src/semgrep/scan.ss`: - -- Built-in JavaScript/TypeScript implicit assignment propagation now includes - typed declaration patterns: `const $L: $T = $R`, `let $L: $T = $R`, and - `var $L: $T = $R`. Without these, the generic `$L = $R` fallback matched - `const host: string = httpOptions.hostname` with `$L=string`, so taint never - reached the real alias `host`. -- Token-taint reach into sinks containing function values is now more precise. - The old guard rejected any token source when the sink text contained `=>`, - which blocked request-option objects that had an unrelated nested callback. - The new guard computes arrow/function body ranges and blocks only when the - source token occurrences are solely inside those function bodies. -- This clears the request-host shape in `taint_object_funcdef_1` and also - clears `taint_propagator1`, where tainted tokens outside nested callbacks - should still reach the enclosing sink. - -New smoke coverage in `tests/smoke.ss`: - -```text -scan TypeScript taint typed hostname into request options -scan JavaScript taint Express params into SQL templates -scan JavaScript taint nested object destructuring assignment -scan taint only propagates through plain assignments -scan JavaScript taint destructured arrow object parameters -scan TypeScript taint combines typed lambda source labels -scan taint focus on source reports original source -scan JavaScript taint combines lambda db and request labels -scan JavaScript taint focused header value under fetch -scan taint labels keep sibling if else assignments separate -scan Python taint filters impossible exception branches -scan taint field source reaches opaque base sink -scan taint whole object source honors field cleans -scan taint ancestor field clean clears descendant sources -scan taint dynamic index field source reaches compatible paths -scan taint by-side-effect field source reaches containing sink -``` - -The field-clean test mirrors the upstream `taint_field_sensitive2` shape: -`x = source` taints `sink(x)`, `sink(x.b)`, `sink(x.b.c)`, and `sink(x.c[j])`, -while clean assignments to `x.a` and `x.c[i].d` suppress only those field -subtrees. The ancestor-clean test mirrors `taint_field_sensitive4`: -`x.a = safe` clears earlier descendant sources under `x.a`. - -## Resolved Recent Cases - -`taint_object_funcdef_1` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_funcdef_1$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -Target files: - -```text -/Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.yaml -/Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.ts -``` - -Rule shape: - -```yaml -pattern-sources: - - pattern: httpOptions.hostname - - pattern: opts.hostname -pattern-sinks: - - patterns: - - pattern: | - (...).request({...}) - - pattern-not: - pattern: | - (...).request({ - beforeRedirect: ($OPTS) => { - ... - sanitizeIP($OPTS.hostname) - ... - }, - ..., - lookup: getLookupFunction(..., sanitize) - }) -options: - symbolic_propagation: true -``` - -Relevant target shape: - -```typescript -const httpOptions = urlToHttpOptions(reqUrl); -const host: string = httpOptions.hostname; - -const req: http.ClientRequest = - (reqUrl.protocol.startsWith('https') ? https: http).request({ - host, - lookup: getLookupFunction(logger, Boolean(opts.useRoundRobinDns), sanitize), - beforeRedirect: (opts) => { - // no sanitizeIP call - }, - }) -``` - -Expected finding range is the whole request call expression from line 8, -column 33 to line 16, column 3. - -What changed: - -- `pattern: httpOptions.hostname` and `pattern: (...).request({...})` already - matched independently. -- The missing dataflow was the typed alias - `const host: string = httpOptions.hostname`; implicit assignment now binds - `$L=host` and `$R=httpOptions.hostname`. -- The broad request-call sink contains an unrelated nested arrow callback. - Token reach no longer rejects the whole sink just because `=>` appears in - the sink text; it only rejects tokens that occur solely inside function body - ranges. - -`taint_propagator1` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_propagator1$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -This was cleared by the same token-reach refinement. In the previous full -sweep it missed three expected JavaScript findings in -`/Users/user/mine/semgrep/tests/rules/taint_propagator1.js`; the current full -sweep no longer lists it as a mismatch. - -`taint_object_funcdef` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_funcdef$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -Target file: - -```text -/Users/user/mine/semgrep/tests/rules/taint_object_funcdef.js -``` - -Rule shape: - -```yaml -pattern-sources: - - patterns: - - pattern-either: - - pattern-inside: $APP.$METHOD(..., function $FUNC($REQ, $RES) {...}) - - metavariable-regex: - metavariable: $METHOD - regex: ^(get|post|put|head|delete|options)$ - - pattern: $REQ.params -pattern-sinks: - - patterns: - - pattern-either: - - patterns: - - pattern-either: - - pattern: | - `$SQLSTR${$EXPR}...` - - metavariable-regex: - metavariable: $SQLSTR - regex: .*\b(?i)(select|delete|insert|create|update|alter|drop)\b.* - - focus-metavariable: $EXPR -``` - -Relevant target shape: - -```javascript -app.get('/ok', async (req, res) => { - const tainted = req.params; - async.parallel({ - somefunc(cb) { - return `SELECT * FROM ${tainted}` - } - }) -}) -``` - -Expected matched lines are now exactly: - -```text -6, 17 -``` - -What changed: - -- Simple `pattern: $REQ.params` already bound `$REQ=req`; the missing source - side was the Express callback `pattern-inside` context and the `$METHOD` - binding needed by the route-method regex. -- The generic function-parameter `pattern-inside` fallback was proving enough - containment to short-circuit the route-specific fallback, but it did not - bind `$METHOD`. The route fallback now runs first for this pattern shape. -- The sink side needed template-literal support for - `` `$SQLSTR${$EXPR}...` `` plus an embedded `(?i)` regex flag. The template - fallback binds `$SQLSTR="SELECT * FROM "` and `$EXPR=tainted`; the later - focus clause reports the `tainted` expression at columns 31 to 38. - -`taint_nested_record_pattern` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_nested_record_pattern$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_nested_record_pattern.js`. -- Shape: - `const { body: { param } } = tainted; sink(param)`. -- Jerboa already supported flat JS object destructuring. The implicit - assignment pattern list now also includes nested shorthand object patterns - for `const`, `let`, and `var`. -- The propagated binding is the nested leaf identifier, so `param` receives - taint from the RHS object while unrelated names stay clean. - -`taint_no_builtin_props` and `taint_no_builtin_props1` are now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_no_builtin_props|taint_no_builtin_props1)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 2 compared -``` - -What changed: - -- Both rules use `options: taint_only_propagate_through_assignments: true`. -- Jerboa previously reported expected assignment-only flows plus false - positives through expression RHS values such as `y + "ok"`, `f(y)`, - `a[y]`, `x.a`, and `x()`. -- Built-in implicit assignment propagators are now tagged and blocked under - this option unless the RHS binding text exactly matches the current source - finding text. -- Explicit user `pattern-propagators` are not tagged as built-in implicit - assignments and are left to their own rule semantics. - -`taint_object_destructure` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_destructure$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_object_destructure.js`. -- Rule source shape: - `({ $X }, $A) => $Z` focused to `$X`. -- Jerboa previously returned no source matches for destructured object - parameters in arrow functions. -- The new JS arrow object-parameter fallback finds object properties in the - first destructured parameter and emits one match per property. Focusing `$X` - produces sources for `x` and `y`, which then reach `sink(x)` and `sink(y)`. -- The fallback intentionally does not export the sibling `$A` binding in the - resulting source finding; exporting `$A=z` caused `sink(z)` to become tainted. - -`taint_lambda4` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_lambda4$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_lambda4.ts`. -- Shape: - `pattern-inside: function ... ($REQ) {...}` plus `pattern: $REQ.query`, - and a second source requiring `__SOURCE__` with - `pattern: $X.$Y. ... + foobar` focused to `$X`. -- Jerboa previously found the sink and simple `$REQ.query` source, but not the - two full source formulas. -- The new JS/TS inside fallback validates that the focused `$REQ` occurrence is - inside a function or arrow body whose parameter list contains the same - identifier, including typed params such as `req: Request`. -- The new member-chain-plus fallback matches `req.query + foobar`, binds - `$X=req` and `$Y=query`, then focus narrows the source to `req` so the - `EDGECASE` label can combine with `__SOURCE__`. - -Expected matched lines for this case are now exactly: - -```text -3, 9 -``` - -`taint_match_on_source` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_match_on_source$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_match_on_source.py`. -- Rule option: `taint_focus_on: source`. -- Jerboa already detected the tainted flow but reported the sink at line 8. - Packaged Semgrep reports the original `source()` call at line 3. -- Taint states now keep an `origin` finding. Propagated states inherit the - origin from the source state that produced them. -- Sink output now checks `taint_focus_on: source`; when enabled and the sink's - label requirements are satisfied, it returns the origin source findings - instead of the sink finding. - -Expected matched line for this case is now exactly: - -```text -3 -``` - -`taint_lambda2` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_lambda2$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -Rule: - -```yaml -rules: -- id: test - message: Test - severity: ERROR - languages: [javascript, typescript] - mode: taint - pattern-sources: - - label: TAINT - pattern: req.query - - label: DB - pattern: client(...) - pattern-sinks: - - patterns: - - pattern-either: - - pattern: $DB.sink($SINK) - requires: TAINT and DB -``` - -Target file: - -```text -/Users/user/mine/semgrep/tests/rules/taint_lambda2.js -``` - -Important target shape: - -```javascript -let db; -foo(() => { - db = client(); -}); - -bar((req, res) => { - db.sink(req.query.id); -}); -``` - -Why it used to fail: - -- The source `client(...)` gave `db` the `DB` label through assignment - propagation. -- The source `req.query` also matched structurally inside the sink expression. -- The sink required `TAINT and DB`, but the `req.query` source sat inside the - sink range after the sink start, so the reachability gate rejected it before - normal source/sink compatibility could combine the labels. - -Why it passes now: - -- Non-token access-path sources can reach compatible containing sink - expressions. -- `req.query` is compatible with the containing sink argument `req.query.id`, - so `TAINT` combines with the `DB` label on `db` for `$DB.sink($SINK)`. -- Token sources still use the stricter containment branch, preserving the - existing typestate and side-effect self-hit behavior. - -`taint_exact_sources` was cleared in `29dd690`: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_exact_sources$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh -``` - -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -Why it used to fail: - -- Jerboa had false positives at lines 23 and 41 in - `/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py`. -- A focused function parameter source tainted `params`. -- Clean indexed assignments such as `params["sql"] = "select xyz from table"` - and `params["name"] = "test"` did not kill the broad parameter source. -- A first attempt to kill those assignments over-blocked expected flows at - lines 7 and 33. - -Why it passes now: - -- Clean indexed assignments kill the earlier broad `params` taint. -- The tainting assignment in `fn1` re-taints `params` from its own RHS because - same-range duplicate implicit propagators no longer kill each other. -- The `for res in results:` loop in `fn3` propagates the tainted `results` - value into `res`, which then re-taints `params` before line 33. - -Expected matched lines for this case are now exactly: +This checkpoint clears the upstream `taint_param_default*` family and also +clears `taint_spread_record_small`, reducing the full upstream frontier from: ```text -7, 28, 33, 49, 60, 69 +218 passed, 21 mismatched, 0 jerboa errors, 2 current errors ``` -`taint_exception` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_exception$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh -``` - -Current result: +to: ```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -What changed: - -- Jerboa previously reported all expected lines plus false positives at - 29, 35, 74, 80, 106, 122, and 179. -- Those false positives came from assignments in impossible `except` or `else` - branches. -- The new Python taint branch filter removes only trivial impossible branches: - `try: pass` suppresses `except`, `try: raise`/`return` suppresses `else`, - and `any_function_call_may_raise()` remains unknown. -- Expected `finally` propagation around lines 180 to 196 remains intact. - -`taint_field_sensitive1` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive1$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh +223 passed, 16 mismatched, 0 jerboa errors, 2 current errors ``` -Current result: - -```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -What changed: - -- Jerboa previously missed line 32, `sink(x)`, after `x.a`, `x.c`, and `x.d` - were tainted. -- The new opaque-base compatibility lets a member source such as `x.a` taint - a whole-object sink argument `x`. -- The compatibility is prefix-bounded, so `x.a` does not taint unrelated - `sink(x.b)`. +Implementation changes in `src/semgrep/scan.ss`: -`taint_field_sensitive2` is now cleared: +- `scan-symbolic-propagation-pattern` now declines broad structural patterns: + bare metavariables such as `$URL` and string wildcard literals such as + `"..."` or `'...'`. +- This matters because `symbolic_propagation: true` was hijacking + `pattern-inside: "..."` in `taint_param_default.yaml`, returning one + propagated variable instead of all string literals. The source formula then + lost the four `http://` literals Semgrep reports. +- Focused `fix-regex` findings now drop the `fix` extra when + `focus-metavariable` changes the reported range. Current Semgrep preserves + template `fix:` under focus, but omits `fix-regex`; `taint_param_default` + exposed this after the ranges were fixed. +- Built-in Python implicit assignment propagation now includes: -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive2$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh -``` + ```text + def $F(..., $L = $R, ...): + ... + ``` -It is also included in the verified focused field-sensitive guardrail above. + This lets taint flow from default parameter values into the function body. +- Built-in JavaScript/TypeScript implicit assignment propagation now includes: -What changed: + ```text + function $F(..., $L = $R, ...) { + ... + } + ``` -- Jerboa previously reported no findings for - `/Users/user/mine/semgrep/tests/rules/taint_field_sensitive2.js`. -- The broad source `x = source` was killed completely by later clean member - assignments `x.a = safe` and `x.c[i].d = safe`. -- The access-path kill logic now keeps the base `x` source available for - `sink(x)`, `sink(x.b)`, `sink(x.b.c)`, and `sink(x.c[j])`. -- The same kill logic suppresses only the cleaned subtrees: `sink(x.a)`, - `sink(x.a.b)`, `sink(x.c[j].d)`, and `sink(x.c[j].d.e)`. + This clears JS default-parameter propagation in `taint_param_default3`. +- Token taint can now reach a containing sink when the source is an unfocused + bare source with no metavariables and the sink syntactically contains it. + This clears `def test(x=sink(taint))` in `taint_param_default1` and also + the spread-record-small shape `sink({...source})`. +- That token-in-sink allowance is intentionally narrow. It requires + `(null? (finding-metavars source))`, so focused by-side-effect typestate + sources such as `lock($L)` do not report their own source call as a sink. -Expected matched lines for this case are now exactly: +New smoke coverage in `tests/smoke.ss`: ```text -8, 10, 12, 14 -``` - -`taint_field_sensitive4` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive4$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh +scan Python taint bare source inside default sink +scan Python taint default parameter reaches session request +scan JavaScript taint default parameter propagation +focus-metavariable omits fix-regex ``` -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_field_sensitive4.js`. -- Shape: - `x.a.b = source; x.a.c = source; ...; x.a = safe; sink(x.a.b); - sink(x.a.c); sink(x.a.b.x)`. -- Jerboa previously kept reporting descendants after the ancestor clean - assignment at `x.a = safe`. -- The kill predicate now also handles the inverse relation from - `taint_field_sensitive2`: when the clean assignment path is an ancestor of - the source path, the source is dead for later sinks. -- This fixed false positives at lines 17, 19, and 21 while preserving the - expected pre-clean findings at lines 8, 10, and 12. - -`taint_field_sensitive3` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive3$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` +These add to the existing taint and symbolic propagation guardrails from prior +checkpoints. -What changed: +## Cleared Cases -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_field_sensitive3.js`. -- Jerboa previously missed the dynamic-index exact and descendant sinks at - lines 11 and 13: - `sink(x.d.e[k].f.g[l].h)` and `sink(x.d.e[k].f.g[l].h.i)`. -- The source path `x.d.e[i].f.g[j].h` now parses to an access path with - wildcard dynamic-index segments, so it is compatible with the sink paths - using `k` and `l`. -- Sibling paths remain separated; `sink(x.a.b.d)` and `sink(x.a.c)` stay clean. - -Expected matched lines for this case are now exactly: +The following cases are clean in this checkpoint: ```text -7, 9, 11, 13, 19, 21, 23, 25, 27, 29 -``` - -`taint_field_sensitive6` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive6$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_field_sensitive6.py`. -- Shape: - `x = source(); x.a = safe; exotic(x.a); sink(x.a)`. -- The sink spec includes a focused `exotic($X.$A)` sink whose finding range is - `$X`. Semgrep treats `exotic(x.a)` as a finding because the actual focused - sink is `x`, and `x` remains tainted even though `x.a` was cleaned. -- The new field-aware kill keeps base `x` tainted while clearing only `x.a`, - so `exotic(x.a)` reports and `sink(x.a)` stays clean. - -`taint_field_sensitive8` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive8$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_field_sensitive8.py`. -- Shape: - `x = source(); x.a = sanitize(); sink(x); x.a.i = source(); sink(x.a.i); - sink(x.a); sink(x)`. -- The sanitizer assignment should clear only `x.a`, not the base `x`. -- The later member source `x.a.i = source()` should re-taint `x.a.i`, make the - ancestor `x.a` reachable again, and keep `sink(x)` reportable. - -`taint_field_sensitive7` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_field_sensitive7$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_field_sensitive7.py`. -- Shape: - `x = safe; sink(x.a.b); sink(x.a)` with a by-side-effect source pattern - `$X.b` focused to `$X`. -- Jerboa previously found only the later `sink(x.a)` at line 9. -- Semgrep reports line 5 as well because matching `$X.b` inside `sink(x.a.b)` - focuses and taints `x.a`, and the containing sink argument `x.a.b` can reach - that focused field source. -- The implementation permits this only for strict source-path prefixes inside a - wider sink argument. This keeps equal focused self-hit cases like `lock(l)` - and `f.close()` from reporting on the same call. - -`taint_final_globals1` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_final_globals1$' LIST_MISMATCHES=1 MAX_DIFFS=240 tests/oracle/upstream-sweep.sh -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_final_globals1.js`. -- Shape: - `let hashvalue = window.location.hash.substring(1);` followed by a - `fetch(..., { headers: { 'Custom': hashvalue } })` sink focused to `$H`. -- Jerboa was already finding the source. It missed the sink candidate because - the object key/value helper only handled double-quoted key/value pairs and - did not understand the parent `headers` property. -- The helper now handles `headers: { '$KEY': $H }`, so both header entries are - structural sink candidates and the taint engine reports only the tainted - focused value `hashvalue`. - -Expected matched line for this case is now exactly: - -```text -14 -``` - -`taint_labels2` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_labels2$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +taint_param_default +taint_param_default1 +taint_param_default2 +taint_param_default3 +taint_spread_record_small ``` -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_labels2.py`. -- Shape: - labels `TAINTED` from `source(...)`, `CLEANED` from `sanitize(...)`, and a - sink requiring `TAINTED and not CLEANED`. -- Jerboa previously had a false positive at line 9 after an `if` branch cleaned - `b` and an `else` branch assigned tainted `b`. -- The later textual `else` assignment no longer kills the `CLEANED` label from - its mutually exclusive sibling branch. -- The straight-line case in `bar` stays clean, and `baz` line 24 remains the - only expected finding. - -Expected matched line for this case is now exactly: +Relevant upstream files: ```text -24 -``` - -`taint_labels3` is now cleared: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_labels3$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh -``` - -What changed: - -- Target file: - `/Users/user/mine/semgrep/tests/rules/taint_labels3.py`. -- Shape: - labels `P` and `Q` from separate sources, sink requiring `P and Q`. -- The fix preserves the expected line 6 finding where `q(a)` combines with the - prior `p()` label. -- It also avoids the branch-union false positive in `boo()`, where `a = p()` - and `a = q()` occur in mutually exclusive `if`/`else` branches. +/Users/user/mine/semgrep/tests/rules/taint_param_default.yaml +/Users/user/mine/semgrep/tests/rules/taint_param_default.py +/Users/user/mine/semgrep/tests/rules/taint_param_default1.yaml +/Users/user/mine/semgrep/tests/rules/taint_param_default1.py +/Users/user/mine/semgrep/tests/rules/taint_param_default2.yaml +/Users/user/mine/semgrep/tests/rules/taint_param_default2.py +/Users/user/mine/semgrep/tests/rules/taint_param_default3.yaml +/Users/user/mine/semgrep/tests/rules/taint_param_default3.js +/Users/user/mine/semgrep/tests/rules/taint_spread_record_small.yaml +/Users/user/mine/semgrep/tests/rules/taint_spread_record_small.js +``` + +Important isolation finding from this checkpoint: + +- On `taint_param_default.py`, the sink side already matched. The source side + failed only with `symbolic_propagation: true`. +- With `symbolic_propagation` enabled, `pattern-inside: "..."` returned a + single propagated `url` candidate instead of the twelve string literal + candidates in the target file. +- After broad-pattern symbolic fallback was disabled, the exact source formula + found the four non-localhost HTTP literals and the taint rule reported the + same four ranges as Semgrep. +- The last mismatch was `fix-regex` metadata. Semgrep emitted no fix for + focused `fix-regex`; Jerboa initially kept replacements such as + `session.get("https://example.com")`. ## Current Full-Sweep Frontier The current full sorted upstream sweep compared 241 rule/target pairs and has -these 21 mismatches: +these 16 mismatches: ```text -taint_param_default -taint_param_default1 -taint_param_default2 -taint_param_default3 taint_param_source1 taint_param_source2 taint_param_source3 @@ -891,7 +226,6 @@ taint_propagator2 taint_react taint_safe_comparisons taint_spread_record_big -taint_spread_record_small taint_unify_mvars ts_decorated_async_property ts_decorated_static_property @@ -903,302 +237,217 @@ vardef_assign_true1 vardef_assign_true2 ``` -Current full-sweep result: +Use this command to refresh the full list: -```text -upstream-sweep: 218 passed, 21 mismatched, 0 jerboa errors, 2 current errors, 241 compared +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner -errors. This result is from the full sweep command with no `MAX_CASES` limit. -For a first-window guardrail, use `MAX_CASES=220`. +## Recommended Next Target: taint_param_source* -## Immediate Next Case: taint_param_default +The next sorted frontier is the `taint_param_source1/2/3` cluster. Focused command: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_param_default$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_param_source[123]$' LIST_MISMATCHES=1 MAX_DIFFS=3 tests/oracle/upstream-sweep.sh ``` Current result: