Clear lambda taint label reach
ober
7cfb751eb92d54272f9a545ab87bf9d318455596
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,6 +1,6 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-28 local workspace time +Date: 2026-05-28 23:30 MDT local workspace time Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` @@ -28,9 +28,10 @@ taint/dataflow, path and target semantics, autofix, and output schemas. ## Current Commit Lineage -Recent checkpoints before this handoff: +Recent checkpoints before this handoff commit: ```text +31db293 Clear taint label branch frontier 34227c0 Clear final global taint frontier 268a616 Clear field-sensitive taint frontier 24796db Advance field-sensitive taint parity @@ -71,7 +72,7 @@ make test Result: ```text -188 tests, 188 passed, 0 failed +189 tests, 189 passed, 0 failed ``` Local oracle: @@ -86,6 +87,26 @@ Result: oracle: 42 passed, 0 failed ``` +Focused lambda and nearby taint guardrail for this checkpoint: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_lambda2|taint_lambda4|taint_typestate|taint_side_effect|taint_source)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +MISMATCH taint_lambda4 +@@ -1,2 +0,0 @@ +-(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_lambda4.ts" 3 8 55 3 26 73 "ERROR" "Test" "") +-(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_lambda4.ts" 9 10 156 9 28 174 "ERROR" "Test" "") +upstream-sweep: 2 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 3 compared +``` + +This confirms `taint_lambda2`, `taint_typestate`, `taint_side_effect`, and +`taint_source` are clean in the current tree. `taint_lambda4` is the next +frontier. + Focused upstream guardrail containing the field-sensitive cases fixed so far: ```sh @@ -110,7 +131,7 @@ Result: upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` -Focused label guardrail cleared in this checkpoint: +Focused label guardrail cleared in the previous checkpoint: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_labels2|taint_labels3|taint_typestate|taint_source_requires|taint_source_requires_reopen)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh @@ -131,11 +152,13 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 M Result: ```text -upstream-sweep: 198 passed, 20 mismatched, 0 jerboa errors, 2 current errors, 220 compared +upstream-sweep: 199 passed, 19 mismatched, 0 jerboa errors, 2 current errors, 220 compared ``` -The full 241-case sweep was not rerun after this checkpoint. The last full -run, before the recent `taint_exception` and field-sensitive fixes, was: +The full 241-case sweep was not rerun after this checkpoint. Do not treat the +following stale all-case result as current; it is preserved only as orientation +from a previous handoff before the recent `taint_exception`, field-sensitive, +label, and lambda fixes: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_LIMIT=220 LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh @@ -152,23 +175,28 @@ errors. ## What Changed In This Checkpoint -This checkpoint clears `taint_labels2` and `taint_labels3`, building on -`34227c0` which cleared `taint_final_globals1`. +This checkpoint clears `taint_lambda2`, building on `31db293` which cleared +`taint_labels2` and `taint_labels3`. Implementation changes in `src/semgrep/scan.ss`: -- Assignment kills now recognize mutually exclusive Python `if`/`elif`/`else` - sibling branches by indentation and branch-chain start. -- Assignments in one sibling branch no longer kill labels produced in another - sibling branch merely because they appear later textually. -- Straight-line assignment kills inside the same branch remain unchanged, so - `b = sanitize()` still kills prior `b = a` taint in the same branch. -- This removes label false positives where a `CLEANED` label from one branch - should coexist with taint from an alternate branch at the post-dominator. +- `source-state-reaches-sink-spec-shape?` now lets an ordinary non-token + access-path source reach a wider containing sink when + `source-access-path-compatible-with-sink?` says the source and sink paths are + compatible. +- This specifically covers a source like `req.query` inside the sink argument + `req.query.id` at `db.sink(req.query.id)`. +- The branch is intentionally restricted to non-token source states. A broader + first attempt regressed same-call typestate and side-effect guardrails such + as `lock(l)` and `f.close()`, where token source self-hits must not report. +- The new reachability path still requires physical range containment and + access-path compatibility. It does not make arbitrary nested source + expressions taint arbitrary enclosing sinks. New smoke coverage in `tests/smoke.ss`: ```text +scan JavaScript taint combines lambda db and request labels scan JavaScript taint focused header value under fetch scan taint labels keep sibling if else assignments separate scan Python taint filters impossible exception branches @@ -179,7 +207,7 @@ scan taint dynamic index field source reaches compatible paths scan taint by-side-effect field source reaches containing sink ``` -The new field-clean test mirrors the upstream `taint_field_sensitive2` shape: +The field-clean test mirrors the upstream `taint_field_sensitive2` shape: `x = source` taints `sink(x)`, `sink(x.b)`, `sink(x.b.c)`, and `sink(x.c[j])`, while clean assignments to `x.a` and `x.c[i].d` suppress only those field subtrees. The ancestor-clean test mirrors `taint_field_sensitive4`: @@ -187,6 +215,76 @@ subtrees. The ancestor-clean test mirrors `taint_field_sensitive4`: ## Resolved Recent Cases +`taint_lambda2` is now cleared: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_lambda2$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +``` + +Current result: + +```text +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared +``` + +Rule: + +```yaml +rules: +- id: test + message: Test + severity: ERROR + languages: [javascript, typescript] + mode: taint + pattern-sources: + - label: TAINT + pattern: req.query + - label: DB + pattern: client(...) + pattern-sinks: + - patterns: + - pattern-either: + - pattern: $DB.sink($SINK) + requires: TAINT and DB +``` + +Target file: + +```text +/Users/user/mine/semgrep/tests/rules/taint_lambda2.js +``` + +Important target shape: + +```javascript +let db; +foo(() => { + db = client(); +}); + +bar((req, res) => { + db.sink(req.query.id); +}); +``` + +Why it used to fail: + +- The source `client(...)` gave `db` the `DB` label through assignment + propagation. +- The source `req.query` also matched structurally inside the sink expression. +- The sink required `TAINT and DB`, but the `req.query` source sat inside the + sink range after the sink start, so the reachability gate rejected it before + normal source/sink compatibility could combine the labels. + +Why it passes now: + +- Non-token access-path sources can reach compatible containing sink + expressions. +- `req.query` is compatible with the containing sink argument `req.query.id`, + so `TAINT` combines with the `DB` label on `db` for `$DB.sink($SINK)`. +- Token sources still use the stricter containment branch, preserving the + existing typestate and side-effect self-hit behavior. + `taint_exact_sources` was cleared in `29dd690`: ```sh @@ -464,10 +562,9 @@ What changed: ## Current First-220 Frontier -The current first-220 sweep has these 20 mismatches: +The current first-220 sweep has these 19 mismatches: ```text -taint_lambda2 taint_lambda4 taint_match_on_source taint_nested_record_pattern @@ -489,7 +586,8 @@ taint_react taint_safe_comparisons ``` -The full 241-case sweep adds these 11 later mismatches: +The full 241-case sweep was not rerun after this checkpoint. Based on the +last full run, expect these 11 later mismatches after the first-220 window: ```text taint_spread_record_big @@ -505,17 +603,21 @@ vardef_assign_true1 vardef_assign_true2 ``` -## Immediate Next Case: taint_lambda2 +## Immediate Next Case: taint_lambda4 Focused command: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_lambda2$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_lambda4$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh ``` Current result: ```text +MISMATCH taint_lambda4 +@@ -1,2 +0,0 @@ +-(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_lambda4.ts" 3 8 55 3 26 73 "ERROR" "Test" "") +-(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_lambda4.ts" 9 10 156 9 28 174 "ERROR" "Test" "") upstream-sweep: 0 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` @@ -527,64 +629,110 @@ rules: message: Test severity: ERROR languages: - - javascript - - typescript + - javascript + - typescript mode: taint + options: + interfile: true pattern-sources: - - label: TAINT - pattern: req.query - - label: DB - pattern: client(...) - pattern-sinks: + - pattern-either: + - patterns: + - pattern-either: + - pattern-inside: function ... ($REQ) {...} + - pattern: $REQ.query + - focus-metavariable: $REQ + - label: EDGECASE + patterns: - patterns: - - pattern-either: - - pattern: $DB.sink($SINK) - requires: TAINT and DB + - pattern: $X.$Y. ... + foobar + - focus-metavariable: $X + requires: __SOURCE__ + pattern-sinks: + - patterns: + - pattern: sink($URL) + - focus-metavariable: $URL + requires: __SOURCE__ and EDGECASE ``` Target: ```text -/Users/user/mine/semgrep/tests/rules/taint_lambda2.js +/Users/user/mine/semgrep/tests/rules/taint_lambda4.ts ``` Relevant target: -```text -1 let db; -2 foo(() => { -3 db = client(); -4 }); -6 bar((req, res) => { -7 // ruleid: test -8 db.sink(req.query.id); -9 }); +```typescript +function test1(req: Request) { + sink(req.query + foobar); +}; + +function test2() { + return (req: Request) => { + sink(req.query + foobar); + }; +}; ``` Packaged Semgrep expects: ```text -8 +3, 9 ``` Jerboa currently reports no findings: ```diff -@@ -1 +0,0 @@ --(finding "test" ".../taint_lambda2.js" 8 3 ... "ERROR" "Test" "") +@@ -1,2 +0,0 @@ +-(finding "test" ".../taint_lambda4.ts" 3 8 ... "ERROR" "Test" "") +-(finding "test" ".../taint_lambda4.ts" 9 10 ... "ERROR" "Test" "") ``` +Isolation already done: + +- A sink-only rule with `pattern: sink($URL)` and `focus-metavariable: $URL` + finds both sinks, binding `$URL` to `req.query + foobar`. +- A simple source-only rule with `pattern: $REQ.query` finds both occurrences + and binds `$REQ` to `req`. +- The first real source formula returns no findings: + + ```yaml + patterns: + - pattern-either: + - pattern-inside: function ... ($REQ) {...} + - pattern: $REQ.query + - focus-metavariable: $REQ + ``` + +- The second source formula also returns no findings: + + ```yaml + patterns: + - patterns: + - pattern: $X.$Y. ... + foobar + - focus-metavariable: $X + requires: __SOURCE__ + ``` + Interpretation for the next fix: -- The likely missing flow is a combination of two facts at the same sink: - `db` receives the `DB` label from `db = client()` in one arrow callback, and - `req.query` inside `req.query.id` receives `TAINT` in another callback. -- Semgrep allows both labels to meet at `db.sink(req.query.id)`. -- First confirm whether Jerboa is missing the `DB` propagation from the - assignment inside the first lambda or the `TAINT` compatibility from - `req.query` to the descendant argument `req.query.id`. -- Preserve the guardrails around source-call arguments: this should not be - solved by treating all arguments of all source calls as tainted. +- The first source formula probably needs `pattern-inside: function ... ($REQ) + {...}` to recognize TypeScript typed parameters in both ordinary function + declarations and arrow functions. It must bind or validate `$REQ=req` for + `function test1(req: Request) { ... }` and `return (req: Request) => { ... }`. +- The second source formula needs structural support for the JS/TS expression + pattern `$X.$Y. ... + foobar` against `req.query + foobar`, with + `focus-metavariable: $X` narrowing the source to `req`. +- `requires: __SOURCE__` on the `EDGECASE` source depends on the first source + being available at the focused `$X=req`. Do not treat `EDGECASE` as an + unconditional label. +- Start inspection around `src/semgrep/scan.ss` pattern clause handling, + especially `pattern-inside`, `inside-clause-apply`, JS sequence/object + helpers, and any JS recursive function-inside helpers near the clause + application code. The taint label plumbing appears to be working once source + specs produce the expected focused sources. +- Add a smoke test for the `taint_lambda4` shape before changing behavior, then + rerun the focused lambda guardrail above plus `make test` and `make oracle`. ## Resolved Earlier Frontier --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -9142,6 +9142,12 @@ (source-state-before-sink? source-state source sink) (and (taint-state-contained? source-state) (finding-range-contains? sink source)) + (and (not (taint-state-token? source-state)) + (finding-range-contains? sink source) + (source-access-path-compatible-with-sink? + source + sink + source-text)) (and (taint-state-token? source-state) (finding-range-contains? sink source) (not (finding-range-equal? sink source)) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,7 +3,7 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "1DF2E7B4678CEE76") + ("src/semgrep/scan.ss" . "AAD636B00C0E8C45") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -9583,6 +9583,12 @@ (source-state-before-sink? source-state source sink) (and (taint-state-contained? source-state) (finding-range-contains? sink source)) + (and (not (taint-state-token? source-state)) + (finding-range-contains? sink source) + (source-access-path-compatible-with-sink? + source + sink + source-text)) (and (taint-state-token? source-state) (finding-range-contains? sink source) (not (finding-range-equal? sink source)) --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -1561,6 +1561,18 @@ (check (finding-message (car findings)) => "header taint 'Custom' hashvalue"))) +(test-case "scan JavaScript taint combines lambda db and request labels" + (let* ([taint-config + "rules:\n - id: demo.taint.lambda.labels\n mode: taint\n languages: [javascript]\n message: lambda label taint\n severity: WARNING\n pattern-sources:\n - label: TAINT\n pattern: req.query\n - label: DB\n pattern: client(...)\n pattern-sinks:\n - requires: TAINT and DB\n patterns:\n - pattern-either:\n - pattern: $DB.sink($SINK)\n"] + [findings + (scan-config-string + taint-config + "javascript" + "demo.js" + "let db;\nfoo(() => {\n db = client();\n});\nbar((req, res) => {\n db.sink(req.query.id);\n});\n")]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 6))) + (test-case "scan taint assignment reaches wildcard sink" (let* ([taint-config "rules:\n - id: demo.taint.assignment.wildcard\n mode: taint\n languages: [python]\n message: assigned taint\n severity: WARNING\n pattern-sources:\n - pattern: source()\n pattern-sinks:\n - pattern: sink(...)\n"]