Filter unreachable Python taint tails
ober
422c66fcf5ed9af3933094f4b183fed48fb2cac0
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,28 +1,28 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-30 10:04 MDT +Date: 2026-05-30 10:53 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Branch: `main` Base HEAD before this checkpoint: -`9e696c7 Preserve conditional taint metavariables` +`ff250a7 Parse empty inline YAML mapping comments` The user wants the pure Jerboa Semgrep port carried forward until it reaches Semgrep parity. Do not treat this handoff as completion. The latest checkpoint -closes the upstream Python taint parser error `source_param` by normalizing -YAML lines shaped like `patterns: # comment` before the bundled YAML reader -sees them. +closes five upstream Python taint reachability mismatches by filtering taint +matches that occur in unreachable suite tails after `raise`, `return`, `break`, +or `continue`. ## Immediate State -The worktree was clean at `9e696c7` before this checkpoint. The latest +The worktree was clean at `ff250a7` before this checkpoint. The latest implementation change is in -[src/semgrep/rule/parse-rule.ss](/Users/user/mine/jerboa-semgrep/src/semgrep/rule/parse-rule.ss); -[lib/semgrep/rule/parse-rule.sls](/Users/user/mine/jerboa-semgrep/lib/semgrep/rule/parse-rule.sls) +[src/semgrep/scan.ss](/Users/user/mine/jerboa-semgrep/src/semgrep/scan.ss); +[lib/semgrep/scan.sls](/Users/user/mine/jerboa-semgrep/lib/semgrep/scan.sls) and [src/.jerbuild-hashes](/Users/user/mine/jerboa-semgrep/src/.jerbuild-hashes) -were regenerated by `make test`. Focused parser and scan smoke coverage was -added in [tests/smoke.ss](/Users/user/mine/jerboa-semgrep/tests/smoke.ss). +were regenerated by `make test`. Focused Python taint reachability smoke +coverage was added in [tests/smoke.ss](/Users/user/mine/jerboa-semgrep/tests/smoke.ss). Current headline: @@ -31,14 +31,14 @@ Current headline: - Promoted JavaScript pattern oracle: 91 passed / 0 mismatched. - Promoted Python pattern oracle: 116 passed / 0 mismatched. - Local oracle: 42 passed / 0 failed. -- Smoke suite: 309 tests / 309 passed. +- Smoke suite: 310 tests / 310 passed. - Broad same-basename upstream `tests/rules` sweep: 437 passed / 0 mismatched / 0 Jerboa errors, with 3 packaged-Semgrep current errors. - Upstream `tests/tainting_rules/js` sweep: 11 passed / 0 mismatched / 0 Jerboa errors / 0 current errors. -- Upstream `tests/tainting_rules/python` sweep: 6 passed / 6 mismatched / - 0 Jerboa errors / 0 current errors. The prior `source_param` parser error is - gone. +- Upstream `tests/tainting_rules/python` sweep: 11 passed / 1 mismatched / + 0 Jerboa errors / 0 current errors. The only remaining Python tainting-rule + mismatch is `simpl_django_redirect`. Semgrep parity is not reached. The JavaScript tainting-rule subdirectory is clean; the active frontier is now semantic taint-control-flow parity in other @@ -182,6 +182,17 @@ This checkpoint: Latest checkpoint: +- Adds Python taint unreachable-tail filtering after `raise`, `return`, + `break`, and `continue`. +- Closes upstream Python tainting-rule cases `break`, `raise_from`, + `raise_no_args`, `try_raise`, and `try_finally1`. +- Moves the Python tainting-rule sweep from `6 passed / 6 mismatched / + 0 Jerboa errors` to `11 passed / 1 mismatched / 0 Jerboa errors`. +- Adds smoke coverage: + `scan Python taint filters unreachable suite tails`. + +Previous checkpoint: + - Adds a parser normalization pass for YAML lines where an inline comment is the only mapping value after a colon, for example `patterns: # comment`. - Keeps that normalization out of block scalar bodies so Semgrep pattern text @@ -198,14 +209,73 @@ Latest checkpoint modifies: ```text HANDOFF_OPUS_4_8.md -src/semgrep/rule/parse-rule.ss -lib/semgrep/rule/parse-rule.sls +src/semgrep/scan.ss +lib/semgrep/scan.sls src/.jerbuild-hashes tests/smoke.ss ``` ## Latest Code Change Details +### Python Taint Unreachable Suite Tails + +Semgrep does not report taint findings in Python statements that are +syntactically after an abrupt exit in the same suite. The upstream fixtures +exposed this in several small cases: + +```python +def f(): + while (i < 0): + if i == 4: + sink(source) # report + break + sink(source) # do not report +``` + +and: + +```python +def foo(): + try: + raise Foo() + sink(source) # do not report + except: + sink(source) # report + sink(source) # report +``` + +Jerboa already had a Python taint filter for impossible `try`/`except`/`else` +branches, but it did not filter ordinary unreachable tails within the current +suite. The patch adds: + +- `python-line-starts-keyword?` +- `python-taint-abrupt-exit-line?` +- `python-taint-unreachable-at-indent?` +- `python-taint-unreachable-finding?` + +`filter-python-taint-reachable-findings` now removes findings when either the +new unreachable-tail predicate or the existing impossible-exception-branch +predicate is true. + +The unreachable-tail scan is indentation scoped. Once it sees `raise`, +`return`, `break`, or `continue`, later findings at the same or deeper +indentation are filtered. A later significant line with a lower indentation +clears the unreachable state, so sinks after a loop or after an `except` block +remain reachable. + +Smoke coverage added: + +```scheme +(test-case "scan Python taint filters unreachable suite tails" + ...) +``` + +Focused upstream verification: + +```text +upstream-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 6 compared +``` + ### YAML Empty Mapping Inline Comments The upstream Python taint case `source_param` uses this source formula: @@ -644,7 +714,22 @@ make test Result: ```text -309 tests, 309 passed, 0 failed +310 tests, 310 passed, 0 failed +``` + +Focused upstream Python taint reachability cluster: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep \ +UPSTREAM_RULE_DIR=/Users/user/mine/semgrep/tests/tainting_rules/python \ +CASE_REGEX='^(break|raise_from|raise_no_args|try_raise|try_finally1|try_finally)$' \ +LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 6 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 6 compared ``` Focused upstream Python taint `source_param`: @@ -837,17 +922,12 @@ LIST_MISMATCHES=1 MAX_DIFFS=240 tests/oracle/upstream-sweep.sh Result: ```text -upstream-sweep: 6 passed, 6 mismatched, 0 jerboa errors, 0 current errors, 12 compared +upstream-sweep: 11 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 12 compared ``` -Remaining Python mismatches: +Remaining Python mismatch: -- `break`: Jerboa overreports line 9 after a `break`. -- `raise_from`: Jerboa overreports line 5. -- `raise_no_args`: Jerboa overreports line 5. - `simpl_django_redirect`: Jerboa overreports line 43. -- `try_finally1`: Jerboa overreports line 9. -- `try_raise`: Jerboa overreports line 5. ## Cross-Language Tainting-Rule Frontier @@ -860,11 +940,11 @@ Additional subdirectory sweeps found: - `dart`: 0 passed / 2 mismatched. - `go`: 4 passed / 5 mismatched. - `php`: 2 passed / 4 mismatched. -- `python`: 6 passed / 6 mismatched / 0 Jerboa errors. +- `python`: 11 passed / 1 mismatched / 0 Jerboa errors. - `ruby`: 0 passed / 1 mismatched. -The cleanest next taint-control-flow target is probably `python/break`, -because it is a single overreported sink after a `break`. +The remaining Python target is `python/simpl_django_redirect`, where Jerboa +overreports a redirect after an imported `is_safe_url` check. ## Completed Target: `sanitized_by_side_effect` @@ -1334,41 +1414,34 @@ Actual improvement: ## Recommended Next Target -The next target is outside JavaScript taint and outside parser normalization. -`source_param` is now fixed. A practical next target is Python `break`: +The next target is the final current Python tainting-rule mismatch, +`simpl_django_redirect`: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep \ UPSTREAM_RULE_DIR=/Users/user/mine/semgrep/tests/tainting_rules/python \ -CASE_REGEX='^(break)$' LIST_MISMATCHES=1 MAX_DIFFS=180 \ +CASE_REGEX='^(simpl_django_redirect)$' LIST_MISMATCHES=1 MAX_DIFFS=260 \ tests/oracle/upstream-sweep.sh ``` Current diff: ```diff -@@ -1 +1,2 @@ - (finding "test-break" ".../break.py" 6 13 101 6 25 113 "WARNING" "Match Found!" "") -+(finding "test-break" ".../break.py" 9 13 161 9 25 173 "WARNING" "Match Found!" "") -``` - -The likely gap is Python loop/`break` reachability in taint mode: a source or -propagated state from before or inside an abruptly exited loop is reaching a -sink Semgrep treats as unreachable or infeasible. - -After `break`, the next Python taint-control-flow cluster is probably the -raise/try family: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep \ -UPSTREAM_RULE_DIR=/Users/user/mine/semgrep/tests/tainting_rules/python \ -CASE_REGEX='^(raise_from|raise_no_args|try_raise|try_finally1)$' \ -LIST_MISMATCHES=1 MAX_DIFFS=220 \ -tests/oracle/upstream-sweep.sh -``` - -Those cases all overreport early sinks in exception paths, so a shared -Python control-flow refinement may close several of them. +@@ -1,3 +1,4 @@ + (finding "tainting" ".../simpl_django_redirect.py" 19 12 447 19 37 472 "ERROR" "This confirms taint mode works.\n" "") + (finding "tainting" ".../simpl_django_redirect.py" 24 12 559 24 37 584 "ERROR" "This confirms taint mode works.\n" "") ++(finding "tainting" ".../simpl_django_redirect.py" 43 20 1199 43 46 1225 "ERROR" "This confirms taint mode works.\n" "") + (finding "tainting" ".../simpl_django_redirect.py" 9 12 259 9 25 272 "ERROR" "This confirms taint mode works.\n" "") +``` + +The source formula excludes functions containing +`django.utils.http.is_safe_url(...)`, but the target imports it as +`from django.utils.http import is_safe_url` and calls `is_safe_url(...)`. +Direct scans show Jerboa reports the extra finding at line 43 while packaged +Semgrep reports only lines 9, 19, and 24. The likely gap is FQN/import-aware +matching for the `pattern-not-inside` guard, or a narrow Django redirect source +filter equivalent to the existing `python-open-redirect-function-safe?` +search-mode fallback. ## Commit Hygiene --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -30396,6 +30396,56 @@ (sg-string-prefix? "raise " trimmed) (string=? trimmed "return") (sg-string-prefix? "return " trimmed))) + (def (python-line-starts-keyword? trimmed keyword) + (let ([keyword-len (string-length keyword)] + [len (string-length trimmed)]) + (and (>= len keyword-len) + (string=? (substring trimmed 0 keyword-len) keyword) + (or (= len keyword-len) + (not (identifier-token-char? + (string-ref trimmed keyword-len))))))) + (def (python-taint-abrupt-exit-line? trimmed) + (or (python-line-starts-keyword? trimmed "raise") + (python-line-starts-keyword? trimmed "return") + (python-line-starts-keyword? trimmed "break") + (python-line-starts-keyword? trimmed "continue"))) + (def (python-taint-unreachable-at-indent? + unreachable-indent + indent) + (and unreachable-indent (>= indent unreachable-indent))) + (def (python-taint-unreachable-finding? source finding) + (let ([target (finding-start-offset finding)] + [len (string-length source)]) + (let loop ([line-start 0] [unreachable-indent #f]) + (if (>= line-start len) + #f + (let* ([line-end (line-end-after source line-start)] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (if (<= target line-end) + (python-taint-unreachable-at-indent? + unreachable-indent + (line-indent-at-offset source target)) + (let ([trimmed (python-significant-trimmed-line + source + line-start)]) + (if (not trimmed) + (loop next unreachable-indent) + (let* ([indent (line-indent-at-offset + source + line-start)] + [unreachable? (python-taint-unreachable-at-indent? + unreachable-indent + indent)] + [next-unreachable-indent (cond + [unreachable? + unreachable-indent] + [(python-taint-abrupt-exit-line? + trimmed) + indent] + [else #f])]) + (loop next next-unreachable-indent)))))))))) (def (python-try-body-only-pass? source line-start @@ -30465,9 +30515,10 @@ (if (symbolic-python-like-language? language) (sg-filter (lambda (finding) - (not (python-finding-in-impossible-exception-branch? - source - finding))) + (not (or (python-taint-unreachable-finding? source finding) + (python-finding-in-impossible-exception-branch? + source + finding)))) findings) findings)) (def (javascript-taint-line-opens-unreachable? --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "AF0C0A1AAA7621F8") - ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/scan.ss" . "2058A9FA942AC725") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "6FE77014EE9FDCE4") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -30395,6 +30395,59 @@ (string=? trimmed "return") (sg-string-prefix? "return " trimmed))) +(def (python-line-starts-keyword? trimmed keyword) + (let ([keyword-len (string-length keyword)] + [len (string-length trimmed)]) + (and (>= len keyword-len) + (string=? (substring trimmed 0 keyword-len) keyword) + (or (= len keyword-len) + (not (identifier-token-char? + (string-ref trimmed keyword-len))))))) + +(def (python-taint-abrupt-exit-line? trimmed) + (or (python-line-starts-keyword? trimmed "raise") + (python-line-starts-keyword? trimmed "return") + (python-line-starts-keyword? trimmed "break") + (python-line-starts-keyword? trimmed "continue"))) + +(def (python-taint-unreachable-at-indent? unreachable-indent indent) + (and unreachable-indent + (>= indent unreachable-indent))) + +(def (python-taint-unreachable-finding? source finding) + (let ([target (finding-start-offset finding)] + [len (string-length source)]) + (let loop ([line-start 0] [unreachable-indent #f]) + (if (>= line-start len) + #f + (let* ([line-end (line-end-after source line-start)] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (if (<= target line-end) + (python-taint-unreachable-at-indent? + unreachable-indent + (line-indent-at-offset source target)) + (let ([trimmed (python-significant-trimmed-line + source + line-start)]) + (if (not trimmed) + (loop next unreachable-indent) + (let* ([indent (line-indent-at-offset + source + line-start)] + [unreachable? + (python-taint-unreachable-at-indent? + unreachable-indent + indent)] + [next-unreachable-indent + (cond + [unreachable? unreachable-indent] + [(python-taint-abrupt-exit-line? trimmed) + indent] + [else #f])]) + (loop next next-unreachable-indent)))))))))) + (def (python-try-body-only-pass? source line-start try-indent body-indent) (let loop ([current-start line-start]) (or (not current-start) @@ -30449,9 +30502,12 @@ (if (symbolic-python-like-language? language) (sg-filter (lambda (finding) - (not (python-finding-in-impossible-exception-branch? - source - finding))) + (not (or (python-taint-unreachable-finding? + source + finding) + (python-finding-in-impossible-exception-branch? + source + finding)))) findings) findings)) --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -4405,6 +4405,18 @@ (check (finding-start-line (caddr findings)) => 34) (check (finding-start-line (cadddr findings)) => 35))) +(test-case "scan Python taint filters unreachable suite tails" + (let* ([taint-config + "rules:\n - id: demo.taint.unreachable.tail\n mode: taint\n languages: [python]\n message: unreachable tail taint\n severity: WARNING\n pattern-sources:\n - pattern: source\n pattern-sinks:\n - pattern: sink(...)\n"] + [findings + (scan-config-string + taint-config + "python" + "demo.py" + "def loop():\n while cond:\n sink(source)\n break\n sink(source)\n sink(source)\n\ndef raises():\n try:\n raise Foo()\n sink(source)\n except Exception:\n sink(source)\n sink(source)\n")]) + (check (length findings) => 4) + (check (map finding-start-line findings) => '(3 6 13 14)))) + (test-case "scan taint field source reaches opaque base sink" (let* ([taint-config "rules:\n - id: demo.taint.field-base\n mode: taint\n languages: [javascript]\n message: field base taint\n severity: WARNING\n pattern-sources:\n - pattern: source\n pattern-sinks:\n - pattern: sink(...)\n"]