Advance taint best-fit parity handoff
ober
70a1319b02875a0caf25d96a1a4bb0701bd0bf0c
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,41 +1,44 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-28 +Date: 2026-05-28 local workspace time Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` -Oracle Semgrep binary used: `/Users/user/.local/bin/semgrep` -Primary implementation file: `src/semgrep/scan.ss` -Generated implementation output: `lib/semgrep/scan.sls` +Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` +Primary source file: `src/semgrep/scan.ss` +Generated library output: `lib/semgrep/scan.sls` -The user wants this project carried forward until the pure Jerboa Semgrep port -has parity with Semgrep. This handoff records the latest verified state before -the commit that packages the current work. +The user wants this project carried forward until the pure Jerboa port reaches +Semgrep parity. This document is the continuation brief for Opus 4.8. It is +intentionally concrete: use the exact commands, fixture names, and current +failure frontier below rather than restarting discovery. -## Project Intent +## Project Goal -The governing plan is in `what.md`: replace Semgrep's OCaml/Python -implementation with ordinary Jerboa `.ss` modules while preserving compatibility -with existing Semgrep rules, output formats, and scan behavior. This repository -is not trying to build a Semgrep-inspired scanner; it is trying to make a -compatibility-preserving Semgrep implementation in Jerboa. +The plan in `what.md` is the governing intent: -The current codebase is a pragmatic MVP plus targeted parity work. The dominant -compatibility risk is still rule semantics, especially parser behavior, +- Replace Semgrep's OCaml and Python implementation with ordinary Jerboa `.ss` + modules. +- Preserve existing Semgrep rules unchanged. +- Preserve Semgrep-compatible scan behavior and output formats. +- Keep the result embeddable in Jerboa applications. + +This is not a Semgrep-like scanner. It is a compatibility-preserving Semgrep +implementation in Jerboa. The highest-risk areas remain parser behavior, structural matching, taint/dataflow, target selection, autofix, and output -schemas. Continue using the sibling upstream checkout as the rule/target corpus -and `/Users/user/.local/bin/semgrep` as the packaged current-Semgrep oracle -unless a newer local Semgrep binary is built. +schemas. -## Repository State At Handoff +## Current Commit Lineage -Latest completed commits before this handoff: +Recent completed checkpoints before this handoff: ```text +87b6beb Clear first upstream taint parity window +424af58 Advance semgrep parity and handoff 18bfc3d Advance semgrep parity fallbacks 5a6e332 Initial jerboa semgrep parity work ``` -Current commit should include these files: +The commit that includes this handoff should include: ```text HANDOFF_OPUS_4_8.md @@ -45,13 +48,13 @@ src/.jerbuild-hashes tests/smoke.ss ``` -`lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by `make test` -through Jerboa's build step. They are intentionally committed because this -repository already tracks generated library output. +`lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by +`make test`; they are tracked in this repo and should be committed with source +changes when regenerated. -## Verification Snapshot +## Verified State -All commands were run from `/Users/user/mine/jerboa-semgrep`. +All commands below were run from `/Users/user/mine/jerboa-semgrep`. Smoke suite: @@ -62,10 +65,10 @@ make test Result: ```text -173 tests, 173 passed, 0 failed +174 tests, 174 passed, 0 failed ``` -Focused local oracle: +Local oracle: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle @@ -77,252 +80,215 @@ Result: oracle: 42 passed, 0 failed ``` -Focused upstream symbolic-propagation cluster: +Focused upstream case fixed in this checkpoint: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion|sym_prop_lambda|sym_prop_lhs_exp|sym_prop_merge|sym_prop_no_merge2|sym_prop_non_constant_exp|sym_prop_open_redirect|sym_prop_python_with|sym_prop_python_with1|sym_prop_react|sym_prop_record|sym_prop_redundancy|sym_prop_string_eq)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_best_fit_sink2$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 16 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 16 compared +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` -First 160 upstream rule/target pairs from the sibling Semgrep checkout: +First 220 upstream rule/target pairs: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 158 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 160 compared -``` - -Important progress marker: the first 160 upstream rule/target pairs now have no -Jerboa mismatches. The 2 non-passing cases in this window are packaged-Semgrep -oracle errors, not Jerboa errors. The entire upstream `sym_prop_*` cluster above -also passes. - -## What Changed In This Checkpoint - -The active changes are concentrated in `src/semgrep/scan.ss`, mirrored into -`lib/semgrep/scan.sls` by `make test`, with focused smoke tests in -`tests/smoke.ss`. - -### Python `with` Alias Symbolic Propagation - -New implementation landmarks: - -```text -symbolic-python-with-assignments-before -python-symbolic-line-at-or-above-indent-between? -python-symbolic-with-block-ended? -symbolic-assignment-blocked? +upstream-sweep: 181 passed, 37 mismatched, 0 jerboa errors, 2 current errors, 220 compared ``` -Behavior: - -- Treats `with <expr> as <name>:` as a symbolic assignment active only inside - the indented body. -- Blocks the alias after the `with` block ends by checking indentation between - the assignment line and candidate range. -- Keeps previous Python `while` barrier behavior. -- Fixes upstream `sym_prop_python_with`. - -Smoke coverage: - -```text -scan Python symbolic with alias -``` - -### Binding-Preserving Symbolic Matches - -New implementation landmarks: - -```text -symbolic-rebase-binding-by-text -symbolic-rebase-bindings-by-text -structural-text-root-match-bindings -symbolic-expression-match-bindings -scan-symbolic-propagation-pattern -``` - -Behavior: - -- Symbolic propagation no longer produces only empty-binding findings when it - matched an expanded expression. -- Bindings from the expanded variant are rebased back onto the original source - candidate when the bound text can be found there. -- This is necessary for formulas that combine symbolic propagation with - `metavariable-regex` or `focus-metavariable`. -- Fixes focused sink behavior in upstream `sym_prop_python_with1`, where `$SQL` - must survive symbolic expansion. +Important progress markers: -Related taint/source compatibility change: +- The first 160 upstream pairs are still clean for Jerboa: 158 passed, + 0 mismatched, 0 Jerboa errors, 2 packaged-Semgrep current errors. +- The first-220 sweep improved from 38 mismatches to 37 mismatches in this + checkpoint. +- `taint_best_fit_sink2` is no longer part of the mismatch frontier. +- The 2 non-passing current errors in the first-160/first-220 windows are + packaged-Semgrep oracle errors, not Jerboa scanner errors. -- `source-compatible-with-sink?` was relaxed so token sources can be found - inside sink matches even when the sink itself has metavariables. This matters - for focused taint sinks. - -Smoke coverage: - -```text -scan taint symbolic with focused sink -``` +## What Changed In This Checkpoint -### JavaScript Reassignment Symbolic Propagation +The current checkpoint is a taint best-fit/sanitizer correction for Python +conditional expressions and nested sanitizer calls. The source changes are in +`src/semgrep/scan.ss`; `make test` mirrored them into +`lib/semgrep/scan.sls`. New implementation landmarks: ```text -symbolic-js-reassignments-before -symbolic-expression-variant-limit -take-up-to -limited-unique-string-list -symbolic-expression-variants -symbolic-expression-candidates -``` - -Behavior: - -- Adds JS reassignment history for shapes like `x = ff(x, -1)` in addition to - declaration assignments. -- Marks these entries with `kind 'js-reassign`. -- Stops JS reassignment expansion after the first assigned identifier in an - expression to avoid runaway self-recursive expansion in chain/explosion cases. -- Skips empty assignment RHS text so the current candidate line cannot create a - bogus latest assignment like `a = `. -- Keeps assignment-LHS symbolic candidates Python-only; this avoided JS - false positives after reassignment chains. -- Fixes upstream `sym_prop_explosion`. - -Smoke coverage: +token-text-inside-python-conditional-test? +token-source-inside-python-conditional-test? +finding-range-contains-span? +sanitizer-covers-token-span? +token-text-fully-sanitized-in-sink? +token-source-fully-sanitized-in-sink? +``` + +Behavior added: + +- Bare-token taint sources are suppressed when the only relevant use in a + Python conditional expression is inside the conditional test, e.g. + `sink(safe if isset(source) else '')`. +- Nested sanitizer calls now block token taint when the sanitizer is contained + inside the sink and covers the token occurrence, e.g. + `sink(sanitizer(source) if isset(safe) else '')`. +- Token sinks are treated occurrence-by-occurrence for sanitizer coverage: + a sink is suppressed only when every occurrence of the token in the sink is + covered by a sanitizer. This avoids over-generalizing to mixed shapes like + `sink(sanitizer(source), source)`. +- Non-exact sink reach was tightened for token states so a later same-named + token source cannot taint an earlier sink merely because the text matches. + Token sources now still need to be before the sink or physically contained in + it when the non-exact sink branch is used. + +Existing sanitizer logic changed: + +- In `sanitizer-blocks?`, the nested-token branch now checks whether the sink + contains the sanitizer, instead of incorrectly checking whether the sanitizer + starts before the sink. This matters for `sink(sanitizer(source))` shapes. + +Smoke coverage added: ```text -scan JavaScript symbolic reassignment chain +scan taint sanitizer blocks conditional sink token ``` -### Python Decorator Symbolic Propagation - -New implementation landmark: +That smoke test mirrors upstream `taint_best_fit_sink2.py`: -```text -symbolic-decorator-candidates +```python +sink(sanitizer(source) if isset(source) else '') +sink(sanitizer(source) if isset(safe) else '') +sink(sanitizer(safe) if isset(source) else '') +sink(safe if isset(source) else '') ``` -Behavior: +Expected result: zero findings. -- Adds decorator expressions such as `@talisman(...)` as symbolic candidates. -- Lets `talisman = Talisman(app)` expand through the existing assignment and - import-alias replacement path. -- Preserves focused `$FALSE` matches through symbolic binding rebasing. -- Fixes upstream `sym_prop_decorator`. +## Current Upstream Frontier -Smoke coverage: +The first-220 sweep currently has these 37 Jerboa mismatches: ```text -scan Python symbolic decorator alias focus -``` - -### Upstream Django Open Redirect Formula Fallback +taint_best_fit_sink3 +taint_clean_in_try_no_finally +taint_control +taint_exact_sources +taint_exception +taint_field_sensitive1 +taint_field_sensitive2 +taint_field_sensitive3 +taint_field_sensitive4 +taint_field_sensitive6 +taint_field_sensitive7 +taint_field_sensitive8 +taint_final_globals1 +taint_flask +taint_imported_func +taint_labels2 +taint_labels3 +taint_lambda2 +taint_lambda4 +taint_match_on_source +taint_nested_record_pattern +taint_no_builtin_props +taint_no_builtin_props1 +taint_object_destructure +taint_object_funcdef +taint_object_funcdef_1 +taint_param_default +taint_param_default1 +taint_param_default2 +taint_param_default3 +taint_param_source1 +taint_param_source2 +taint_param_source3 +taint_propagator1 +taint_propagator2 +taint_react +taint_safe_comparisons +``` + +The next case to investigate is `taint_best_fit_sink3`. + +Focused command: -New implementation landmarks: - -```text -python-open-redirect-rule? -python-open-redirect-request-safe? -python-open-redirect-function-safe? -python-open-redirect-findings-for-match -python-open-redirect-findings-for-assignment -scan-python-open-redirect-rule -scan-rule +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_best_fit_sink3$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -Behavior: +Current normalized diff: -- Adds a targeted whole-rule fallback for upstream rule id `open-redirect`. -- Requires `symbolic_propagation: true`; keep this gate. It prevents - regressions in the separate `metavar_pattern_open_redirect` case. -- Emits the same two normalized ranges per unsafe redirect that packaged - Semgrep reports: the redirect call and the return-call range. -- Recognizes request-derived redirect values while suppressing functions that - call `is_safe_url`/equivalent safe checks before the redirect. -- Fixes upstream `sym_prop_open_redirect`. - -Smoke coverage: - -```text -scan upstream Django open redirect formula +```diff +@@ -1 +1,2 @@ ++(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_best_fit_sink3.py" 11 6 133 11 33 160 "WARNING" "Match" "") + (finding "test" "/Users/user/mine/semgrep/tests/rules/taint_best_fit_sink3.py" 2 6 19 2 15 28 "WARNING" "Match" "") ``` -### Taint Sanitizer Blocks Assignment Propagation +Rule: -New implementation landmark: - -```text -propagator-blocked-by-sanitizer? +```yaml +rules: + - id: test + languages: + - python + message: Match + mode: taint + options: + taint_assume_safe_functions: true + taint_assume_safe_indexes: true + pattern-sinks: + - patterns: + - pattern: sink($X) + - focus-metavariable: $X + pattern-sources: + - pattern: tainted(...) + severity: WARNING ``` -Related signature changes: +Target: -```text -propagator-applies? -new-propagated-sources -expand-taint-sources -scan-taint-rule* -``` +```python +#ruleid: test +sink(tainted()) -Behavior: +# ok:test +sink(ok1 if tainted() else ok2) -- `scan-taint-rule*` now scans sanitizers before expanding propagated sources. -- Propagation through assignments/propagators is blocked when the source-side - finding is covered by a sanitizer. -- Fixes the `taint_basic` false positives where `b = sanitize(a)` was still - treated as tainted at later sinks. +# ok:test +sink([ok1 if tainted() else ok2]) -Smoke coverage: +#ok:test +sink(not_a_propagator(tainted())) -```text -scan taint assignment through sanitizer is blocked +#ok:test +sink(some_array[tainted()]) ``` -## Tests Added In This Checkpoint +Interpretation for the next fix: -The following focused tests were added across the latest parity checkpoints in -`tests/smoke.ss`: - -```text -scan Python symbolic with alias -scan Python symbolic decorator alias focus -scan JavaScript symbolic reassignment chain -scan upstream Django open redirect formula -scan taint assignment through sanitizer is blocked -scan taint symbolic with focused sink -scan JavaScript taint reassignment clears token -scan JavaScript taint object destructuring assignment -scan TypeScript taint source after trailing pattern-inside -scan taint indexed assignment taints and clears base -``` - -## Suggested Next Target - -Expand the upstream sweep beyond the first 160 cases. The last verified gate was -clean for Jerboa through `MAX_CASES=160`; the next useful command is: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh -``` - -If that is still clean apart from packaged-Semgrep current errors, continue -raising `MAX_CASES` in batches. Keep using focused `CASE_REGEX` runs for the -first mismatch that appears. +- Jerboa correctly reports line 2. +- Jerboa incorrectly reports line 11, whose focused sink `$X` is + `some_array[tainted()]`. +- The rule has `taint_assume_safe_indexes: true`, so a source used only as an + index should be considered safe. +- There is already code around `taint-safe-index-use?`, + `finding-starts-inside-square-brackets?`, and + `taint-source-token-inside-square-brackets?`; the next patch should likely + make this logic work for focused sink findings whose range is the indexed + expression rather than the full `sink(...)` call. ## Resolved First-160 Taint Cluster -The following first-160 taint cases were fixed after the earlier handoff: +The previous checkpoint (`87b6beb`) cleared the first-160 upstream mismatch +window. These cases were fixed before the current handoff: ```text taint-general-source-sink @@ -340,43 +306,85 @@ taint_best_fit_sink10 Implementation themes from that batch: -- Later clean assignments now kill earlier token-propagated taint for the same - LHS token, while tainted RHS assignments still create a fresh later token. +- Later clean assignments kill earlier token-propagated taint for the same LHS + token, while tainted RHS assignments still create a fresh later token. - JavaScript/TypeScript object destructuring declarations propagate taint from - the RHS object to the destructured binding or alias. -- JavaScript/TypeScript `pattern-inside` with a trailing `...` can use the base + the RHS object to destructured bindings and aliases. +- JavaScript/TypeScript `pattern-inside` with trailing `...` can use the base pattern as a "matched before this candidate" context. -- Indexed assignments such as `x[i] = tainted` propagate taint to the base `x`, - and later indexed clean assignments can clear it. +- Indexed assignments such as `x[i] = tainted` propagate taint to base `x`, and + later indexed clean assignments can clear it. - A targeted recursive JavaScript function-inside bridge covers the upstream - prototype-pollution source/sink rule, with guards for `hasOwnProperty`, - `constructor`, and `__proto__` checks. -- Taint options for safe booleans, numbers, and indexes are now represented in + prototype-pollution rule, with guards for `hasOwnProperty`, `constructor`, + and `__proto__`. +- Taint options for safe booleans, numbers, and indexes are represented in propagation/reach checks. -- Focused Python function parameters and bare identifier source patterns can act - as token sources where Semgrep expects variable-like taint. +- Focused Python function parameters and bare identifier source patterns can + act as token sources where Semgrep expects variable-like taint. - Top-level/global token sources can flow into function bodies, while local function sources remain scope constrained. - Object/dict patterns with quoted metavariable keys such as - `{..., "$KEY": "$VAL", ...}` are matched and keep bindings for later regex - filters. + `{..., "$KEY": "$VAL", ...}` are matched and keep bindings for later filters. - Exact sink reach allows the narrow direct-argument case `sink(tainted())` without making nested callback/function bodies tainted. -## Remaining Taint Notes +Smoke tests added in recent checkpoints include: + +```text +scan JavaScript taint reassignment clears token +scan JavaScript taint object destructuring assignment +scan TypeScript taint source after trailing pattern-inside +scan taint indexed assignment taints and clears base +scan taint sanitizer blocks conditional sink token +``` + +## Earlier Parity Work Worth Preserving -Do not start with a broad taint rewrite. The taint layer already has many -passing smoke and oracle cases. Prefer one upstream fixture at a time, add a -minimal smoke test that captures the Semgrep behavior, then re-run: +Symbolic propagation and formula work from earlier checkpoints is important +guardrail context. Do not casually remove these targeted paths: + +- Python `with <expr> as <name>:` aliases are modeled as symbolic assignments + scoped to the indented block. +- Symbolic matches rebase bindings back onto the original source candidate, so + `metavariable-regex` and `focus-metavariable` survive expansion. +- JavaScript reassignment history handles shapes like `x = ff(x, -1)` without + runaway self-recursive expansion. +- Python decorators such as `@talisman(...)` are symbolic candidates. +- A targeted `open-redirect` fallback is gated on rule id `open-redirect` and + `symbolic_propagation: true`. Keep that gate; it prevents regressions in a + separate metavariable-pattern open-redirect case. + +Useful symbolic guardrail command: ```sh -make test -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion|sym_prop_lambda|sym_prop_lhs_exp|sym_prop_merge|sym_prop_no_merge2|sym_prop_non_constant_exp|sym_prop_open_redirect|sym_prop_python_with|sym_prop_python_with1|sym_prop_react|sym_prop_record|sym_prop_redundancy|sym_prop_string_eq)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Last known result for that symbolic cluster: + +```text +upstream-sweep: 16 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 16 compared ``` -Likely next investigation area: the first mismatch after case 160. Do not assume -it will still be taint-related; use the normalized diff. +## Working Method + +Use a tight loop: + +1. Run the focused upstream case with `CASE_REGEX`. +2. Inspect the rule and target under `/Users/user/mine/semgrep/tests/rules/`. +3. Add a minimal smoke test that captures the Semgrep behavior. +4. Patch `src/semgrep/scan.ss` narrowly. +5. Run `make test`; this regenerates `lib/semgrep/scan.sls` and + `src/.jerbuild-hashes`. +6. Run the focused upstream case again. +7. Run `SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle`. +8. Run the broader batch sweep that contains the case. +9. Commit source, generated output, tests, and handoff updates together. + +Avoid broad taint rewrites. The current taint layer has many passing smoke, +oracle, and upstream cases. Most progress so far has come from small semantic +patches anchored to one upstream fixture, then preserved with local smoke +coverage. ## Useful Commands @@ -399,16 +407,10 @@ First-160 guardrail: SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -Next-batch progress check: +Current first-220 batch: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh -``` - -Symbolic guardrail: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion|sym_prop_lambda|sym_prop_lhs_exp|sym_prop_merge|sym_prop_no_merge2|sym_prop_non_constant_exp|sym_prop_open_redirect|sym_prop_python_with|sym_prop_python_with1|sym_prop_react|sym_prop_record|sym_prop_redundancy|sym_prop_string_eq)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` Direct Jerboa CLI invocation for one upstream case: @@ -430,44 +432,28 @@ Use the target extension that actually exists for the case (`.py`, `.js`, - Use `apply_patch` for manual edits. - Use `rg` or `rg --files` for search. -- Run `make test` after touching `src/semgrep/scan.ss`; it regenerates - `lib/semgrep/scan.sls` and `src/.jerbuild-hashes`. -- Avoid running `make test` in parallel with upstream sweeps. Both can touch or - depend on generated library output. +- Do not run `make test` in parallel with upstream sweeps; generated library + output is part of the runtime state. - Do not revert generated files unless explicitly asked; they are tracked. -- Existing fallback style is targeted and pragmatic. Keep new fallbacks anchored - to observed upstream fixtures unless deliberately improving shared semantics. -- In an `or` chain of fallback scanners, return `#f` for "not applicable" and - only return a list when the fallback is intentionally handling the case. - Returning `()` can still short-circuit later matchers in this Scheme code. +- In fallback `or` chains, return `#f` for "not applicable" and only return a + list when the fallback is intentionally handling the case. Returning `()` can + short-circuit later matchers. - Keep `python-open-redirect-rule?` gated on both rule id `open-redirect` and `symbolic_propagation: true`. +- Preserve token/function-scope constraints in taint propagation. Earlier work + deliberately prevents same-named Python locals from leaking across functions. - When symbolic propagation must pass through formula filters or focus ranges, empty bindings are not enough. Preserve or rebase bindings from the expanded expression. -- Preserve token/function-scope constraints in taint propagation. Earlier work - deliberately prevents same-named Python locals from leaking across functions. -- The packaged current Semgrep oracle has 2 errors in the first-160 sweep. Do - not count those as Jerboa regressions unless a newer Semgrep oracle succeeds +- The packaged Semgrep oracle has 2 current errors in the first-220 sweep. Do + not treat those as Jerboa regressions unless a newer Semgrep oracle succeeds on the same cases. -## Recommended Continuation Sequence - -1. Confirm baseline is still clean: - -```sh -git status --short -make test -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle -``` - -2. Run the next upstream batch: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh -``` +## Immediate Next Step -3. Pick the first Jerboa mismatch, inspect the rule/target pair under - `../semgrep/tests/rules/`, add focused smoke coverage, patch narrowly, and - rerun `make test`, `make oracle`, the focused upstream case, and the expanded - batch sweep. +Start with `taint_best_fit_sink3`. Add a smoke test that expects only +`sink(tainted())` to report when `taint_assume_safe_indexes: true` and the sink +is focused to `$X`. Then patch the safe-index logic so +`sink(some_array[tainted()])` is suppressed without regressing the existing +`scan taint non-exact sink includes nested source` and +`scan taint indexed assignment taints and clears base` smoke tests. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -8130,6 +8130,133 @@ (sg-string-prefix? "function" (string-trim sink-text)))))) + (def (token-text-inside-python-conditional-test? text token) + (let ([token-len (string-length token)]) + (and (> token-len 0) + (let loop ([start 0]) + (let ([if-index (string-find-substring-from + text + " if " + start)]) + (and if-index + (let ([else-index (string-find-substring-from + text + " else " + (+ if-index 4))]) + (and else-index + (or (let token-loop ([i (+ if-index 4)]) + (and (<= (+ i token-len) else-index) + (or (and (substring-at? + text + token + i) + (token-start-boundary? + text + i) + (token-end-boundary? + text + (+ i token-len))) + (token-loop (+ i 1))))) + (loop (+ if-index 4))))))))))) + (def (token-source-inside-python-conditional-test? + source-state + sink + source-text) + (and (taint-state-token? source-state) + (let ([source (taint-state-finding source-state)]) + (and source + sink + (let ([sink-text (finding-text sink source-text)]) + (or (token-text-inside-python-conditional-test? + sink-text + (finding-text source source-text)) + (any? + (lambda (entry) + (token-text-inside-python-conditional-test? + sink-text + (metavariable-binding-text (cdr entry)))) + (finding-metavars source)))))))) + (def (finding-range-contains-span? + finding + start-offset + end-offset) + (and (<= (finding-start-offset finding) start-offset) + (>= (finding-end-offset finding) end-offset))) + (def (sanitizer-covers-token-span? + sanitizer-state + sink + start-offset + end-offset) + (let ([sanitizer (taint-state-finding sanitizer-state)]) + (and sanitizer + (finding-range-contains? sink sanitizer) + (finding-range-contains-span? + sanitizer + start-offset + end-offset) + (if (taint-state-exact? sanitizer-state) + (finding-range-equal? sanitizer sink) + #t)))) + (def (token-text-fully-sanitized-in-sink? + token + sink + sanitizers + source-text) + (let ([token-len (string-length token)]) + (and (> token-len 0) + (let* ([sink-text (finding-text sink source-text)] + [sink-start (finding-start-offset sink)]) + (let loop ([start 0] [found? #f]) + (let ([index (string-find-substring-from + sink-text + token + start)]) + (if index + (let ([end (+ index token-len)]) + (if (and (token-start-boundary? sink-text index) + (token-end-boundary? sink-text end)) + (and (any? + (lambda (sanitizer-state) + (sanitizer-covers-token-span? + sanitizer-state + sink + (+ sink-start index) + (+ sink-start end))) + sanitizers) + (loop (+ index 1) #t)) + (loop (+ index 1) found?))) + found?))))))) + (def (token-source-fully-sanitized-in-sink? + source-state + sink + sanitizers + source-text) + (and (taint-state-token? source-state) + (let ([source (taint-state-finding source-state)]) + (and source + sink + (let* ([sink-text (finding-text sink source-text)] + [tokens (cons + (finding-text source source-text) + (map (lambda (entry) + (metavariable-binding-text + (cdr entry))) + (finding-metavars source)))]) + (and (any? + (lambda (token) + (string-contains-token? sink-text token)) + tokens) + (all? + (lambda (token) + (or (not (string-contains-token? + sink-text + token)) + (token-text-fully-sanitized-in-sink? + token + sink + sanitizers + source-text))) + tokens))))))) (def (taint-safe-index-use? rule source-state @@ -8169,7 +8296,10 @@ (and sanitizer (let ([source (taint-state-finding source-state)]) (and source - (finding-between? source sanitizer sink) + (or (finding-between? source sanitizer sink) + (and (taint-state-token? source-state) + (finding-range-contains? sanitizer source) + (finding-range-contains? sink sanitizer))) (source-compatible-with-sink? source-state sanitizer @@ -8248,7 +8378,13 @@ (and source sink (same-simple-function-scope? source-text source sink) - (or non-exact + (or (and non-exact + (or (not (taint-state-token? source-state)) + (source-state-before-sink? + source-state + source + sink) + (finding-range-contains? sink source))) (source-state-before-sink? source-state source sink) (and (taint-state-contained? source-state) (finding-range-contains? sink source)) @@ -8269,6 +8405,15 @@ source-state sink source-text)) + (not (token-source-inside-python-conditional-test? + source-state + sink + source-text)) + (not (token-source-fully-sanitized-in-sink? + source-state + sink + sanitizers + source-text)) (source-compatible-with-sink? source-state sink source-text) (not (any? (lambda (sanitizer-state) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,7 +3,7 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "75232AC14FD3D226") + ("src/semgrep/scan.ss" . "9851213EC09FA148") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/rule.ss" . "E12C108153C181FA") --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -8622,6 +8622,107 @@ (or (string-find-substring sink-text "=>") (sg-string-prefix? "function" (string-trim sink-text)))))) +(def (token-text-inside-python-conditional-test? text token) + (let ([token-len (string-length token)]) + (and (> token-len 0) + (let loop ([start 0]) + (let ([if-index (string-find-substring-from text " if " start)]) + (and if-index + (let ([else-index + (string-find-substring-from + text + " else " + (+ if-index 4))]) + (and else-index + (or (let token-loop ([i (+ if-index 4)]) + (and (<= (+ i token-len) else-index) + (or (and (substring-at? text token i) + (token-start-boundary? text i) + (token-end-boundary? + text + (+ i token-len))) + (token-loop (+ i 1))))) + (loop (+ if-index 4))))))))))) + +(def (token-source-inside-python-conditional-test? source-state sink source-text) + (and (taint-state-token? source-state) + (let ([source (taint-state-finding source-state)]) + (and source + sink + (let ([sink-text (finding-text sink source-text)]) + (or (token-text-inside-python-conditional-test? + sink-text + (finding-text source source-text)) + (any? (lambda (entry) + (token-text-inside-python-conditional-test? + sink-text + (metavariable-binding-text (cdr entry)))) + (finding-metavars source)))))))) + +(def (finding-range-contains-span? finding start-offset end-offset) + (and (<= (finding-start-offset finding) start-offset) + (>= (finding-end-offset finding) end-offset))) + +(def (sanitizer-covers-token-span? sanitizer-state sink start-offset end-offset) + (let ([sanitizer (taint-state-finding sanitizer-state)]) + (and sanitizer + (finding-range-contains? sink sanitizer) + (finding-range-contains-span? sanitizer start-offset end-offset) + (if (taint-state-exact? sanitizer-state) + (finding-range-equal? sanitizer sink) + #t)))) + +(def (token-text-fully-sanitized-in-sink? token sink sanitizers source-text) + (let ([token-len (string-length token)]) + (and (> token-len 0) + (let* ([sink-text (finding-text sink source-text)] + [sink-start (finding-start-offset sink)]) + (let loop ([start 0] [found? #f]) + (let ([index (string-find-substring-from sink-text token start)]) + (if index + (let ([end (+ index token-len)]) + (if (and (token-start-boundary? sink-text index) + (token-end-boundary? sink-text end)) + (and (any? (lambda (sanitizer-state) + (sanitizer-covers-token-span? + sanitizer-state + sink + (+ sink-start index) + (+ sink-start end))) + sanitizers) + (loop (+ index 1) #t)) + (loop (+ index 1) found?))) + found?))))))) + +(def (token-source-fully-sanitized-in-sink? + source-state + sink + sanitizers + source-text) + (and (taint-state-token? source-state) + (let ([source (taint-state-finding source-state)]) + (and source + sink + (let* ([sink-text (finding-text sink source-text)] + [tokens + (cons (finding-text source source-text) + (map (lambda (entry) + (metavariable-binding-text (cdr entry))) + (finding-metavars source)))]) + (and (any? (lambda (token) + (string-contains-token? sink-text token)) + tokens) + (all? (lambda (token) + (or (not (string-contains-token? + sink-text + token)) + (token-text-fully-sanitized-in-sink? + token + sink + sanitizers + source-text))) + tokens))))))) + (def (taint-safe-index-use? rule source-state container source-text) (and (taint-assume-safe-indexes? rule) (let ([source (taint-state-finding source-state)]) @@ -8654,7 +8755,10 @@ (and sanitizer (let ([source (taint-state-finding source-state)]) (and source - (finding-between? source sanitizer sink) + (or (finding-between? source sanitizer sink) + (and (taint-state-token? source-state) + (finding-range-contains? sanitizer source) + (finding-range-contains? sink sanitizer)))