Clear taint exact source frontier
ober
29dd690348464594361343d27c8d8d89c4f764f4
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -8,13 +8,13 @@ Primary source file: `src/semgrep/scan.ss` Generated library output: `lib/semgrep/scan.sls` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. This document is the continuation brief for Opus 4.8. It is -intentionally concrete: use the exact commands, fixture names, and current -failure frontier below rather than restarting discovery. +Semgrep parity. Continue from this state. Do not restart discovery from +scratch; the current mismatch frontier and the exact commands below are the +working map. ## Project Goal -The plan in `what.md` is the governing intent: +The intent from `what.md` remains the governing target: - Replace Semgrep's OCaml and Python implementation with ordinary Jerboa `.ss` modules. @@ -22,23 +22,25 @@ The plan in `what.md` is the governing intent: - Preserve Semgrep-compatible scan behavior and output formats. - Keep the result embeddable in Jerboa applications. -This is not a Semgrep-like scanner. It is a compatibility-preserving Semgrep -implementation in Jerboa. The highest-risk areas remain parser behavior, -structural matching, taint/dataflow, target selection, autofix, and output -schemas. +This is a compatibility-preserving Semgrep implementation in Jerboa, not a +similar scanner. High-risk parity areas remain structural matching, +taint/dataflow, path and target semantics, autofix, and output schemas. ## Current Commit Lineage -Recent completed checkpoints before this handoff: +Recent checkpoints before this handoff: ```text +7beb039 Advance taint control parity +e429d34 Advance taint safe function parity +70a1319 Advance taint best-fit parity handoff 87b6beb Clear first upstream taint parity window 424af58 Advance semgrep parity and handoff 18bfc3d Advance semgrep parity fallbacks 5a6e332 Initial jerboa semgrep parity work ``` -The commit that includes this handoff should include: +The commit containing this document should include: ```text HANDOFF_OPUS_4_8.md @@ -49,8 +51,7 @@ tests/smoke.ss ``` `lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by -`make test`; they are tracked in this repo and should be committed with source -changes when regenerated. +`make test`; they are tracked and should be committed with source changes. ## Verified State @@ -65,7 +66,7 @@ make test Result: ```text -178 tests, 178 passed, 0 failed +180 tests, 180 passed, 0 failed ``` Local oracle: @@ -80,101 +81,135 @@ Result: oracle: 42 passed, 0 failed ``` -Focused upstream cases fixed in this checkpoint: +Focused upstream case fixed in this checkpoint: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_control|taint_flask|taint_imported_func)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_exact_sources$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` -Focused safe-option/propagation guardrail: +Intended first-220 upstream sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_best_fit_sink|taint_best_fit_sink2|taint_best_fit_sink3|taint_clean_in_try_no_finally|taint_assume_safe_indexes|taint_assume_safe_numbers|taint_assume_safe_booleans)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 7 compared +upstream-sweep: 187 passed, 31 mismatched, 0 jerboa errors, 2 current errors, 220 compared ``` -First 220 upstream rule/target pairs: +Full current upstream sweep, accidentally run with `CASE_LIMIT=220` which this +script ignores: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_LIMIT=220 LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 186 passed, 32 mismatched, 0 jerboa errors, 2 current errors, 220 compared +upstream-sweep: 197 passed, 42 mismatched, 0 jerboa errors, 2 current errors, 241 compared ``` -Important progress markers: - -- The first 160 upstream pairs are still clean for Jerboa: 158 passed, - 0 mismatched, 0 Jerboa errors, 2 packaged-Semgrep current errors. -- The first-220 sweep improved from 35 mismatches to 32 mismatches in this - checkpoint. -- `taint_control`, `taint_flask`, and `taint_imported_func` are no longer part - of the mismatch frontier. -- The 2 non-passing current errors in the first-160/first-220 windows are - packaged-Semgrep oracle errors, not Jerboa scanner errors. +The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner +errors. ## What Changed In This Checkpoint -The current checkpoint adds control-taint reachability and Python import-alias -pattern matching for taint specs. The source changes are in -`src/semgrep/scan.ss`; `make test` mirrored them into -`lib/semgrep/scan.sls`. +This checkpoint clears `taint_exact_sources`, which was the next mismatch after +`7beb039`. + +Implementation changes in `src/semgrep/scan.ss`: + +- Python implicit propagation now includes loop variables with: + + ```scheme + "for $L in $R:\n ..." + ``` + + This models `for res in results:` as propagation from `results` to `res`. + +- Assignment-kill matching is broader and container-aware. A clean assignment + can now kill an earlier source when the assignment LHS contains the source + token, not only when the assignment target binding is exactly equal to the + source text. This is what suppresses stale function-parameter taint after + clean writes like `params["sql"] = "safe"`. + +- Propagator applicability now checks whether an earlier clean assignment has + killed the source before the propagator RHS. This prevents stale container + taint from re-propagating after a clean write. + +- Duplicate implicit assignment matches for the same target statement are + excluded by finding range, not just by object identity. This matters because + `params["sql"] = params[...]` matches both `$L = $R` and `$L[$I] = $R`. + Without the range-based exclusion, one implicit propagator kills the other + while it is trying to inspect the same RHS. + +- `expand-taint-sources` now receives the implicit assignment propagators as + assignment-kill facts, so propagation and sink reachability use the same + clean-assignment information. + +New smoke coverage in `tests/smoke.ss`: + +```text +scan taint indexed assignment can re-taint same base +scan Python taint propagates through loop variables +``` + +The first test locks the subtle `params["sql"] = "x" % params["test"]` behavior: +a tainted write should re-taint `params`, while a clean indexed write should +not. The second test locks basic `for item in tainted_iterable` propagation. + +## Resolved Case: taint_exact_sources + +Focused command: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_exact_sources$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +``` -New implementation landmarks: +Current result: ```text -taint-state-control? -python-import-local-pattern-candidates -scan-python-import-local-pattern -scan-python-import-local-pattern-with-bindings -``` - -Behavior added: - -- `control: true` taint sources now carry a control bit in the taint state and - can reach later sinks in the same simple function scope by order/control - reachability rather than value containment. -- The control bit is preserved through label changes and participates in - taint-state identity, so label/requires logic does not merge control and - ordinary value taint states. -- Python patterns written with imported fully qualified names can match local - imported calls. For example, a rule pattern - `framework.db_access.mysql_update($SINK)` now matches target code after - `from framework import db_access` as `db_access.mysql_update(params)`. -- The Python import-local fallback returns `#f` when it has no findings, so it - does not short-circuit later structural/symbolic matchers in fallback `or` - chains. -- Clean reassignment killing now applies to any source whose source text equals - the assigned LHS, not only token-marked states. This helps focused - function-parameter sources and variable-like source patterns. - -Smoke coverage added: +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared +``` + +Why it used to fail: + +- Jerboa had false positives at lines 23 and 41 in + `/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py`. +- A focused function parameter source tainted `params`. +- Clean indexed assignments such as `params["sql"] = "select xyz from table"` + and `params["name"] = "test"` did not kill the broad parameter source. +- A first attempt to kill those assignments over-blocked expected flows at + lines 7 and 33. + +Why it passes now: + +- Clean indexed assignments kill the earlier broad `params` taint. +- The tainting assignment in `fn1` re-taints `params` from its own RHS because + same-range duplicate implicit propagators no longer kill each other. +- The `for res in results:` loop in `fn3` propagates the tainted `results` + value into `res`, which then re-taints `params` before line 33. + +Expected matched lines for this case are now exactly: ```text -scan taint control source reaches later sinks -scan Python taint imported fully qualified sink +7, 28, 33, 49, 60, 69 ``` -## Current Upstream Frontier +## Current First-220 Frontier -The first-220 sweep currently has these 32 Jerboa mismatches: +The current first-220 sweep has these 31 mismatches: ```text -taint_exact_sources taint_exception taint_field_sensitive1 taint_field_sensitive2 @@ -208,98 +243,160 @@ taint_react taint_safe_comparisons ``` -The next case to investigate is `taint_exact_sources`. +The full 241-case sweep adds these 11 later mismatches: + +```text +taint_spread_record_big +taint_spread_record_small +taint_unify_mvars +ts_decorated_async_property +ts_decorated_static_property +unicode_string +vardef_assign_false +vardef_assign_false1 +vardef_assign_true +vardef_assign_true1 +vardef_assign_true2 +``` + +## Immediate Next Case: taint_exception Focused command: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_exact_sources$' LIST_MISMATCHES=1 MAX_DIFFS=240 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_exception$' LIST_MISMATCHES=1 MAX_DIFFS=200 tests/oracle/upstream-sweep.sh ``` -Current normalized diff: +Current result: -```diff -@@ -1,5 +1,7 @@ -+(finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 23 36 776 23 42 782 "WARNING" "Semgrep found a match" "") - (finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 28 28 990 28 34 996 "WARNING" "Semgrep found a match" "") - (finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 33 32 1161 33 38 1167 "WARNING" "Semgrep found a match" "") -+(finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 41 28 1483 41 34 1489 "WARNING" "Semgrep found a match" "") - (finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 49 28 1654 49 34 1660 "WARNING" "Semgrep found a match" "") - (finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 60 28 2029 60 34 2035 "WARNING" "Semgrep found a match" "") - (finding "sql-injection" "/Users/user/mine/semgrep/tests/rules/taint_exact_sources.py" 69 28 2232 69 34 2238 "WARNING" "Semgrep found a match" "") +```text +upstream-sweep: 0 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` -Interpretation for the next fix: +Rule: + +```yaml +rules: +- id: python-exception + mode: taint + pattern-sources: + - pattern: input + pattern-sanitizers: + - pattern: sanitize(...) + pattern-sinks: + - pattern: sink(...) + message: Match found + languages: + - python + severity: ERROR +``` + +Target: -- The Python import-local fallback moved this case from all-missing findings to - mostly matching the expected sink family. -- Remaining false positives are line 23 (`results = db_access.mysql_dict(params)`) - after a clean assignment to `params["sql"]`, and line 41 after a clean - assignment to `params["name"]`. -- Continue in the assignment-kill / field-sensitivity area. The broad - source-text-equals-LHS kill is not enough because later self-use and indexed - assignment semantics still need to distinguish clean field writes from - tainting writes. +```text +/Users/user/mine/semgrep/tests/rules/taint_exception.py +``` -## Resolved First-160 Taint Cluster +Packaged Semgrep expects these sink lines: + +```text +108, 124, 139, 141, 174, 192, 194, 196 +``` -The previous checkpoint (`87b6beb`) cleared the first-160 upstream mismatch -window. These cases were fixed before the current handoff: +Jerboa currently reports all expected lines plus false positives at: ```text -taint-general-source-sink -taint_array -taint_assign_cleans_shape -taint_assign_record -taint_assign_record1 -taint_assume_safe_booleans -taint_assume_safe_indexes -taint_assume_safe_numbers -taint_async -taint_best_fit_sink -taint_best_fit_sink10 -``` - -Implementation themes from that batch: - -- Later clean assignments kill earlier token-propagated taint for the same LHS - token, while tainted RHS assignments still create a fresh later token. +29, 35, 74, 80, 106, 122, 179 +``` + +Normalized diff summary: + +```diff +@@ -1,8 +1,15 @@ ++line 106 sink(clean) + line 108 sink(dirty) ++line 122 sink(clean) + line 124 sink(dirty) + line 139 sink(dirty1) + line 141 sink(dirty2) + line 174 sink(clean1) ++line 179 sink(clean2) + line 192 sink(dirty1) + line 194 sink(dirty2) + line 196 sink(dirty3) ++line 29 sink(clean) ++line 35 sink(clean) ++line 74 sink(clean) ++line 80 sink(clean) +``` + +Interpretation for the next fix: + +- This is path-sensitive Python exception reachability, not import matching or + basic assignment propagation. +- The target intentionally includes unreachable sinks after unconditional + `raise` and `return`, and impossible `except` or `else` paths when the `try` + body is trivially known to raise or not raise. +- Current Jerboa taint is mostly path-insensitive across Python `try`, + `except`, `else`, and `finally` blocks. It propagates assignments that + packaged Semgrep considers unreachable for this fixture. +- Be careful around `finally`: packaged Semgrep expects finally assignments to + remain reachable in the nested case around lines 180 to 196. + +Likely next implementation area: + +- Add a narrow Python reachability guard for taint source/assignment matches + that are after unconditional `raise` or `return` in the same simple block. +- Add enough try/except/else/finally reasoning for trivial cases: + `try: raise ...` should make `except` reachable and `else` unreachable; + `try: pass` should make `else` reachable and `except` unreachable; + calls like `any_function_call_may_raise()` should keep both paths possible. +- Preserve existing passing guardrails, especially + `taint_clean_in_try_no_finally` and the first-160 taint cluster. + +## Resolved Earlier Frontier + +The previous checkpoints cleared these important upstream clusters: + +- first 160 upstream pairs: 158 passed, 0 mismatched, 0 Jerboa errors, + 2 packaged-Semgrep current errors at the last recorded check. +- `taint_control`, `taint_flask`, and `taint_imported_func` are clean. +- safe-option and best-fit guardrails are clean: + + ```sh + SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_best_fit_sink|taint_best_fit_sink2|taint_best_fit_sink3|taint_clean_in_try_no_finally|taint_assume_safe_indexes|taint_assume_safe_numbers|taint_assume_safe_booleans)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh + ``` + + Last known result: + + ```text + upstream-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 7 compared + ``` + +Implementation themes already in place: + +- `control: true` taint sources carry a control bit and can reach later sinks in + the same simple function scope. +- Python fully qualified imported patterns can match local imported calls, for + example `framework.db_access.mysql_update($SINK)` against + `from framework import db_access` followed by + `db_access.mysql_update(params)`. - JavaScript/TypeScript object destructuring declarations propagate taint from - the RHS object to destructured bindings and aliases. -- JavaScript/TypeScript `pattern-inside` with trailing `...` can use the base - pattern as a "matched before this candidate" context. + RHS objects to destructured bindings and aliases. - Indexed assignments such as `x[i] = tainted` propagate taint to base `x`, and - later indexed clean assignments can clear it. -- A targeted recursive JavaScript function-inside bridge covers the upstream - prototype-pollution rule, with guards for `hasOwnProperty`, `constructor`, - and `__proto__`. -- Taint options for safe booleans, numbers, and indexes are represented in - propagation/reach checks. + later clean indexed assignments can clear it. +- Taint options for safe booleans, numbers, indexes, and functions have + targeted guards. - Focused Python function parameters and bare identifier source patterns can act as token sources where Semgrep expects variable-like taint. - Top-level/global token sources can flow into function bodies, while local function sources remain scope constrained. -- Object/dict patterns with quoted metavariable keys such as - `{..., "$KEY": "$VAL", ...}` are matched and keep bindings for later filters. - Exact sink reach allows the narrow direct-argument case `sink(tainted())` without making nested callback/function bodies tainted. -Smoke tests added in recent checkpoints include: - -```text -scan JavaScript taint reassignment clears token -scan JavaScript taint object destructuring assignment -scan TypeScript taint source after trailing pattern-inside -scan taint indexed assignment taints and clears base -scan taint sanitizer blocks conditional sink token -scan taint assume safe functions blocks wrapper -scan taint sanitizer blocks later assignment propagation -``` +## Symbolic Work Worth Preserving -## Earlier Parity Work Worth Preserving - -Symbolic propagation and formula work from earlier checkpoints is important -guardrail context. Do not casually remove these targeted paths: +Do not casually remove these targeted symbolic paths: - Python `with <expr> as <name>:` aliases are modeled as symbolic assignments scoped to the indented block. @@ -308,9 +405,8 @@ guardrail context. Do not casually remove these targeted paths: - JavaScript reassignment history handles shapes like `x = ff(x, -1)` without runaway self-recursive expansion. - Python decorators such as `@talisman(...)` are symbolic candidates. -- A targeted `open-redirect` fallback is gated on rule id `open-redirect` and - `symbolic_propagation: true`. Keep that gate; it prevents regressions in a - separate metavariable-pattern open-redirect case. +- The targeted `open-redirect` fallback is gated on rule id `open-redirect` + and `symbolic_propagation: true`; keep that gate. Useful symbolic guardrail command: @@ -318,7 +414,7 @@ Useful symbolic guardrail command: SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion|sym_prop_lambda|sym_prop_lhs_exp|sym_prop_merge|sym_prop_no_merge2|sym_prop_non_constant_exp|sym_prop_open_redirect|sym_prop_python_with|sym_prop_python_with1|sym_prop_react|sym_prop_record|sym_prop_redundancy|sym_prop_string_eq)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -Last known result for that symbolic cluster: +Last known result: ```text upstream-sweep: 16 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 16 compared @@ -326,7 +422,7 @@ upstream-sweep: 16 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 16 c ## Working Method -Use a tight loop: +Use this loop: 1. Run the focused upstream case with `CASE_REGEX`. 2. Inspect the rule and target under `/Users/user/mine/semgrep/tests/rules/`. @@ -336,7 +432,7 @@ Use a tight loop: `src/.jerbuild-hashes`. 6. Run the focused upstream case again. 7. Run `SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle`. -8. Run the broader batch sweep that contains the case. +8. Run `MAX_CASES=220` or the relevant focused cluster sweep. 9. Commit source, generated output, tests, and handoff updates together. Avoid broad taint rewrites. The current taint layer has many passing smoke, @@ -359,16 +455,16 @@ Focused upstream sweep: SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^CASE_NAME$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -First-160 guardrail: +First-220 guardrail: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -Current first-220 batch: +Full upstream sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` Direct Jerboa CLI invocation for one upstream case: @@ -383,8 +479,8 @@ LD_LIBRARY_PATH=/Users/user/mine/jerboa-treesitter \ scan --json --config ../semgrep/tests/rules/CASE.yaml ../semgrep/tests/rules/CASE.py ``` -Use the target extension that actually exists for the case (`.py`, `.js`, -`.ts`, `.tsx`, `.json`, `.txt`, `.generic`, etc.). +Use the target extension that actually exists for the case: `.py`, `.js`, +`.ts`, `.tsx`, `.json`, `.txt`, `.generic`, and so on. ## Coding Notes And Guardrails @@ -394,24 +490,14 @@ Use the target extension that actually exists for the case (`.py`, `.js`, output is part of the runtime state. - Do not revert generated files unless explicitly asked; they are tracked. - In fallback `or` chains, return `#f` for "not applicable" and only return a - list when the fallback is intentionally handling the case. Returning `()` can - short-circuit later matchers. + list when the fallback is intentionally handling the case. Returning `()` + can short-circuit later matchers. - Keep `python-open-redirect-rule?` gated on both rule id `open-redirect` and `symbolic_propagation: true`. - Preserve token/function-scope constraints in taint propagation. Earlier work - deliberately prevents same-named Python locals from leaking across functions. + deliberately prevents same-named Python locals from leaking across + functions. - When symbolic propagation must pass through formula filters or focus ranges, empty bindings are not enough. Preserve or rebase bindings from the expanded expression. -- The packaged Semgrep oracle has 2 current errors in the first-220 sweep. Do - not treat those as Jerboa regressions unless a newer Semgrep oracle succeeds - on the same cases. - -## Immediate Next Step - -Start with `taint_exact_sources`. The remaining work is not import matching; -that now works. Focus on suppressing the two false positives at lines 23 and -41 while preserving the expected reports at lines 28, 33, 49, 60, and 69. -Likely areas are assignment-kill ordering, indexed-field sensitivity, and -whether clean indexed assignments should block the original focused parameter -source until a later genuinely tainted write re-taints the container. +- Remember that this sweep script uses `MAX_CASES`, not `CASE_LIMIT`. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -7863,6 +7863,11 @@ "const { $L } = $R" "let { $L } = $R" "var { $L } = $R" "const { $FIELD: $L } = $R" "let { $FIELD: $L } = $R" "var { $FIELD: $L } = $R")] + [(or (string=? language "python") + (string=? language "python2") + (string=? language "python3") + (string=? language "py")) + '("$L = $R" "$L[$I] = $R" "for $L in $R:\n ...")] [else '("$L = $R" "$L[$I] = $R")])) (def (scan-implicit-assignment-propagators rule language path source target-root) @@ -8489,6 +8494,23 @@ sanitizer-state sink source-text))))))) + (def (assignment-target-matches-source? + source + assignment-finding + to-binding + source-text) + (let ([source-name (finding-text source source-text)]) + (or (and to-binding + (string=? + source-name + (metavariable-binding-text to-binding))) + (let* ([assignment-text (finding-text + assignment-finding + source-text)] + [eq-index (string-find-substring assignment-text "=")] + [lhs (and eq-index + (substring assignment-text 0 eq-index))]) + (and lhs (string-contains-token? lhs source-name)))))) (def (taint-assignment-kills-source? source-state assignment @@ -8519,9 +8541,11 @@ (finding-start-offset source)) (< (finding-start-offset assignment-finding) (finding-start-offset sink)) - (string=? - (finding-text source source-text) - (metavariable-binding-text to-binding))))) + (assignment-target-matches-source? + source + assignment-finding + to-binding + source-text)))) (def (taint-source-killed-before-sink? source-state sink @@ -8535,6 +8559,31 @@ sink source-text)) assignments)) + (def (same-propagator-assignment? left right) + (or (eq? left right) + (let ([left-finding (and left + (alist-ref/default left 'finding #f))] + [right-finding (and right + (alist-ref/default + right + 'finding + #f))]) + (and left-finding + right-finding + (finding-range-equal? left-finding right-finding))))) + (def (taint-source-killed-before-finding? source-state finding assignments excluded-assignment + source-text) + (any? + (lambda (assignment) + (and (not (same-propagator-assignment? + assignment + excluded-assignment)) + (taint-assignment-kills-source? + source-state + assignment + finding + source-text))) + assignments)) (def (source-state-reaches-finding? source-state finding @@ -8776,6 +8825,10 @@ source))))) (def (propagator-applies? rule source-state propagator sanitizers source) + (propagator-applies* rule source-state propagator sanitizers + '() source)) + (def (propagator-applies* rule source-state propagator + sanitizers assignment-kills source) (let* ([finding (alist-ref/default propagator 'finding #f)] [from (alist-ref/default propagator 'from #f)] [to (alist-ref/default propagator 'to #f)] @@ -8802,6 +8855,8 @@ (alist-ref/default propagator 'requires #f)) (source-taints-binding? source-state finding from-binding from source) + (not (taint-source-killed-before-finding? source-state from-finding assignment-kills propagator + source)) (not (propagator-blocked-by-sanitizer? source-state from-finding @@ -8921,7 +8976,7 @@ (cons next acc) acc)))))) (def (new-propagated-sources rule sources known propagators - sanitizers source seed) + sanitizers assignment-kills source seed) (let source-loop ([remaining sources] [acc seed]) (if (null? remaining) (reverse acc) @@ -8930,8 +8985,8 @@ (let prop-loop ([props propagators] [prop-acc acc]) (if (null? props) prop-acc - (let ([next (and (propagator-applies? rule (car remaining) (car props) - sanitizers source) + (let ([next (and (propagator-applies* rule (car remaining) (car props) + sanitizers assignment-kills source) (propagated-source rule (car remaining) @@ -8948,8 +9003,8 @@ prop-acc))) (cons next prop-acc) prop-acc))))))))) - (def (expand-taint-sources rule initial-sources - source-matches propagators sanitizers source) + (def (expand-taint-sources rule initial-sources source-matches propagators sanitizers + assignment-kills source) (let loop ([known initial-sources]) (let* ([required (new-required-sources source-matches @@ -8957,7 +9012,8 @@ source)] [known-with-required (append known required)] [new (new-propagated-sources rule known-with-required known-with-required - propagators sanitizers source required)]) + propagators sanitizers assignment-kills source + required)]) (if (null? new) known (loop (append known new)))))) (def jquery-html-concat-sink-regex "(?:'[^']*'|\"[^\"]*\")[ \\t]*\\+[ \\t]*[A-Za-z_$][A-Za-z0-9_$]*") @@ -9024,7 +9080,7 @@ language path source target-root #f)] [sources (expand-taint-sources rule initial-sources source-matches (append propagators implicit-propagators) - sanitizers source)] + sanitizers implicit-propagators source)] [sinks0 (scan-taint-sinks rule (alist-ref/default taint 'sinks '()) language path source target-root)] --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "471D16BC922EC36F") - ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/scan.ss" . "BDFF8384FA069AE1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "D56FC2D2EB449BA6")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -8382,6 +8382,13 @@ "const { $FIELD: $L } = $R" "let { $FIELD: $L } = $R" "var { $FIELD: $L } = $R")] + [(or (string=? language "python") + (string=? language "python2") + (string=? language "python3") + (string=? language "py")) + '("$L = $R" + "$L[$I] = $R" + "for $L in $R:\n ...")] [else '("$L = $R" "$L[$I] = $R")])) @@ -8975,6 +8982,18 @@ sink source-text))))))) +(def (assignment-target-matches-source? source assignment-finding to-binding source-text) + (let ([source-name (finding-text source source-text)]) + (or (and to-binding + (string=? source-name + (metavariable-binding-text to-binding))) + (let* ([assignment-text (finding-text assignment-finding source-text)] + [eq-index (string-find-substring assignment-text "=")] + [lhs (and eq-index + (substring assignment-text 0 eq-index))]) + (and lhs + (string-contains-token? lhs source-name)))))) + (def (taint-assignment-kills-source? source-state assignment sink source-text) (let* ([source (taint-state-finding source-state)] [assignment-finding (alist-ref/default assignment 'finding #f)] @@ -8992,8 +9011,11 @@ (finding-start-offset source)) (< (finding-start-offset assignment-finding) (finding-start-offset sink)) - (string=? (finding-text source source-text) - (metavariable-binding-text to-binding))))) + (assignment-target-matches-source? + source + assignment-finding + to-binding + source-text)))) (def (taint-source-killed-before-sink? source-state sink assignments source-text) (any? (lambda (assignment) @@ -9004,6 +9026,29 @@ source-text)) assignments)) +(def (same-propagator-assignment? left right) + (or (eq? left right) + (let ([left-finding (and left (alist-ref/default left 'finding #f))] + [right-finding (and right (alist-ref/default right 'finding #f))]) + (and left-finding + right-finding + (finding-range-equal? left-finding right-finding))))) + +(def (taint-source-killed-before-finding? + source-state + finding + assignments + excluded-assignment + source-text) + (any? (lambda (assignment) + (and (not (same-propagator-assignment? assignment excluded-assignment)) + (taint-assignment-kills-source? + source-state + assignment + finding + source-text))) + assignments)) + (def (source-state-reaches-finding? source-state finding source-text) (let ([source (taint-state-finding source-state)]) (and source @@ -9249,6 +9294,15 @@ source))))) (def (propagator-applies? rule source-state propagator sanitizers source) + (propagator-applies* rule source-state propagator sanitizers '() source)) + +(def (propagator-applies* + rule + source-state + propagator + sanitizers + assignment-kills + source) (let* ([finding (alist-ref/default propagator 'finding #f)] [from (alist-ref/default propagator 'from #f)] [to (alist-ref/default propagator 'to #f)] @@ -9272,6 +9326,12 @@ from-binding from source) + (not (taint-source-killed-before-finding? + source-state + from-finding + assignment-kills + propagator + source)) (not (propagator-blocked-by-sanitizer? source-state from-finding @@ -9391,7 +9451,15 @@ (cons next acc) acc)))))) -(def (new-propagated-sources rule sources known propagators sanitizers source seed) +(def (new-propagated-sources + rule + sources + known + propagators + sanitizers + assignment-kills + source + seed) (let source-loop ([remaining sources] [acc seed]) (if (null? remaining) (reverse acc) @@ -9401,11 +9469,12 @@ (if (null? props) prop-acc (let ([next - (and (propagator-applies? rule - (car remaining) - (car props) - sanitizers - source) + (and (propagator-applies* rule + (car remaining) + (car props) + sanitizers + assignment-kills + source) (propagated-source rule (car remaining) (car props) @@ -9426,6 +9495,7 @@ source-matches propagators sanitizers + assignment-kills source) (let loop ([known initial-sources]) (let* ([required (new-required-sources source-matches known source)] @@ -9436,6 +9506,7 @@ known-with-required propagators sanitizers + assignment-kills source required)]) (if (null? new) @@ -9525,6 +9596,7 @@ (append propagators implicit-propagators) sanitizers + implicit-propagators source)] [sinks0 (scan-taint-sinks rule --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -1569,6 +1569,30 @@ (check (length findings) => 1) (check (finding-start-line (car findings)) => 2))) +(test-case "scan taint indexed assignment can re-taint same base" + (let* ([taint-config + "rules:\n - id: demo.taint.indexed.parameter\n mode: taint\n languages: [python]\n message: indexed parameter\n severity: WARNING\n pattern-sources:\n - patterns:\n - pattern: |\n def $FN(..., $PARAMS, ...):\n ...\n - focus-metavariable: $PARAMS\n pattern-sinks:\n - pattern: sink($X)\n"] + [findings + (scan-config-string + taint-config + "python" + "demo.py" + "def f(params):\n params[\"sql\"] = \"x\" % params[\"test\"]\n sink(params)\n\ndef g(params):\n params[\"sql\"] = \"x\"\n sink(params)\n")]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 3))) + +(test-case "scan Python taint propagates through loop variables" + (let* ([taint-config + "rules:\n - id: demo.taint.loop-variable\n mode: taint\n languages: [python]\n message: loop taint\n severity: WARNING\n pattern-sources:\n - pattern: source()\n pattern-sinks:\n - pattern: sink(...)\n"] + [findings + (scan-config-string + taint-config + "python" + "demo.py" + "results = source()\nfor res in results:\n sink(res)\nsink(other)\n")]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 3))) + (test-case "scan taint by-side-effect source reaches wildcard sink" (let* ([taint-config "rules:\n - id: demo.taint.by-side-effect-source\n mode: taint\n languages: [python]\n message: side-effect taint\n severity: WARNING\n pattern-sources:\n - by-side-effect: true\n patterns:\n - pattern: $X = source()\n - focus-metavariable: $X\n pattern-sinks:\n - pattern: sink(...)\n"]