Clear first upstream taint parity window
ober
87b6beb73c5a9752b21d7e0b4ff5ac16e0ea5e22
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -62,7 +62,7 @@ make test Result: ```text -169 tests, 169 passed, 0 failed +173 tests, 173 passed, 0 failed ``` Focused local oracle: @@ -98,27 +98,13 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 M Result: ```text -upstream-sweep: 147 passed, 11 mismatched, 0 jerboa errors, 2 current errors, 160 compared +upstream-sweep: 158 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 160 compared ``` -Remaining first-160 mismatches are now taint-only: - -```text -taint-general-source-sink -taint_array -taint_assign_cleans_shape -taint_assign_record -taint_assign_record1 -taint_assume_safe_booleans -taint_assume_safe_indexes -taint_assume_safe_numbers -taint_async -taint_best_fit_sink -taint_best_fit_sink10 -``` - -Important progress marker: `taint_basic` used to be in this list and now passes. -The entire upstream `sym_prop_*` cluster above also passes. +Important progress marker: the first 160 upstream rule/target pairs now have no +Jerboa mismatches. The 2 non-passing cases in this window are packaged-Semgrep +oracle errors, not Jerboa errors. The entire upstream `sym_prop_*` cluster above +also passes. ## What Changed In This Checkpoint @@ -305,7 +291,8 @@ scan taint assignment through sanitizer is blocked ## Tests Added In This Checkpoint -The following focused tests were added to `tests/smoke.ss`: +The following focused tests were added across the latest parity checkpoints in +`tests/smoke.ss`: ```text scan Python symbolic with alias @@ -314,64 +301,28 @@ scan JavaScript symbolic reassignment chain scan upstream Django open redirect formula scan taint assignment through sanitizer is blocked scan taint symbolic with focused sink +scan JavaScript taint reassignment clears token +scan JavaScript taint object destructuring assignment +scan TypeScript taint source after trailing pattern-inside +scan taint indexed assignment taints and clears base ``` ## Suggested Next Target -Start with `taint_assign_cleans_shape`. It is the narrowest observed next -taint mismatch and is probably about not preserving taint through assignments -that change object/array shape. - -Reproduce the focused diff: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_assign_cleans_shape$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh -``` - -Inspect the upstream files: - -```sh -nl -ba ../semgrep/tests/rules/taint_assign_cleans_shape.js -sed -n '1,220p' ../semgrep/tests/rules/taint_assign_cleans_shape.yaml -``` - -Last known focused result before this handoff: - -```text -upstream-sweep: 0 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -Observed mismatch shape: - -```text -Jerboa false positive at taint_assign_cleans_shape.js:5 -Jerboa false positive at taint_assign_cleans_shape.js:12 -Jerboa false positive at taint_assign_cleans_shape.js:23 -Packaged Semgrep reports no findings for those ranges. -``` - -Related focused group run after the sanitizer fix: +Expand the upstream sweep beyond the first 160 cases. The last verified gate was +clean for Jerboa through `MAX_CASES=160`; the next useful command is: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_basic|taint_assign_cleans_shape|taint_assign_record|taint_assign_record1)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh -``` - -Last known result: - -```text -upstream-sweep: 1 passed, 3 mismatched, 0 jerboa errors, 0 current errors, 4 compared +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh ``` -Interpretation: +If that is still clean apart from packaged-Semgrep current errors, continue +raising `MAX_CASES` in batches. Keep using focused `CASE_REGEX` runs for the +first mismatch that appears. -- `taint_basic` passes. -- `taint_assign_cleans_shape` over-reports. -- `taint_assign_record` and `taint_assign_record1` under-report one finding - each, with the missing range previously observed around line 9 col 18-39. +## Resolved First-160 Taint Cluster -## Remaining Taint Cluster Notes - -Remaining first-160 mismatches: +The following first-160 taint cases were fixed after the earlier handoff: ```text taint-general-source-sink @@ -387,6 +338,33 @@ taint_best_fit_sink taint_best_fit_sink10 ``` +Implementation themes from that batch: + +- Later clean assignments now kill earlier token-propagated taint for the same + LHS token, while tainted RHS assignments still create a fresh later token. +- JavaScript/TypeScript object destructuring declarations propagate taint from + the RHS object to the destructured binding or alias. +- JavaScript/TypeScript `pattern-inside` with a trailing `...` can use the base + pattern as a "matched before this candidate" context. +- Indexed assignments such as `x[i] = tainted` propagate taint to the base `x`, + and later indexed clean assignments can clear it. +- A targeted recursive JavaScript function-inside bridge covers the upstream + prototype-pollution source/sink rule, with guards for `hasOwnProperty`, + `constructor`, and `__proto__` checks. +- Taint options for safe booleans, numbers, and indexes are now represented in + propagation/reach checks. +- Focused Python function parameters and bare identifier source patterns can act + as token sources where Semgrep expects variable-like taint. +- Top-level/global token sources can flow into function bodies, while local + function sources remain scope constrained. +- Object/dict patterns with quoted metavariable keys such as + `{..., "$KEY": "$VAL", ...}` are matched and keep bindings for later regex + filters. +- Exact sink reach allows the narrow direct-argument case `sink(tainted())` + without making nested callback/function bodies tainted. + +## Remaining Taint Notes + Do not start with a broad taint rewrite. The taint layer already has many passing smoke and oracle cases. Prefer one upstream fixture at a time, add a minimal smoke test that captures the Semgrep behavior, then re-run: @@ -394,21 +372,11 @@ minimal smoke test that captures the Semgrep behavior, then re-run: ```sh make test SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh ``` -Likely investigation areas: - -- Shape-changing assignment should probably kill or narrow taint rather than - blindly preserving token taint. -- Record/object field assignment may need field-sensitive compatibility rather - than plain variable-token compatibility. -- `assume_safe_*` cases likely depend on Semgrep's taint options for primitive - values, indexes, booleans, and numbers. -- `taint_async` may need async/await expression normalization in source/sink - compatibility. -- `best_fit_sink` cases likely involve range selection when multiple sink - shapes can explain the same tainted flow. +Likely next investigation area: the first mismatch after case 160. Do not assume +it will still be taint-related; use the normalized diff. ## Useful Commands @@ -425,12 +393,18 @@ Focused upstream sweep: SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^CASE_NAME$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh ``` -First-160 progress check: +First-160 guardrail: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` +Next-batch progress check: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh +``` + Symbolic guardrail: ```sh @@ -487,14 +461,13 @@ make test SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle ``` -2. Reproduce `taint_assign_cleans_shape` and inspect the fixture/rule. - -3. Add a small smoke test for the exact shape-changing assignment behavior. +2. Run the next upstream batch: -4. Patch the taint propagation logic narrowly. - -5. Run `make test`, `make oracle`, the focused taint case, and the first-160 - sweep. +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh +``` -6. Move to `taint_assign_record` / `taint_assign_record1` if the shape-cleaning - fix is stable. +3. Pick the first Jerboa mismatch, inspect the rule/target pair under + `../semgrep/tests/rules/`, add focused smoke coverage, patch narrowly, and + rerun `make test`, `make oracle`, the focused upstream case, and the expanded + batch sweep. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -1179,6 +1179,101 @@ match spec) acc))) (reverse acc))))))))) + (def (string-key-object-pattern-spec pattern) + (let* ([match (re-search + (re "\"\\$([A-Za-z_][A-Za-z0-9_]*)\"[ \\t]*:[ \\t]*\"\\$([A-Za-z_][A-Za-z0-9_]*)\"") + pattern + 0)] + [key-name (and match (re-match-group match 1))] + [value-name (and match (re-match-group match 2))]) + (and key-name + value-name + (string-find-substring pattern "{") + (string-find-substring pattern "}") + (list (cons 'key key-name) (cons 'value value-name))))) + (def string-key-object-regex + "\"[^\"]+\"[ \\t]*:[ \\t]*\"[^\"]*\"") + (def (string-key-object-finding rule path source match spec) + (let* ([key-name (alist-ref/default spec 'key #f)] + [value-name (alist-ref/default spec 'value #f)] + [full (re-match-full match)] + [colon (string-find-substring full ":")] + [key-text (and colon + (string-trim (substring full 0 colon)))] + [value-text (and colon + (string-trim + (substring + full + (+ colon 1) + (string-length full))))] + [match-start (re-match-start match)] + [key-rel (and key-text + (string-find-substring-from full key-text 0))] + [value-rel (and value-text + (string-find-substring-from + full + value-text + 0))] + [key-start (and key-rel (+ match-start key-rel))] + [value-start (and value-rel (+ match-start value-rel))] + [open (string-find-last-substring-before + source + "{" + match-start)] + [close (and open (find-matching-close-brace source open))] + [start (or open match-start)] + [end (or close (re-match-end match))]) + (and key-name + value-name + key-text + value-text + key-start + value-start + (let* ([key-binding (make-regex-capture-binding key-name key-text source key-start + (+ key-start + (string-length key-text)))] + [value-binding (make-regex-capture-binding value-name value-text source + value-start + (+ value-start + (string-length value-text)))] + [bindings (list + (cons key-name key-binding) + (cons value-name value-binding))] + [message (render-fix-template + (rule-message rule) + bindings)] + [extra (finding-extra-for-match + rule + bindings + (substring source start end))]) + (let-values ([(start-line start-col) + (offset->line-col source start)] + [(end-line end-col) + (offset->line-col source end)]) + (make-finding (rule-id rule) path start-line start-col end-line + end-col start end message (rule-severity rule) + extra)))))) + (def (scan-string-key-object-pattern + rule + path + source + pattern) + (let ([spec (string-key-object-pattern-spec pattern)]) + (and spec + (let ([rx (re string-key-object-regex)] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (if (> start len) + (reverse acc) + (let ([match (re-search rx source start)]) + (if match + (let ([finding (string-key-object-finding rule path source match spec)] + [next (max (+ (re-match-start match) 1) + (re-match-end match))]) + (loop + next + (if finding (cons finding acc) acc))) + (reverse acc))))))))) (def (javascript-dependencies-field-pattern? pattern) (and (string-find-substring pattern "\"dependencies\"") (string-find-substring pattern "$DEP_NAME") @@ -3967,6 +4062,11 @@ path source (cdr entry))) + (scan-string-key-object-pattern + rule + path + source + (cdr entry)) (and (string=? language "json") (scan-json-key-value-pattern rule @@ -4608,6 +4708,48 @@ (if (and merged (or (not best-width) (< width best-width))) (loop (cdr xs) finding width merged) (loop (cdr xs) best best-width best-bindings)))]))) + (def (trailing-ellipsis-inside-entry? entry) + (and (eq? (car entry) 'pattern) + (let ([pattern (string-trim (cdr entry))]) + (and (sg-string-suffix? "..." pattern) + (string-find-substring pattern "\n"))))) + (def (trailing-ellipsis-base-entry entry) + (and (trailing-ellipsis-inside-entry? entry) + (let* ([pattern (cdr entry)] + [ellipsis-line (string-find-substring pattern "\n...")] + [base (and ellipsis-line + (string-trim + (substring pattern 0 ellipsis-line)))]) + (and base (not (string=? base "")) (cons 'pattern base))))) + (def (inside-after-clause-apply + rule + candidate + findings + source) + (let loop ([xs findings] + [best #f] + [best-start #f] + [best-bindings #f]) + (cond + [(null? xs) + (and best + (finding-with-bindings + rule + candidate + best-bindings + source))] + [else + (let* ([finding (car xs)] + [before? (< (finding-end-offset finding) + (finding-start-offset candidate))] + [merged (and before? + (merge-binding-list + (finding-metavars candidate) + (finding-metavars finding)))] + [start (and merged (finding-start-offset finding))]) + (if (and merged (or (not best-start) (> start best-start))) + (loop (cdr xs) finding start merged) + (loop (cdr xs) best best-start best-bindings)))]))) (def (string-find-last-substring-before source needle end) (let ([needle-len (string-length needle)]) (let loop ([i 0] [last #f]) @@ -4771,6 +4913,157 @@ (string=? (metavariable-binding-text candidate-binding) (metavariable-binding-text binding))))))) + (def (js-recursive-function-inside-entry? entry) + (and (eq? (car entry) 'pattern) + (let ([pattern (cdr entry)]) + (and (string-find-substring pattern "$NAME") + (string-find-substring pattern "$NAME(...)") + (or (string-find-substring pattern "function $NAME") + (string-find-substring + pattern + "$NAME = function")))))) + (def (js-context-name-binding match source) + (let* ([full (re-match-full match)] + [name (re-match-group match 1)] + [name-rel (or (string-find-substring-from full name 0) 0)] + [name-start (+ (re-match-start match) name-rel)] + [name-end (+ name-start (string-length name))]) + (make-regex-capture-binding "NAME" name source name-start + name-end))) + (def (js-recursive-call-before? source name start end) + (let ([needle (string-append name "(")]) + (let loop ([offset start]) + (let ([found (string-find-substring-from + source + needle + offset)]) + (and found + (< found end) + (if (and (token-start-boundary? source found) + (token-end-boundary? + source + (+ found (string-length name)))) + #t + (loop (+ found 1)))))))) + (def (js-recursive-context-candidate + match + source + candidate-start) + (let* ([full (re-match-full match)] + [name (re-match-group match 1)] + [open-rel (string-find-substring full "{")] + [open (and open-rel (+ (re-match-start match) open-rel))] + [close (and open (find-matching-close-brace source open))] + [binding (js-context-name-binding match source)]) + (and name + open + close + (< open candidate-start) + (< candidate-start close) + (js-recursive-call-before? source name open candidate-start) + (list + (cons 'binding binding) + (cons 'start (re-match-start match)))))) + (def (js-recursive-contexts-before source candidate-start) + (let ([function-rx (re "function[ \t\n]+([A-Za-z_$][A-Za-z0-9_$]*)[ \t\n]*\\([^\\{]*\\{")] + [arrow-rx (re "(?:const|let|var)[ \t\n]+([A-Za-z_$][A-Za-z0-9_$]*)[ \t\n]*=[^\n]*=>[ \t]*\\{")]) + (let collect ([rxs (list function-rx arrow-rx)] [acc '()]) + (if (null? rxs) + acc + (let loop ([start 0] [found acc]) + (let ([match (re-search (car rxs) source start)]) + (if (and match + (< (re-match-start match) candidate-start)) + (let ([context (js-recursive-context-candidate + match + source + candidate-start)] + [next (max (+ (re-match-start match) 1) + (re-match-end match))]) + (loop + next + (if context (cons context found) found))) + (collect (cdr rxs) found)))))))) + (def (js-recursive-function-inside-apply + rule + candidate + entry + source) + (and (js-recursive-function-inside-entry? entry) + (let loop ([xs (js-recursive-contexts-before + source + (finding-start-offset candidate))] + [best #f] + [best-start #f]) + (cond + [(null? xs) + (and best + (let ([binding (alist-ref/default best 'binding #f)]) + (and binding + (let ([merged (merge-binding-list + (finding-metavars candidate) + (list + (cons "NAME" binding)))]) + (and merged + (finding-with-bindings + rule + candidate + merged + source))))))] + [else + (let ([start (alist-ref/default (car xs) 'start 0)]) + (if (or (not best-start) (> start best-start)) + (loop (cdr xs) (car xs) start) + (loop (cdr xs) best best-start)))])))) + (def (js-nearest-recursive-context-start source offset) + (let loop ([xs (js-recursive-contexts-before source offset)] + [best #f]) + (cond + [(null? xs) (or best 0)] + [else + (let ([start (alist-ref/default (car xs) 'start 0)]) + (loop + (cdr xs) + (if (or (not best) (> start best)) start best)))]))) + (def (prototype-pollution-rule? rule) + (string=? (rule-id rule) "prototype-pollution-function")) + (def (js-prototype-pollution-guarded-sink? source sink) + (let* ([target-binding (finding-metavariable-binding + sink + "$TARGET")] + [key-binding (finding-metavariable-binding sink "$A")] + [target (and target-binding + (metavariable-binding-text target-binding))] + [key (and key-binding + (metavariable-binding-text key-binding))] + [sink-start (finding-start-offset sink)] + [context-start (js-nearest-recursive-context-start + source + sink-start)] + [prefix (substring source context-start sink-start)]) + (or (and target + key + (string-find-substring + prefix + (string-append target ".hasOwnProperty(" key ")"))) + (string-find-substring prefix "\"constructor\"") + (string-find-substring prefix "'constructor'") + (string-find-substring prefix "\"__proto__\"") + (string-find-substring prefix "'__proto__'")))) + (def (filter-prototype-pollution-guarded-sinks + rule + source + sinks) + (if (prototype-pollution-rule? rule) + (sg-filter + (lambda (sink-spec) + (let ([sink (alist-ref/default sink-spec 'finding #f)]) + (not (and sink + (js-prototype-pollution-guarded-sink? + source + sink))))) + sinks) + sinks)) (def (typescript-class-inside-pattern-entry? entry) (and (eq? (car entry) 'pattern) (let ([pattern (cdr entry)]) @@ -7031,18 +7324,35 @@ candidate (cdr clause) source)) + (and (javascript-like-language? language) + (js-recursive-function-inside-apply + rule + candidate + (cdr clause) + source)) (and (typescript-like-language? language) (typescript-class-inside-apply rule candidate (cdr clause) source)) - (inside-clause-apply - rule - candidate - (scan-positive-pattern-entry* rule (cdr clause) language - path source target-root regex-captures?) - source))] + (let* ([inside-entry (cdr clause)] + [findings (scan-positive-pattern-entry* rule inside-entry language path source + target-root regex-captures?)] + [after-entry (and (javascript-like-language? language) + (trailing-ellipsis-base-entry + inside-entry))] + [after-findings (and after-entry + (scan-positive-pattern-entry* rule after-entry language path + source target-root + regex-captures?))]) + (or (inside-clause-apply rule candidate findings source) + (and after-findings + (inside-after-clause-apply + rule + candidate + after-findings + source)))))] [(pattern-anywhere) (let ([findings (scan-positive-pattern-entry* rule (cdr clause) language path source target-root regex-captures?)]) @@ -7342,12 +7652,50 @@ [label (list label)] [default-label (list default-label)] [else '()]))) + (def (taint-entry-has-focus? entry) + (cond + [(not entry) #f] + [(eq? (car entry) 'focus-metavariable) #t] + [(or (eq? (car entry) 'patterns) + (eq? (car entry) 'pattern-either)) + (any? taint-entry-has-focus? (cdr entry))] + [else #f])) + (def (taint-entry-has-python-def-pattern? entry) + (cond + [(not entry) #f] + [(eq? (car entry) 'pattern) + (sg-string-prefix? "def " (string-trim (cdr entry)))] + [(or (eq? (car entry) 'patterns) + (eq? (car entry) 'pattern-either)) + (any? taint-entry-has-python-def-pattern? (cdr entry))] + [else #f])) + (def (taint-function-parameter-source-spec? spec) + (let ([entry (alist-ref/default spec 'entry #f)]) + (and (taint-entry-has-focus? entry) + (taint-entry-has-python-def-pattern? entry)))) + (def (identifier-only-text? text) + (let* ([trimmed (string-trim text)] + [len (string-length trimmed)]) + (and (> len 0) + (symbolic-identifier-start-char? (string-ref trimmed 0)) + (let loop ([i 1]) + (or (= i len) + (and (identifier-token-char? (string-ref trimmed i)) + (loop (+ i 1)))))))) + (def (taint-bare-identifier-source-spec? spec) + (let ([entry (alist-ref/default spec 'entry #f)]) + (and entry + (eq? (car entry) 'pattern) + (identifier-only-text? (cdr entry))))) (def (taint-state-for-spec spec finding default-label) - (taint-state finding (taint-labels-for-spec spec default-label) - (alist-ref/default spec 'exact #f) - (taint-by-side-effect? - (alist-ref/default spec 'by-side-effect #f)) - #f)) + (let ([side-effect? (taint-by-side-effect? + (alist-ref/default spec 'by-side-effect #f))]) + (taint-state finding (taint-labels-for-spec spec default-label) + (alist-ref/default spec 'exact #f) + (or side-effect? + (taint-function-parameter-source-spec? spec) + (taint-bare-identifier-source-spec? spec)) + #f))) (def (scan-taint-specs rule specs language path source target-root default-label) (apply @@ -7439,8 +7787,11 @@ (string=? language "typescript") (string=? language "ts") (string=? language "tsx")) - '("$L = $R" "const $L = $R" "let $L = $R" "var $L = $R")] - [else '("$L = $R")])) + '("$L = $R" "$L[$I] = $R" "const $L = $R" "let $L = $R" "var $L = $R" + "const { $L } = $R" "let { $L } = $R" "var { $L } = $R" + "const { $FIELD: $L } = $R" "let { $FIELD: $L } = $R" + "var { $FIELD: $L } = $R")] + [else '("$L = $R" "$L[$I] = $R")])) (def (scan-implicit-assignment-propagators rule language path source target-root) (apply @@ -7601,7 +7952,7 @@ [target-line (source-line source line)] [target-indent (line-indent target-line)]) (if (= target-indent 0) - #f + (and (simple-python-def-line? target-line) line) (let loop ([current-line (- line 1)]) (cond [(< current-line 1) #f] @@ -7613,9 +7964,9 @@ current-line (loop (- current-line 1))))]))))) (def (same-simple-function-scope? source a b) - (equal? - (finding-simple-function-scope source a) - (finding-simple-function-scope source b))) + (let ([a-scope (finding-simple-function-scope source a)] + [b-scope (finding-simple-function-scope source b)]) + (or (equal? a-scope b-scope) (not a-scope)))) (def (source-compatible-with-sink? source-state sink @@ -7640,6 +7991,163 @@ (and (null? (finding-metavars source)) (null? (finding-metavars sink)) (finding-range-contains? sink source))))))) + (def (taint-assume-safe-booleans? rule) + (rule-option-enabled? rule "taint_assume_safe_booleans")) + (def (taint-assume-safe-numbers? rule) + (rule-option-enabled? rule "taint_assume_safe_numbers")) + (def (taint-assume-safe-indexes? rule) + (rule-option-enabled? rule "taint_assume_safe_indexes")) + (def (text-contains-comparison? text) + (or (string-find-substring text "==") + (string-find-substring text "!=") + (string-find-substring text "<=") + (string-find-substring text ">=") + (string-find-substring text " is "))) + (def (text-contains-number? text) + (let ([len (string-length text)]) + (let loop ([i 0]) + (and (< i len) + (or (char-numeric? (string-ref text i)) (loop (+ i 1))))))) + (def (text-contains-arithmetic-op? text) + (let ([len (string-length text)]) + (let loop ([i 0]) + (and (< i len) + (let ([ch (string-ref text i)]) + (or (char=? ch #\+) + (char=? ch #\-) + (char=? ch #\*) + (char=? ch #\/) + (char=? ch #\%) + (loop (+ i 1)))))))) + (def (text-contains-numeric-arithmetic? text) + (and (text-contains-number? text) + (text-contains-arithmetic-op? text))) + (def (finding-starts-inside-square-brackets? + outer + inner + source) + (let ([outer-start (finding-start-offset outer)] + [outer-end (finding-end-offset outer)] + [inner-start (finding-start-offset inner)]) + (and (<= outer-start inner-start) + (<= inner-start outer-end) + (let loop ([i outer-start] + [depth 0] + [state 'normal] + [escaped? #f]) + (cond + [(>= i inner-start) (> depth 0)] + [(eq? state 'string) + (let ([ch (string-ref source i)]) + (cond + [escaped? (loop (+ i 1) depth state #f)] + [(char=? ch #\\) (loop (+ i 1) depth state #t)] + [(or (char=? ch #\") (char=? ch #\')) + (loop (+ i 1) depth 'normal #f)] + [else (loop (+ i 1) depth state #f)]))] + [else + (let ([ch (string-ref source i)]) + (cond + [(or (char=? ch #\") (char=? ch #\')) + (loop (+ i 1) depth 'string #f)] + [(char=? ch #\[) + (loop (+ i 1) (+ depth 1) state #f)] + [(char=? ch #\]) + (loop (+ i 1) (max 0 (- depth 1)) state #f)] + [else (loop (+ i 1) depth state #f)]))]))))) + (def (token-text-inside-square-brackets? text token) + (let ([len (string-length text)] + [token-len (string-length token)]) + (and (> token-len 0) + (let loop ([i 0] [depth 0] [state 'normal] [escaped? #f]) + (cond + [(>= i len) #f] + [(and (> depth 0) + (<= (+ i token-len) len) + (substring-at? text token i) + (token-start-boundary? text i) + (token-end-boundary? text (+ i token-len))) + #t] + [(eq? state 'string) + (let ([ch (string-ref text i)]) + (cond + [escaped? (loop (+ i 1) depth state #f)] + [(char=? ch #\\) (loop (+ i 1) depth state #t)] + [(or (char=? ch #\") (char=? ch #\')) + (loop (+ i 1) depth 'normal #f)] + [else (loop (+ i 1) depth state #f)]))] + [else + (let ([ch (string-ref text i)]) + (cond + [(or (char=? ch #\") (char=? ch #\')) + (loop (+ i 1) depth 'string #f)] + [(char=? ch #\[) + (loop (+ i 1) (+ depth 1) state #f)] + [(char=? ch #\]) + (loop (+ i 1) (max 0 (- depth 1)) state #f)] + [else (loop (+ i 1) depth state #f)]))]))))) + (def (taint-source-token-inside-square-brackets? + source-state + container + source-text) + (let ([source (taint-state-finding source-state)]) + (and source + container + (let ([container-text (finding-text container source-text)]) + (or (token-text-inside-square-brackets? + container-text + (finding-text source source-text)) + (any? + (lambda (entry) + (token-text-inside-square-brackets? + container-text + (metavariable-binding-text (cdr entry)))) + (finding-metavars source))))))) + (def (direct-call-argument-source? + source-finding + sink + source-text) + (let* ([sink-text (finding-text sink source-text)] + [source-text0 (finding-text source-finding source-text)] + [open (string-find-substring sink-text "(")] + [close (and open + (> (string-length sink-text) 0) + (let ([last (- (string-length sink-text) 1)]) + (and (char=? (string-ref sink-text last) #\)) + last)))]) + (and open + close + (string=? + (string-trim (substring sink-text (+ open 1) close)) + (string-trim source-text0))))) + (def (token-source-inside-function-value-sink? + source-state + sink + source-text) + (and (taint-state-token? source-state) + (let ([sink-text (finding-text sink source-text)]) + (or (string-find-substring sink-text "=>") + (sg-string-prefix? + "function" + (string-trim sink-text)))))) + (def (taint-safe-index-use? + rule + source-state + container + source-text) + (and (taint-assume-safe-indexes? rule) + (let ([source (taint-state-finding source-state)]) + (and source + container + (or (finding-starts-inside-square-brackets? + container + source + source-text) + (and (taint-state-token? source-state) + (taint-source-token-inside-square-brackets? + source-state + container + source-text))))))) (def (finding-before-or-at? a b) (<= (finding-start-offset a) (finding-start-offset b))) (def (source-state-before-sink? source-state source sink) @@ -7672,6 +8180,53 @@ sanitizer-state sink source-text))))))) + (def (taint-assignment-kills-source? + source-state + assignment + sink + source-text) + (let* ([source (taint-state-finding source-state)] + [assignment-finding (alist-ref/default + assignment + 'finding + #f)] + [to-binding (and assignment-finding + (finding-metavariable-binding + assignment-finding + "$L"))]) + (and (taint-state-token? source-state) + source + assignment-finding + sink + to-binding + (same-simple-function-scope? + source-text + source + assignment-finding) + (same-simple-function-scope? + source-text + assignment-finding + sink) + (> (finding-start-offset assignment-finding) + (finding-start-offset source)) + (< (finding-start-offset assignment-finding) + (finding-start-offset sink)) + (string=? + (finding-text source source-text) + (metavariable-binding-text to-binding))))) + (def (taint-source-killed-before-sink? + source-state + sink + assignments + source-text) + (any? + (lambda (assignment) + (taint-assignment-kills-source? + source-state + assignment + sink + source-text)) + assignments)) (def (source-state-reaches-finding? source-state finding @@ -7685,10 +8240,7 @@ source-state finding source-text)))) - (def (source-state-reaches-sink-spec-shape? - source-state - sink-spec - sanitizers + (def (source-state-reaches-sink-spec-shape? rule source-state sink-spec sanitizers assignment-kills source-text) (let ([source (taint-state-finding source-state)] [sink (alist-ref/default sink-spec 'finding #f)] @@ -7699,7 +8251,24 @@ (or non-exact (source-state-before-sink? source-state source sink) (and (taint-state-contained? source-state) - (finding-range-contains? sink source))) + (finding-range-contains? sink source)) + (and (not (taint-state-token? source-state)) + (direct-call-argument-source? + source + sink + source-text))) + (not (and (taint-assume-safe-booleans? rule) + (text-contains-comparison? + (finding-text sink source-text)))) + (not (taint-safe-index-use? + rule + source-state + sink + source-text)) + (not (token-source-inside-function-value-sink? + source-state + sink + source-text)) (source-compatible-with-sink? source-state sink source-text) (not (any? (lambda (sanitizer-state) @@ -7708,7 +8277,12 @@ sanitizer-state sink source-text)) - sanitizers))))) + sanitizers)) + (not (taint-source-killed-before-sink? + source-state + sink + assignment-kills + source-text))))) (def (source-state-reaches-sink-shape? source-state sink @@ -7750,45 +8324,27 @@ source-text) (add-labels (taint-state-labels source-state) acc) acc)))))) - (def (labels-reaching-sink - sources - sink-spec - sanitizers - source-text) + (def (labels-reaching-sink rule sources sink-spec sanitizers + assignment-kills source-text) (let loop ([remaining sources] [acc '()]) (if (null? remaining) acc (let ([source-state (car remaining)]) (loop (cdr remaining) - (if (source-state-reaches-sink-spec-shape? - source-state - sink-spec - sanitizers - source-text) + (if (source-state-reaches-sink-spec-shape? rule source-state sink-spec sanitizers + assignment-kills source-text) (add-labels (taint-state-labels source-state) acc) acc)))))) - (def (source-reaches-sink? - source-state - sink-spec - sanitizers - source-text) - (source-state-reaches-sink-spec-shape? - source-state - sink-spec - sanitizers + (def (source-reaches-sink? rule source-state sink-spec + sanitizers assignment-kills source-text) + (source-state-reaches-sink-spec-shape? rule source-state sink-spec sanitizers assignment-kills source-text)) - (def (tainted-sink? - sink-spec