Clear Express SQL taint frontier
ober
8b8005b788f82dbd3ae02212ffa6ef6829abbec9
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,6 +1,6 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 00:15 MDT local workspace time +Date: 2026-05-29 00:40 MDT local workspace time Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` @@ -31,6 +31,7 @@ taint/dataflow, path and target semantics, autofix, and output schemas. Recent checkpoints before this handoff commit: ```text +38bfcf1 Clear destructuring taint frontier 19f45f7 Advance lambda and source taint parity 7cfb751 Clear lambda taint label reach 31db293 Clear taint label branch frontier @@ -74,7 +75,7 @@ make test Result: ```text -194 tests, 194 passed, 0 failed +195 tests, 195 passed, 0 failed ``` Local oracle: @@ -89,50 +90,51 @@ Result: oracle: 42 passed, 0 failed ``` -Focused destructuring and assignment-only taint guardrail for this checkpoint: +Focused route-template, destructuring, and assignment-only taint guardrail for +this checkpoint: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_nested_record_pattern|taint_no_builtin_props|taint_no_builtin_props1|taint_object_destructure)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_object_funcdef|taint_nested_record_pattern|taint_no_builtin_props|taint_no_builtin_props1|taint_object_destructure)$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 4 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 4 compared +upstream-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 5 compared ``` -This confirms `taint_nested_record_pattern`, `taint_no_builtin_props`, -`taint_no_builtin_props1`, and `taint_object_destructure` are clean in the -current tree. +This confirms `taint_object_funcdef`, `taint_nested_record_pattern`, +`taint_no_builtin_props`, `taint_no_builtin_props1`, and +`taint_object_destructure` are clean in the current tree. -Focused upstream guardrail containing the field-sensitive cases fixed so far: +Focused route-template case cleared in this checkpoint: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_field_sensitive1|taint_field_sensitive2|taint_field_sensitive3|taint_field_sensitive4|taint_field_sensitive6|taint_field_sensitive7|taint_field_sensitive8)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_funcdef$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 7 compared +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` -Nearby regression guardrail: +Focused upstream guardrail containing the field-sensitive cases fixed so far: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_field_sensitive2|taint_field_sensitive1|taint_exact_sources)$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_field_sensitive1|taint_field_sensitive2|taint_field_sensitive3|taint_field_sensitive4|taint_field_sensitive6|taint_field_sensitive7|taint_field_sensitive8)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared +upstream-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 7 compared ``` -Focused label guardrail cleared in the previous checkpoint: +Nearby regression guardrail: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_labels2|taint_labels3|taint_typestate|taint_source_requires|taint_source_requires_reopen)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_field_sensitive2|taint_field_sensitive1|taint_exact_sources)$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh ``` Result: @@ -141,59 +143,70 @@ Result: upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` -Current first-220 upstream sweep: +Focused label guardrail cleared in the previous checkpoint: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=220 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_labels2|taint_labels3|taint_typestate|taint_source_requires|taint_source_requires_reopen)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 205 passed, 13 mismatched, 0 jerboa errors, 2 current errors, 220 compared +upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` -The full 241-case sweep was not rerun after this checkpoint. Do not treat the -following stale all-case result as current; it is preserved only as orientation -from a previous handoff before the recent `taint_exception`, field-sensitive, -label, and lambda fixes: +Current full upstream sweep: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_LIMIT=220 LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_LIMIT=220 LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 197 passed, 42 mismatched, 0 jerboa errors, 2 current errors, 241 compared +upstream-sweep: 216 passed, 23 mismatched, 0 jerboa errors, 2 current errors, 241 compared ``` -The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner -errors. +Important: `CASE_LIMIT` is not used by `tests/oracle/upstream-sweep.sh`; this +command effectively ran the full sorted 241-case upstream rule sweep. Use +`MAX_CASES=220` when a first-220-only guardrail is needed. The two current +errors are packaged-Semgrep oracle errors, not Jerboa scanner errors. ## What Changed In This Checkpoint -This checkpoint clears `taint_nested_record_pattern`, `taint_no_builtin_props`, -`taint_no_builtin_props1`, and `taint_object_destructure`, building on -`19f45f7` which cleared `taint_lambda4` and `taint_match_on_source`. +This checkpoint clears `taint_object_funcdef`, building on `38bfcf1` which +cleared `taint_nested_record_pattern`, `taint_no_builtin_props`, +`taint_no_builtin_props1`, and `taint_object_destructure`. Implementation changes in `src/semgrep/scan.ss`: -- JS/TS implicit assignment propagation now includes one-level nested object - destructuring declarations such as `const { body: { param } } = tainted`. -- Built-in implicit assignment propagators now honor - `taint_only_propagate_through_assignments`: when the option is set, a - built-in assignment propagates only if the RHS binding text is exactly the - current source text. This preserves `x = source(); y = x` while suppressing - propagation through `x.a`, `x()`, `y + "ok"`, `f(y)`, and `a[y]`. -- JS arrow function source matching now handles `({ $X }, $A) => $Z` and - emits one focused source for each destructured object parameter property. -- The arrow destructuring fallback exports only the focused `$X` binding so the - sibling parameter `$A` does not become tainted. +- `regex-pattern-for-engine` now accepts embedded `(?i)` in addition to a + leading `(?i)`. The Semgrep SQL keyword regex in `taint_object_funcdef` is + shaped as `.*\b(?i)(select|delete|insert|create|update|alter|drop)\b.*`; + Jerboa's regex engine previously raised an internal pregexp error for that + form. The implemented behavior treats embedded `(?i)` as "case-insensitive + from here onward", which is enough for this fixture and compatible with the + common Semgrep rule idiom. +- JavaScript/TypeScript positive pattern scanning now has a targeted template + literal fallback for the sink shape `` `$SQLSTR${$EXPR}...` ``. It finds + backtick template literals, binds `$SQLSTR` to the static prefix before the + interpolation, binds `$EXPR` to the trimmed expression inside `${...}`, and + returns a whole-template finding so the later `focus-metavariable: $EXPR` + clause narrows the reported range to the expression. +- `pattern-inside` now has a route-callback fallback for the exact + `$APP.$METHOD(..., function $FUNC($REQ, $RES) {...})` pattern family. It also + recognizes Express-style arrow callbacks inside the call, including + `app.get('/ok', async (req, res) => { ... })`, and merges `$APP`, `$METHOD`, + `$REQ`, and `$RES` bindings onto the source candidate. +- The route-callback fallback intentionally runs before the older generic + function-parameter fallback. The generic fallback could prove containment + for this rule but only preserved `$REQ`; the later `$METHOD` regex filter + then dropped the source candidate. New smoke coverage in `tests/smoke.ss`: ```text +scan JavaScript taint Express params into SQL templates scan JavaScript taint nested object destructuring assignment scan taint only propagates through plain assignments scan JavaScript taint destructured arrow object parameters @@ -218,6 +231,80 @@ subtrees. The ancestor-clean test mirrors `taint_field_sensitive4`: ## Resolved Recent Cases +`taint_object_funcdef` is now cleared: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_funcdef$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh +``` + +Current result: + +```text +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared +``` + +Target file: + +```text +/Users/user/mine/semgrep/tests/rules/taint_object_funcdef.js +``` + +Rule shape: + +```yaml +pattern-sources: + - patterns: + - pattern-either: + - pattern-inside: $APP.$METHOD(..., function $FUNC($REQ, $RES) {...}) + - metavariable-regex: + metavariable: $METHOD + regex: ^(get|post|put|head|delete|options)$ + - pattern: $REQ.params +pattern-sinks: + - patterns: + - pattern-either: + - patterns: + - pattern-either: + - pattern: | + `$SQLSTR${$EXPR}...` + - metavariable-regex: + metavariable: $SQLSTR + regex: .*\b(?i)(select|delete|insert|create|update|alter|drop)\b.* + - focus-metavariable: $EXPR +``` + +Relevant target shape: + +```javascript +app.get('/ok', async (req, res) => { + const tainted = req.params; + async.parallel({ + somefunc(cb) { + return `SELECT * FROM ${tainted}` + } + }) +}) +``` + +Expected matched lines are now exactly: + +```text +6, 17 +``` + +What changed: + +- Simple `pattern: $REQ.params` already bound `$REQ=req`; the missing source + side was the Express callback `pattern-inside` context and the `$METHOD` + binding needed by the route-method regex. +- The generic function-parameter `pattern-inside` fallback was proving enough + containment to short-circuit the route-specific fallback, but it did not + bind `$METHOD`. The route fallback now runs first for this pattern shape. +- The sink side needed template-literal support for + `` `$SQLSTR${$EXPR}...` `` plus an embedded `(?i)` regex flag. The template + fallback binds `$SQLSTR="SELECT * FROM "` and `$EXPR=tainted`; the later + focus clause reports the `tainted` expression at columns 31 to 38. + `taint_nested_record_pattern` is now cleared: ```sh @@ -703,12 +790,12 @@ What changed: - It also avoids the branch-union false positive in `boo()`, where `a = p()` and `a = q()` occur in mutually exclusive `if`/`else` branches. -## Current First-220 Frontier +## Current Full-Sweep Frontier -The current first-220 sweep has these 13 mismatches: +The current full sorted upstream sweep compared 241 rule/target pairs and has +these 23 mismatches: ```text -taint_object_funcdef taint_object_funcdef_1 taint_param_default taint_param_default1 @@ -721,12 +808,6 @@ taint_propagator1 taint_propagator2 taint_react taint_safe_comparisons -``` - -The full 241-case sweep was not rerun after this checkpoint. Based on the -last full run, expect these 11 later mismatches after the first-220 window: - -```text taint_spread_record_big taint_spread_record_small taint_unify_mvars @@ -740,24 +821,67 @@ vardef_assign_true1 vardef_assign_true2 ``` -## Immediate Next Case: taint_object_funcdef +Current full-sweep result: + +```text +upstream-sweep: 216 passed, 23 mismatched, 0 jerboa errors, 2 current errors, 241 compared +``` + +The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner +errors. The command accidentally used `CASE_LIMIT=220`, which this script +ignores, so the result above is a full sweep. For a first-window guardrail, +use `MAX_CASES=220`. + +## Immediate Next Case: taint_object_funcdef_1 Focused command: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_funcdef$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_object_funcdef_1$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh ``` Current result: ```text -MISMATCH taint_object_funcdef -@@ -1,2 +0,0 @@ --(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_object_funcdef.js" 17 31 358 17 38 365 "ERROR" "Matched SQL injection." "") --(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_object_funcdef.js" 6 31 157 6 38 164 "ERROR" "Matched SQL injection." "") +MISMATCH taint_object_funcdef_1 +@@ -1 +0,0 @@ +-(finding "test" "/Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.ts" 8 33 209 16 3 556 "WARNING" "Semgrep found a match" "") upstream-sweep: 0 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 1 compared ``` +Rule and target files: + +```text +/Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.yaml +/Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.ts +``` + +Why this should be next: + +- It is adjacent to the case just cleared and likely reuses some of the route + callback and SQL sink groundwork. +- It is TypeScript and reports a much wider sink range, so do not assume the + `taint_object_funcdef` fix fully generalizes. +- Start by isolating source formula findings and sink formula findings exactly + as was done for `taint_object_funcdef`. + +Useful inspection commands: + +```sh +sed -n '1,220p' /Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.yaml +sed -n '1,220p' /Users/user/mine/semgrep/tests/rules/taint_object_funcdef_1.ts +``` + +The next likely question is whether Jerboa misses the TypeScript source +context, the sink range, or both. Avoid broad taint rewrites until that split +is known. + +## Recently Cleared Case Notes: taint_object_funcdef + +The old immediate-next case was `taint_object_funcdef`; it is now clean. The +details below are kept because the next case is adjacent and may need the same +debug pattern. + Rule: ```yaml @@ -819,7 +943,7 @@ Packaged Semgrep expects: 6, 17 ``` -Jerboa currently reports no findings: +Before this checkpoint Jerboa reported no findings: ```diff @@ -1,2 +0,0 @@ @@ -827,20 +951,18 @@ Jerboa currently reports no findings: -(finding "test" ".../taint_object_funcdef.js" 6 31 ... "ERROR" "Matched SQL injection." "") ``` -Isolation already done: - -- Simple `pattern: $REQ.params` finds `req.params` and binds `$REQ=req`. -- The full source formula returns no findings. The likely missing source side - is `pattern-inside: $APP.$METHOD(..., function $FUNC($REQ, $RES) {...})` - matching Express arrow callbacks like `app.get('/ok', async (req, res) => { ... })`. -- The sink formula also returns no findings for - ``return `SELECT * FROM ${tainted}` ``. The likely missing sink side is - template-string support for the pattern `` `$SQLSTR${$EXPR}...` `` focused - to `$EXPR`, with `$SQLSTR` satisfying the SQL keyword regex. -- Fix this in two small pieces if possible: first make the route callback - context produce the focused source, then make the SQL template sink produce - a focused `$EXPR` finding. Add smoke coverage that expects `tainted` inside - the template string to be the reported sink range. +Isolation done during the fix: + +- Simple `pattern: $REQ.params` found `req.params` and bound `$REQ=req`. +- The full source formula returned no findings because the generic + function-param inside fallback shadowed the route-specific context and did + not bind `$METHOD`. +- The sink formula needed two pieces: template-string support for the pattern + `` `$SQLSTR${$EXPR}...` `` focused to `$EXPR`, and embedded `(?i)` regex flag + handling for the SQL keyword filter. +- After the patch, the full source formula binds `$METHOD=get`, the sink + formula reports focused `$EXPR=tainted`, and the taint rule matches lines 6 + and 17. ## Resolved Earlier Frontier --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -251,11 +251,22 @@ (loop (+ j 1) (cons #\( acc))))] [else (loop (+ i 1) (cons (string-ref pattern i) acc))])))) (def (regex-pattern-for-engine pattern) - (if (regex-leading-case-insensitive? pattern) - (regex-pattern-for-engine-raw - (regex-expand-case-insensitive - (substring pattern 4 (string-length pattern)))) - (regex-pattern-for-engine-raw pattern))) + (cond + [(regex-leading-case-insensitive? pattern) + (regex-pattern-for-engine-raw + (regex-expand-case-insensitive + (substring pattern 4 (string-length pattern))))] + [(string-find-substring pattern "(?i)") => + (lambda (index) + (string-append + (regex-pattern-for-engine-raw (substring pattern 0 index)) + (regex-pattern-for-engine-raw + (regex-expand-case-insensitive + (substring + pattern + (+ index 4) + (string-length pattern))))))] + [else (regex-pattern-for-engine-raw pattern)])) (def (substring-at? s needle i) (let ([needle-len (string-length needle)]) (and (<= (+ i needle-len) (string-length s)) @@ -4239,6 +4250,12 @@ source (cdr entry))) (and (javascript-like-language? language) + (scan-javascript-template-sql-expression-pattern + rule + path + source + (cdr entry))) + (and (javascript-like-language? language) (scan-javascript-new-function-sequence-pattern rule path @@ -4427,6 +4444,8 @@ (and (javascript-like-language? language) (scan-javascript-member-ellipsis-plus-pattern-with-bindings rule path source (cdr entry) initial-bindings)) (and (javascript-like-language? language) + (scan-javascript-template-sql-expression-pattern-with-bindings rule path source (cdr entry) initial-bindings)) + (and (javascript-like-language? language) (scan-javascript-new-function-sequence-pattern rule path @@ -5344,6 +5363,249 @@ candidate merged source)))))))) + (def (js-route-callback-inside-entry? entry) + (and (eq? (car entry) 'pattern) + (let ([pattern (cdr entry)]) + (and (string-find-substring pattern "$APP.$METHOD") + (string-find-substring pattern "function $FUNC") + (string-find-substring pattern "$REQ") + (string-find-substring pattern "$RES") + (string-find-substring pattern "{...}"))))) + (def (js-simple-param-bindings-in-range + source + start + end + names) + (let loop ([i start] [remaining names] [acc '()]) + (cond + [(null? remaining) (reverse acc)] + [else + (let ([j (js-skip-horizontal-and-newline source i end)]) + (cond + [(>= j end) #f] + [(char=? (string-ref source j) #\,) + (loop (+ j 1) remaining acc)] + [(identifier-char? (string-ref source j)) + (let* ([name-end (access-path-identifier-end source j)] + [binding (make-regex-capture-binding (car remaining) + (substring source j name-end) source j + name-end)] + [next (js-find-next-comma-or-end + source + name-end + end)]) + (loop + next + (cdr remaining) + (cons (cons (car remaining) binding) acc)))] + [else #f]))]))) + (def (js-route-callback-param-bindings + source + param-start + param-end + desired-req) + (let ([bindings (js-simple-param-bindings-in-range + source + param-start + param-end + '("REQ" "RES"))]) + (and bindings + (let ([req-binding (assoc "REQ" bindings)]) + (and req-binding + (or (not desired-req) + (string=? + desired-req + (metavariable-binding-text (cdr req-binding)))) + bindings))))) + (def (js-route-method-context-bindings source match) + (let* ([full (re-match-full match)] + [start (re-match-start match)] + [app (re-match-group match 1)] + [method (re-match-group match 2)] + [app-rel (or (string-find-substring-from full app 0) 0)] + [method-rel (or (string-find-substring-from + full + method + (+ app-rel (string-length app))) + 0)] + [app-start (+ start app-rel)] + [app-end (+ app-start (string-length app))] + [method-start (+ start method-rel)] + [method-end (+ method-start (string-length method))]) + (list + (cons + "APP" + (make-regex-capture-binding "APP" app source app-start + app-end)) + (cons + "METHOD" + (make-regex-capture-binding "METHOD" method source + method-start method-end))))) + (def (js-route-arrow-callback-bindings + source + arrow-index + candidate-start + desired-req) + (let* ([body-open (skip-whitespace + source + (+ arrow-index 2))] + [body-close (and (< body-open (string-length source)) + (char=? (string-ref source body-open) #\{) + (find-matching-close-brace + source + body-open))] + [param-range (js-arrow-param-range-before + source + arrow-index)]) + (and body-close + param-range + (< body-open candidate-start) + (< candidate-start body-close) + (js-route-callback-param-bindings + source + (car param-range) + (cdr param-range) + desired-req)))) + (def (js-route-function-callback-bindings + source + function-index + candidate-start + desired-req) + (and (js-token-start-boundary? source function-index) + (js-token-end-boundary? + source + (+ function-index (string-length "function"))) + (let* ([name-start (skip-whitespace + source + (+ function-index + (string-length "function")))] + [name-end (and (< name-start (string-length source)) + (identifier-char? + (string-ref source name-start)) + (access-path-identifier-end + source + name-start))] + [param-open (and name-end + (string-find-substring-from + source + "(" + name-end))] + [param-close (and param-open + (find-matching-close-paren + source + param-open))] + [body-open (and param-close + (skip-whitespace source param-close))] + [body-close (and body-open + (< body-open (string-length source)) + (char=? + (string-ref source body-open) + #\{) + (find-matching-close-brace + source + body-open))] + [param-bindings (and param-open + param-close + (js-route-callback-param-bindings + source + (+ param-open 1) + (- param-close 1) + desired-req))] + [func-binding (and name-end + (make-regex-capture-binding "FUNC" + (substring + source + name-start + name-end) + source name-start name-end))]) + (and body-close + param-bindings + (< body-open candidate-start) + (< candidate-start body-close) + (if func-binding + (cons (cons "FUNC" func-binding) param-bindings) + param-bindings))))) + (def (js-route-callback-bindings-in-call source call-open call-close candidate-start desired-req) + (or (let loop ([offset call-open]) + (let ([arrow-index (string-find-substring-from + source + "=>" + offset)]) + (and arrow-index + (< arrow-index candidate-start) + (< arrow-index call-close) + (or (js-route-arrow-callback-bindings + source + arrow-index + candidate-start + desired-req) + (loop (+ arrow-index 2)))))) + (let loop ([offset call-open]) + (let ([function-index (string-find-substring-from + source + "function" + offset)]) + (and function-index + (< function-index candidate-start) + (< function-index call-close) + (or (js-route-function-callback-bindings + source + function-index + candidate-start + desired-req) + (loop (+ function-index 1)))))))) + (def (js-route-callback-inside-bindings source candidate) + (let* ([existing (finding-metavariable-binding + candidate + "$REQ")] + [desired-req (and existing + (metavariable-binding-text existing))] + [candidate-start (finding-start-offset candidate)] + [route-rx (re "([A-Za-z_$][A-Za-z0-9_$]*)\\.([A-Za-z_$][A-Za-z0-9_$]*)[ \t\n]*\\(")]) + (let loop ([offset 0]) + (let ([match (re-search route-rx source offset)]) + (and match + (< (re-match-start match) candidate-start) + (let* ([full (re-match-full match)] + [call-open (- (+ (re-match-start match) + (string-length full)) + 1)] + [call-close (find-matching-close-paren + source + call-open)] + [method-bindings (js-route-method-context-bindings + source + match)] + [param-bindings (and call-close + (< candidate-start + call-close) + (js-route-callback-bindings-in-call source call-open call-close + candidate-start + desired-req))]) + (or (and param-bindings + (append method-bindings param-bindings)) + (loop + (max (+ (re-match-start match) 1) + (re-match-end match)))))))))) + (def (js-route-callback-inside-apply + rule + candidate + entry + source) + (and (js-route-callback-inside-entry? entry) + (let ([bindings (js-route-callback-inside-bindings + source + candidate)]) + (and bindings + (let ([merged (merge-binding-list + (finding-metavars candidate) + bindings)]) + (and merged + (finding-with-bindings + rule + candidate + merged + source))))))) (def (js-member-ellipsis-plus-pattern-spec pattern) (let* ([trimmed (string-trim pattern)] [match (re-search @@ -5440,6 +5702,136 @@ source pattern) (scan-javascript-member-ellipsis-plus-pattern-with-bindings rule path source pattern '())) + (def (javascript-template-sql-expression-pattern? pattern) + (and (string-find-substring pattern "`") + (string-find-substring pattern "$SQLSTR") + (string-find-substring pattern "${$EXPR}") + (string-find-substring pattern "..."))) + (def (javascript-template-expression-close + source + open-index + limit) + (let loop ([i open-index] + [depth 0] + [state 'normal] + [escaped? #f]) + (cond + [(>= i limit) #f] + [(eq? state 'normal) + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\") (loop (+ i 1) depth 'double #f)] + [(char=? ch #\') (loop (+ i 1) depth 'single #f)] + [(char=? ch #\`) (loop (+ i 1) depth 'backtick #f)] + [(char=? ch #\{) (loop (+ i 1) (+ depth 1) state #f)] + [(char=? ch #\}) + (if (= depth 0) i (loop (+ i 1) (- depth 1) state #f))] + [else (loop (+ i 1) depth state #f)]))] + [escaped? (loop (+ i 1) depth state #f)] + [(char=? (string-ref source i) #\\) + (loop (+ i 1) depth state #t)] + [(and (eq? state 'double) + (char=? (string-ref source i) #\")) + (loop (+ i 1) depth 'normal #f)] + [(and (eq? state 'single) + (char=? (string-ref source i) #\')) + (loop (+ i 1) depth 'normal #f)] + [(and (eq? state 'backtick) + (char=? (string-ref source i) #\`)) + (loop (+ i 1) depth 'normal #f)] + [else (loop (+ i 1) depth state #f)]))) + (def (trim-range-horizontal source start end) + (let ([left (let loop ([i start]) + (if (and (< i end) + (let ([ch (string-ref source i)]) + (or (char=? ch #\space) + (char=? ch #\tab) + (char=? ch #\newline)))) + (loop (+ i 1)) + i))] + [right (let loop ([i end]) + (if (and (> i start) + (let ([ch (string-ref source (- i 1))]) + (or (char=? ch #\space) + (char=? ch #\tab) + (char=? ch #\newline)))) + (loop (- i 1)) + i))]) + (cons left right))) + (def (javascript-template-sql-expression-finding rule path source template-start template-end expr-open + expr-close initial-bindings) + (let* ([sql-start (+ template-start 1)] + [sql-end expr-open] + [expr-range (trim-range-horizontal + source + (+ expr-open 2) + expr-close)] + [expr-start (car expr-range)] + [expr-end (cdr expr-range)] + [sql-binding (make-regex-capture-binding "SQLSTR" (substring source sql-start sql-end) + source sql-start sql-end)] + [expr-binding (and (< expr-start expr-end) + (make-regex-capture-binding "EXPR" + (substring source expr-start expr-end) + source expr-start expr-end))] + [merged (and expr-binding + (merge-binding-list + initial-bindings + (list + (cons "SQLSTR" sql-binding) + (cons "EXPR" expr-binding))))]) + (and merged + (finding-for-range-with-bindings rule path source + template-start template-end merged)))) + (def (scan-javascript-template-sql-expression-pattern-with-bindings rule path source pattern initial-bindings) + (and (javascript-template-sql-expression-pattern? pattern) + (let ([len (string-length source)]) + (let loop ([offset 0] [acc '()]) + (if (>= offset len) + (nonempty-findings (reverse acc)) + (let ([template-start (string-find-substring-from + source + "`" + offset)]) + (if template-start + (let* ([template-end (scan-js-string-literal-end + source + template-start + #\`)] + [expr-open (and template-end + (string-find-substring-from + source + "${" + (+ template-start 1)))] + [expr-close (and expr-open + (< expr-open + template-end) + (javascript-template-expression-close + source + (+ expr-open 2) + (- template-end 1)))] + [finding (and template-end + expr-open + expr-close + (< expr-close template-end) + (javascript-template-sql-expression-finding rule path source + template-start + template-end expr-open + expr-close + initial-bindings))] + [next (if template-end + template-end + (+ template-start 1))]) + (loop + next + (if finding (cons finding acc) acc))) + (nonempty-findings (reverse acc))))))))) + (def (scan-javascript-template-sql-expression-pattern + rule + path + source + pattern) + (scan-javascript-template-sql-expression-pattern-with-bindings rule path source pattern '())) (def (js-arrow-object-param-pattern-spec pattern) (let* ([trimmed (string-trim pattern)] [match (re-search @@ -7926,6 +8318,12 @@ (cdr clause) source)) (and (javascript-like-language? language) + (js-route-callback-inside-apply + rule + candidate + (cdr clause) + source)) + (and (javascript-like-language? language) (js-function-param-inside-apply rule candidate --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "B020B5A7F3281D03") + ("src/semgrep/scan.ss" . "B22CC16B489E44F2") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "D56FC2D2EB449BA6")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -267,11 +267,22 @@ [else (loop (+ i 1) (cons (string-ref pattern i) acc))])))) (def (regex-pattern-for-engine pattern) - (if (regex-leading-case-insensitive? pattern) - (regex-pattern-for-engine-raw - (regex-expand-case-insensitive - (substring pattern 4 (string-length pattern)))) - (regex-pattern-for-engine-raw pattern))) + (cond + [(regex-leading-case-insensitive? pattern) + (regex-pattern-for-engine-raw + (regex-expand-case-insensitive + (substring pattern 4 (string-length pattern))))] + [(string-find-substring pattern "(?i)") + => (lambda (index) + (string-append + (regex-pattern-for-engine-raw + (substring pattern 0 index)) + (regex-pattern-for-engine-raw + (regex-expand-case-insensitive + (substring pattern + (+ index 4) + (string-length pattern))))))] + [else (regex-pattern-for-engine-raw pattern)])) (def (substring-at? s needle i) (let ([needle-len (string-length needle)]) @@ -4410,6 +4421,12 @@ source (cdr entry))) (and (javascript-like-language? language) + (scan-javascript-template-sql-expression-pattern + rule + path + source + (cdr entry))) + (and (javascript-like-language? language) (scan-javascript-new-function-sequence-pattern rule path @@ -4659,6 +4676,13 @@ (cdr entry) initial-bindings)) (and (javascript-like-language? language) + (scan-javascript-template-sql-expression-pattern-with-bindings + rule + path + source + (cdr entry) + initial-bindings)) + (and (javascript-like-language? language) (scan-javascript-new-function-sequence-pattern rule path @@ -5664,6 +5688,248 @@ merged source)))))))) +(def (js-route-callback-inside-entry? entry) + (and (eq? (car entry) 'pattern) + (let ([pattern (cdr entry)]) + (and (string-find-substring pattern "$APP.$METHOD") + (string-find-substring pattern "function $FUNC") + (string-find-substring pattern "$REQ") + (string-find-substring pattern "$RES") + (string-find-substring pattern "{...}"))))) + +(def (js-simple-param-bindings-in-range source start end names) + (let loop ([i start] [remaining names] [acc '()]) + (cond + [(null? remaining) (reverse acc)] + [else + (let ([j (js-skip-horizontal-and-newline source i end)]) + (cond + [(>= j end) #f] + [(char=? (string-ref source j) #\,) + (loop (+ j 1) remaining acc)] + [(identifier-char? (string-ref source j)) + (let* ([name-end (access-path-identifier-end source j)] + [binding + (make-regex-capture-binding + (car remaining) + (substring source j name-end) + source + j + name-end)] + [next (js-find-next-comma-or-end source name-end end)]) + (loop next + (cdr remaining) + (cons (cons (car remaining) binding) acc)))] + [else #f]))]))) + +(def (js-route-callback-param-bindings + source + param-start + param-end + desired-req) + (let ([bindings + (js-simple-param-bindings-in-range + source + param-start + param-end + '("REQ" "RES"))]) + (and bindings + (let ([req-binding (assoc "REQ" bindings)]) + (and req-binding + (or (not desired-req) + (string=? desired-req + (metavariable-binding-text + (cdr req-binding)))) + bindings))))) + +(def (js-route-method-context-bindings source match) + (let* ([full (re-match-full match)] + [start (re-match-start match)] + [app (re-match-group match 1)]