Clear decorated TypeScript and Unicode frontiers
ober
5ddcbd32fd01d3f81aa984f3c8302975a39adf6f
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,18 +1,19 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 02:51 MDT +Date: 2026-05-29 03:15 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Current pre-handoff HEAD: `9ed0adb Clear React prop spread taint frontier` +Base HEAD before this checkpoint: `113620a Honor taint metavariable unification` The user wants this project carried forward until the pure Jerboa port reaches -Semgrep parity. Continue from this state. Do not restart broad discovery from -scratch; use the current frontier and commands below as the working map. +Semgrep parity. Continue from the current frontier below. Do not restart broad +discovery from scratch; the oracle commands and case list here are the working +map. ## Project Goal -The intent from `what.md` remains the governing target: +The target from `what.md` is still the contract: - Replace Semgrep's OCaml and Python implementation with ordinary Jerboa `.ss` modules. @@ -22,7 +23,8 @@ The intent from `what.md` remains the governing target: This is a compatibility-preserving Semgrep implementation in Jerboa, not a similar scanner. High-risk parity areas remain structural matching, -taint/dataflow, path and target semantics, autofix, and output schemas. +taint/dataflow, path and target semantics, autofix, output schemas, and full +language parser coverage. ## Files In This Checkpoint @@ -37,7 +39,8 @@ tests/smoke.ss ``` `lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by -`make test`; they are tracked and must be committed with source changes. +`make test`/`make oracle`; they are tracked and must be committed with source +changes. ## Verified State @@ -52,7 +55,7 @@ make test Result: ```text -208 tests, 208 passed, 0 failed +211 tests, 211 passed, 0 failed ``` Local oracle: @@ -67,32 +70,18 @@ Result: oracle: 42 passed, 0 failed ``` -Focused `taint_unify_mvars` case: +Focused decorated/Unicode guardrail: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^taint_unify_mvars$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(ts_decorated_async_property|ts_decorated_static_property|unicode_string)$' LIST_MISMATCHES=1 MAX_DIFFS=180 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared +upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` -Nearby taint guardrail: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_unify_mvars|taint_spread_record_big|taint_safe_comparisons|taint_react)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh -``` - -Result: - -```text -upstream-sweep: 4 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 4 compared -``` - -The next mismatch is outside this taint guardrail: `ts_decorated_async_property`. - Full upstream sweep: ```sh @@ -102,46 +91,70 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 231 passed, 8 mismatched, 0 jerboa errors, 2 current errors, 241 compared +upstream-sweep: 234 passed, 5 mismatched, 0 jerboa errors, 2 current errors, 241 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner -errors. Use `MAX_CASES=220` when a shorter first-window guardrail is enough; -the command above intentionally ran the full sorted 241-case sweep. +errors. The full sweep is slow and can run for several minutes because it +invokes upstream Semgrep for each sorted case. ## What Changed In This Checkpoint -This checkpoint clears upstream `taint_unify_mvars`, reducing the full sweep -frontier from: +This checkpoint clears three upstream mismatches: ```text -230 passed, 9 mismatched, 0 jerboa errors, 2 current errors +ts_decorated_async_property +ts_decorated_static_property +unicode_string ``` -to: +The full sweep moved from the previous checkpoint: ```text 231 passed, 8 mismatched, 0 jerboa errors, 2 current errors ``` +to: + +```text +234 passed, 5 mismatched, 0 jerboa errors, 2 current errors +``` + Implementation changes in `src/semgrep/scan.ss`: -- Added `taint-unify-mvars?` for the Semgrep rule option - `taint_unify_mvars`. -- Added `taint-mvars-compatible?`, which requires source and sink findings to - have compatible shared metavariable bindings when that option is enabled. -- Applied the compatibility check inside `source-state-reaches-sink-spec-shape?` - before normal taint reachability checks. +- Added a TypeScript decorated class-member scanner for patterns of the form: + `@Decorator(...) async name`, `@Decorator(...) async name(...)`, + `@Decorator(...) async name(...){ ... }`, and the matching `static` forms. +- The scanner parses the decorator token, drops pattern decorator call suffixes + such as `@Query(...)`, records the required modifier (`async` or `static`), + target member name, and shape (`property`, `method`, or `method-body`). +- Target matching accepts `static async` for both async and static method + patterns where Semgrep does. For async patterns, `static` is optional before + `async`; for static patterns, `async` is optional after `static`. +- Property-shaped patterns such as `async foo` and `static foo` deliberately do + not match method declarations such as `async foo(){}` or `static foo(){}`. +- Method-shaped patterns end at the parameter close paren. Method-body-shaped + patterns extend through the matching method body close brace. +- The scanner is wired into the positive pattern paths used by direct pattern + rules, formula entries, and binding-aware positive entries. +- `offset->line-col` now counts UTF-8 byte width for columns, matching Semgrep + JSON positions for non-ASCII source. Source offsets already used Semgrep byte + offsets for tree-sitter-backed findings; the mismatch was column width. New smoke coverage in `tests/smoke.ss`: ```text -scan JavaScript taint unifies source and sink metavariables +scan Python unicode string byte columns +scan TypeScript decorated async methods +scan TypeScript decorated static methods ``` +The Unicode smoke uses ASCII Scheme `\x1f385;` escapes so the test file itself +does not need literal non-ASCII characters. + ## Recently Cleared Cases -Clean in the recent taint checkpoints: +Clean in recent commits: ```text taint_param_source1 @@ -152,54 +165,41 @@ taint_react taint_safe_comparisons taint_spread_record_big taint_unify_mvars +ts_decorated_async_property +ts_decorated_static_property +unicode_string ``` -Relevant upstream files: +Useful upstream files for the latest cleared cases: ```text -/Users/user/mine/semgrep/tests/rules/taint_param_source1.yaml -/Users/user/mine/semgrep/tests/rules/taint_param_source1.js -/Users/user/mine/semgrep/tests/rules/taint_param_source2.yaml -/Users/user/mine/semgrep/tests/rules/taint_param_source2.ts -/Users/user/mine/semgrep/tests/rules/taint_param_source3.yaml -/Users/user/mine/semgrep/tests/rules/taint_param_source3.js -/Users/user/mine/semgrep/tests/rules/taint_propagator2.yaml -/Users/user/mine/semgrep/tests/rules/taint_propagator2.py -/Users/user/mine/semgrep/tests/rules/taint_react.yaml -/Users/user/mine/semgrep/tests/rules/taint_react.js -/Users/user/mine/semgrep/tests/rules/taint_safe_comparisons.yaml -/Users/user/mine/semgrep/tests/rules/taint_safe_comparisons.py -/Users/user/mine/semgrep/tests/rules/taint_spread_record_big.yaml -/Users/user/mine/semgrep/tests/rules/taint_spread_record_big.js -/Users/user/mine/semgrep/tests/rules/taint_unify_mvars.yaml -/Users/user/mine/semgrep/tests/rules/taint_unify_mvars.js +/Users/user/mine/semgrep/tests/rules/ts_decorated_async_property.yaml +/Users/user/mine/semgrep/tests/rules/ts_decorated_async_property.ts +/Users/user/mine/semgrep/tests/rules/ts_decorated_static_property.yaml +/Users/user/mine/semgrep/tests/rules/ts_decorated_static_property.ts +/Users/user/mine/semgrep/tests/rules/unicode_string.yaml +/Users/user/mine/semgrep/tests/rules/unicode_string.py ``` -Important isolation finding for `taint_unify_mvars`: - -- Source findings already carried `$FUNC` from - `pattern-inside: function $FUNC(...) { ... }`: - - ```text - source() in foo has FUNC=foo - source() in bar has FUNC=bar - ``` +Important observations: -- Sink findings already carried `$FUNC` and `$TAINT` from - `pattern-inside: $FUNC($TAINT)`. -- The missing behavior was honoring `taint_unify_mvars: true` during source to - sink reachability. Without that, `source()` inside `foo` could reach - `bar(x)` and vice versa. +- `ts_decorated_async_property` expected four findings at the decorator lines + for `async bar`, `static async bar`, `async qux`, and `static async qux`. +- `ts_decorated_static_property` expected the analogous four findings for + `static bar`, `static async bar`, `static qux`, and `static async qux`. +- In both decorated cases, the `foo` property-shaped pattern did not match the + `foo(){}` methods. Preserve this distinction. +- `unicode_string` had identical byte offsets before the fix: + `start offset 69`, `end offset 75`. Only the end column differed: Semgrep + reported `7`; Jerboa reported `4` because it counted the emoji as one + character instead of four UTF-8 bytes. ## Current Full-Sweep Frontier The current full sorted upstream sweep compared 241 rule/target pairs and has -these 8 mismatches: +these 5 mismatches: ```text -ts_decorated_async_property -ts_decorated_static_property -unicode_string vardef_assign_false vardef_assign_false1 vardef_assign_true @@ -207,84 +207,72 @@ vardef_assign_true1 vardef_assign_true2 ``` -Use this command to refresh the full list: +Refresh the frontier with: ```sh SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -## Recommended Next Target: ts_decorated_async_property +## Recommended Next Target: vardef_assign_false -The next sorted frontier is `ts_decorated_async_property`. - -Focused command: +The next sorted frontier is `vardef_assign_false`. Start with the whole family +because these cases almost certainly share one vardef/assignment semantic gap: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^ts_decorated_async_property$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^vardef_assign_(false|false1|true|true1|true2)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh ``` -Current result: +Current focused result: ```text -MISMATCH ts_decorated_async_property -@@ -1,4 +0,0 @@ --(finding "decorated-async-property" ".../ts_decorated_async_property.ts" 12 3 122 13 14 144 "ERROR" "Matching a decorated async property" "") --(finding "decorated-async-property" ".../ts_decorated_async_property.ts" 17 3 191 18 21 220 "ERROR" "Matching a decorated async property" "") --(finding "decorated-async-property" ".../ts_decorated_async_property.ts" 22 3 267 24 4 294 "ERROR" "Matching a decorated async property" "") --(finding "decorated-async-property" ".../ts_decorated_async_property.ts" 27 3 336 29 4 370 "ERROR" "Matching a decorated async property" "") -upstream-sweep: 0 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 1 compared -``` - -Rule shape: - -```yaml -rules: -- id: decorated-async-property - languages: [typescript] - pattern-either: - - pattern: | - @Query(...) - async foo - - pattern: | - @Query(...) - async bar(...) - - pattern: | - @Query(...) - async qux(...){ - ... - } +MISMATCH vardef_assign_false +@@ -0,0 +1 @@ ++(finding "test" ".../vardef_assign_false.js" 2 1 10 2 10 19 "WARNING" "Match Found!" "") +MISMATCH vardef_assign_false1 +@@ -0,0 +1 @@ ++(finding "test" ".../vardef_assign_false1.js" 1 1 0 3 1 21 "WARNING" "Match Found!" "") +MISMATCH vardef_assign_true +@@ -1 +1 @@ +-(finding "test" ".../vardef_assign_true.js" 2 5 18 2 10 23 "WARNING" "Match Found!" "") ++(finding "test" ".../vardef_assign_true.js" 2 1 14 2 10 23 "WARNING" "Match Found!" "") +MISMATCH vardef_assign_true1 +@@ -1 +1 @@ +-(finding "test" ".../vardef_assign_true1.js" 2 1 14 2 10 23 "WARNING" "Match Found!" "") ++(finding "test" ".../vardef_assign_true1.js" 1 1 0 3 1 25 "WARNING" "Match Found!" "") +MISMATCH vardef_assign_true2 +@@ -1 +0,0 @@ +-(finding "test" ".../vardef_assign_true2.js" 2 5 18 2 10 23 "WARNING" "Match Found!" "") +upstream-sweep: 0 passed, 5 mismatched, 0 jerboa errors, 0 current errors, 5 compared ``` -Target: - -```typescript -class Foo { - @Query() - async bar(){ - } - - @Query() - static async bar(){ - } +Then inspect the upstream fixtures: - @Query() - async qux(){ - } - - @Query() - static async qux(){ - } -} +```sh +sed -n '1,160p' /Users/user/mine/semgrep/tests/rules/vardef_assign_false.yaml +sed -n '1,160p' /Users/user/mine/semgrep/tests/rules/vardef_assign_false1.yaml +sed -n '1,160p' /Users/user/mine/semgrep/tests/rules/vardef_assign_true.yaml +sed -n '1,160p' /Users/user/mine/semgrep/tests/rules/vardef_assign_true1.yaml +sed -n '1,160p' /Users/user/mine/semgrep/tests/rules/vardef_assign_true2.yaml ``` Likely fix area: -- This is a false-negative TypeScript structural matching gap for decorated - async class methods/properties. -- Existing smoke has `scan TypeScript decorated class fields`; start by - comparing its pattern support with these `@Query()` async method patterns. -- `ts_decorated_static_property` is adjacent and likely shares the same matcher - surface, so verify both after any patch. +- The rules all scan `var x = 1;` with `pattern: x = 1`, a statement-sequence + pattern ending in `x = 1;`, or expression-ellipsis pattern + `<... x = 1 ...>`. +- `options: { vardef_assign: false }` should suppress matches where `x = 1` + comes only from the initializer in `var x = 1;`. +- `options: { vardef_assign: true }` should allow the initializer to match, + but the expected range differs by pattern shape: plain `x = 1` and + `<... x = 1 ...>` report the `x = 1` slice starting at column 5, while the + multiline `...; x = 1;` pattern reports the whole `var x = 1;` statement. +- `rule-option-enabled?` and `rule-options` already exist in + `src/semgrep/scan.ss`; the missing piece is likely matcher behavior around + JavaScript `variable_declaration`/`variable_declarator` nodes, range + selection, and expression-ellipsis support for assignment-equivalent + vardefs. +- Add a focused smoke test that covers the smallest upstream shape before + changing shared structural matching. ## Useful Commands @@ -306,7 +294,7 @@ Full upstream guardrail: SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh ``` -Local tests: +Local gates: ```sh make test @@ -330,20 +318,17 @@ SCM ## Development Notes - Prefer narrow smoke tests that reproduce an upstream shape before changing - shared taint logic. -- `scan-symbolic-propagation-pattern` declines broad structural patterns such - as bare metavariables and string literal ellipses. Recheck this if symbolic - rules unexpectedly stop matching. -- `focus-metavariable` preserves finding extras by default. Current Semgrep - omits `fix-regex` under focus but keeps template `fix:`. + shared matching or taint logic. +- When `make test` rebuilds `lib/semgrep/scan.sls`, commit that generated file + and `src/.jerbuild-hashes` with the source change. - JavaScript function scope is deliberately conservative: only top-level `function ...` lines are recognized as simple scopes. This avoids sibling leaks without breaking nested callback taint cases already covered by smoke tests. -- After touching TypeScript decorated async matching, run at minimum: - - ```sh - make test - SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(ts_decorated_async_property|ts_decorated_static_property|taint_unify_mvars)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh - SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle - ``` +- `focus-metavariable` preserves finding extras by default. Current Semgrep + omits `fix-regex` under focus but keeps template `fix:`. +- Regex-only rules can run without an AST parser. Be careful not to introduce + parser requirements for `generic`/`regex` `pattern-regex` formulas. +- TypeScript and TSX currently use the JavaScript parser fallback for + JS-compatible syntax. The decorated-member scanner is a targeted compatibility + fallback, not full TypeScript AST parity. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -84,7 +84,13 @@ [(or (= i offset) (= i len)) (values line col)] [(char=? (string-ref source i) #\newline) (loop (+ i 1) (+ line 1) 1)] - [else (loop (+ i 1) line (+ col 1))])))) + [else + (loop + (+ i 1) + line + (+ col + (semgrep-source-byte-length + (string-ref source i))))])))) (def (char-utf8-byte-length ch) (let ([codepoint (char->integer ch)]) (cond @@ -1087,6 +1093,191 @@ next (if finding (cons finding acc) acc))) (reverse acc))))))))) + (def (drop-decorator-call-suffix text) + (let ([open (string-find-substring text "(")]) + (if open (substring text 0 open) text))) + (def (first-line-end text) + (or (string-find-substring text "\n") (string-length text))) + (def (decorated-async-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)]) + (and (> len 0) + (char=? (string-ref trimmed 0) #\@) + (let* ([line-end (first-line-end trimmed)] + [decorator (drop-decorator-call-suffix + (string-trim + (substring trimmed 1 line-end)))] + [tail (string-trim (substring trimmed line-end len))] + [modifier (cond + [(sg-string-prefix? "async " tail) 'async] + [(sg-string-prefix? "static " tail) + 'static] + [else #f])] + [name-start (case modifier + [(async) (string-length "async ")] + [(static) (string-length "static ")] + [else #f])] + [name-range (and name-start + (jsx-name-range-at tail name-start))] + [name (and name-range + (substring + tail + (car name-range) + (cdr name-range)))] + [after-name (and name-range + (skip-whitespace + tail + (cdr name-range)))] + [method? (and after-name + (< after-name (string-length tail)) + (char=? + (string-ref tail after-name) + #\())] + [body? (and (string-find-substring tail "{") #t)]) + (and (> (string-length decorator) 0) + name + (list + (cons 'decorator decorator) + (cons 'modifier modifier) + (cons 'name name) + (cons + 'kind + (cond + [body? 'method-body] + [method? 'method] + [else 'property])))))))) + (def (decorator-token-boundary? source index) + (or (>= index (string-length source)) + (not (identifier-char? (string-ref source index))))) + (def (decorated-modifier-at? source index modifier) + (let ([end (+ index (string-length modifier))]) + (and (substring-at? source modifier index) + (or (>= end (string-length source)) + (not (identifier-char? (string-ref source end))))))) + (def (decorated-modifier-end source index modifier) + (and (decorated-modifier-at? source index modifier) + (skip-whitespace + source + (+ index (string-length modifier))))) + (def (decorated-async-target-at source start spec) + (let* ([decorator (alist-ref/default spec 'decorator "")] + [decorator-end (+ start 1 (string-length decorator))] + [kind (alist-ref/default spec 'kind #f)] + [modifier (alist-ref/default spec 'modifier #f)] + [target-name (alist-ref/default spec 'name "")] + [after-decorator0 (skip-whitespace source decorator-end)] + [after-decorator (and (decorator-token-boundary? + source + decorator-end) + (if (and (< after-decorator0 + (string-length source)) + (char=? + (string-ref + source + after-decorator0) + #\()) + (find-matching-close-paren + source + after-decorator0) + after-decorator0))] + [line-end (and after-decorator + (line-end-after source after-decorator))] + [decl-start (and line-end + (< line-end (string-length source)) + (skip-whitespace source (+ line-end 1)))] + [after-static (and decl-start + (decorated-modifier-end + source + decl-start + "static"))] + [after-optional-static (or after-static decl-start)] + [after-async (and after-optional-static + (decorated-modifier-end + source + after-optional-static + "async"))] + [after-static-optional-async (and after-static + (or (decorated-modifier-end + source + after-static + "async") + after-static))] + [after-modifiers (case modifier + [(async) after-async] + [(static) after-static-optional-async] + [else #f])] + [name-end (and after-modifiers + (< after-modifiers (string-length source)) + (identifier-start-char? + (string-ref source after-modifiers)) + (access-path-identifier-end + source + after-modifiers))] + [name (and name-end + (substring source after-modifiers name-end))] + [after-name (and name-end + (skip-whitespace source name-end))] + [method-open? (and after-name + (< after-name (string-length source)) + (char=? + (string-ref source after-name) + #\())] + [param-close (and method-open? + (find-matching-close-paren + source + after-name))] + [body-open (and param-close + (skip-whitespace source param-close))] + [body-close (and body-open + (< body-open (string-length source)) + (char=? (string-ref source body-open) #\{) + (find-matching-close-brace + source + body-open))] + [end (and name + (string=? name target-name) + (case kind + [(property) (and (not method-open?) name-end)] + [(method) param-close] + [(method-body) body-close] + [else #f]))]) + (and end (cons start end)))) + (def (decorated-async-finding rule path source range) + (finding-for-range-with-bindings rule path source + (car range) (cdr range) '())) + (def (scan-typescript-decorated-async-pattern + rule + path + source + pattern) + (let ([spec (decorated-async-pattern-spec pattern)]) + (and spec + (let* ([decorator (alist-ref/default spec 'decorator "")] + [needle (string-append "@" decorator)] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from + source + needle + start)]) + (if (or (not index) (> index len)) + (reverse acc) + (let* ([range (decorated-async-target-at + source + index + spec)] + [finding (and range + (decorated-async-finding + rule + path + source + range))] + [next (if range + (max (+ index 1) (cdr range)) + (+ index 1))]) + (loop + next + (if finding (cons finding acc) acc)))))))))) (def (json-key-value-pattern-spec pattern) (let* ([match (re-search (re "\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*:[ \\t]*([^,}\\n]+)") @@ -4686,6 +4877,12 @@ path source (cdr entry))) + (and (typescript-like-language? language) + (scan-typescript-decorated-async-pattern + rule + path + source + (cdr entry))) (scan-string-key-object-pattern rule path @@ -4926,6 +5123,12 @@ path source (cdr entry))) + (and (typescript-like-language? language) + (scan-typescript-decorated-async-pattern + rule + path + source + (cdr entry))) (and (string=? language "json") (scan-json-key-value-pattern rule @@ -11489,6 +11692,12 @@ path source (rule-pattern rule))) + (and (typescript-like-language? language) + (scan-typescript-decorated-async-pattern + rule + path + source + (rule-pattern rule))) (and (string=? language "json") (scan-json-key-value-pattern rule --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "6240D2B852D655ED") + ("src/semgrep/scan.ss" . "E3D9FE6B8D7E1CE8") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "D56FC2D2EB449BA6")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -97,7 +97,11 @@ [(char=? (string-ref source i) #\newline) (loop (+ i 1) (+ line 1) 1)] [else - (loop (+ i 1) line (+ col 1))])))) + (loop (+ i 1) + line + (+ col + (semgrep-source-byte-length + (string-ref source i))))])))) (def (char-utf8-byte-length ch) (let ([codepoint (char->integer ch)]) @@ -1169,6 +1173,179 @@ acc))) (reverse acc))))))))) +(def (drop-decorator-call-suffix text) + (let ([open (string-find-substring text "(")]) + (if open (substring text 0 open) text))) + +(def (first-line-end text) + (or (string-find-substring text "\n") + (string-length text))) + +(def (decorated-async-pattern-spec pattern) + (let* ([trimmed (string-trim pattern)] + [len (string-length trimmed)]) + (and (> len 0) + (char=? (string-ref trimmed 0) #\@) + (let* ([line-end (first-line-end trimmed)] + [decorator + (drop-decorator-call-suffix + (string-trim (substring trimmed 1 line-end)))] + [tail (string-trim + (substring trimmed line-end len))] + [modifier + (cond + [(sg-string-prefix? "async " tail) 'async] + [(sg-string-prefix? "static " tail) 'static] + [else #f])] + [name-start + (case modifier + [(async) (string-length "async ")] + [(static) (string-length "static ")] + [else #f])] + [name-range + (and name-start + (jsx-name-range-at tail name-start))] + [name (and name-range + (substring tail + (car name-range) + (cdr name-range)))] + [after-name + (and name-range + (skip-whitespace tail (cdr name-range)))] + [method? (and after-name + (< after-name (string-length tail)) + (char=? (string-ref tail after-name) #\())] + [body? (and (string-find-substring tail "{") #t)]) + (and (> (string-length decorator) 0) + name + (list (cons 'decorator decorator) + (cons 'modifier modifier) + (cons 'name name) + (cons 'kind + (cond + [body? 'method-body] + [method? 'method] + [else 'property])))))))) + +(def (decorator-token-boundary? source index) + (or (>= index (string-length source)) + (not (identifier-char? (string-ref source index))))) + +(def (decorated-modifier-at? source index modifier) + (let ([end (+ index (string-length modifier))]) + (and (substring-at? source modifier index) + (or (>= end (string-length source)) + (not (identifier-char? (string-ref source end))))))) + +(def (decorated-modifier-end source index modifier) + (and (decorated-modifier-at? source index modifier) + (skip-whitespace source (+ index (string-length modifier))))) + +(def (decorated-async-target-at source start spec) + (let* ([decorator (alist-ref/default spec 'decorator "")] + [decorator-end (+ start 1 (string-length decorator))] + [kind (alist-ref/default spec 'kind #f)] + [modifier (alist-ref/default spec 'modifier #f)] + [target-name (alist-ref/default spec 'name "")] + [after-decorator0 (skip-whitespace source decorator-end)] + [after-decorator + (and (decorator-token-boundary? source decorator-end) + (if (and (< after-decorator0 (string-length source)) + (char=? (string-ref source after-decorator0) #\()) + (find-matching-close-paren source after-decorator0) + after-decorator0))] + [line-end (and after-decorator + (line-end-after source after-decorator))] + [decl-start + (and line-end + (< line-end (string-length source)) + (skip-whitespace source (+ line-end 1)))] + [after-static + (and decl-start + (decorated-modifier-end source decl-start "static"))] + [after-optional-static + (or after-static decl-start)] + [after-async + (and after-optional-static + (decorated-modifier-end + source + after-optional-static + "async"))] + [after-static-optional-async + (and after-static + (or (decorated-modifier-end source after-static "async") + after-static))] + [after-modifiers + (case modifier + [(async) after-async] + [(static) after-static-optional-async] + [else #f])] + [name-end + (and after-modifiers + (< after-modifiers (string-length source)) + (identifier-start-char? (string-ref source after-modifiers)) + (access-path-identifier-end source after-modifiers))] + [name + (and name-end + (substring source after-modifiers name-end))] + [after-name (and name-end (skip-whitespace source name-end))] + [method-open? + (and after-name + (< after-name (string-length source)) + (char=? (string-ref source after-name) #\())] + [param-close (and method-open? + (find-matching-close-paren source after-name))] + [body-open (and param-close + (skip-whitespace source param-close))] + [body-close + (and body-open + (< body-open (string-length source)) + (char=? (string-ref source body-open) #\{) + (find-matching-close-brace source body-open))] + [end + (and name + (string=? name target-name) + (case kind + [(property) (and (not method-open?) name-end)] + [(method) param-close] + [(method-body) body-close] + [else #f]))]) + (and end (cons start end)))) + +(def (decorated-async-finding rule path source range) + (finding-for-range-with-bindings + rule + path + source + (car range) + (cdr range) + '())) + +(def (scan-typescript-decorated-async-pattern rule path source pattern) + (let ([spec (decorated-async-pattern-spec pattern)]) + (and spec + (let* ([decorator (alist-ref/default spec 'decorator "")] + [needle (string-append "@" decorator)] + [len (string-length source)]) + (let loop ([start 0] [acc '()]) + (let ([index (string-find-substring-from source needle start)]) + (if (or (not index) (> index len)) + (reverse acc) + (let* ([range + (decorated-async-target-at source index spec)] + [finding + (and range + (decorated-async-finding + rule + path + source + range))] + [next (if range + (max (+ index 1) (cdr range)) + (+ index 1))]) + (loop next + (if finding (cons finding acc) acc)))))))))) + (def (json-key-value-pattern-spec pattern) (let* ([match (re-search (re "\\$([A-Za-z_][A-Za-z0-9_]*)[ \\t]*:[ \\t]*([^,}\\n]+)") @@ -4877,6 +5054,12 @@ path source (cdr entry))) + (and (typescript-like-language? language) + (scan-typescript-decorated-async-pattern + rule + path + source + (cdr entry))) (scan-string-key-object-pattern rule path @@ -5168,6 +5351,12 @@ path source (cdr entry))) + (and (typescript-like-language? language) + (scan-typescript-decorated-async-pattern + rule + path + source + (cdr entry))) (and (string=? language "json") (scan-json-key-value-pattern rule @@ -12151,6 +12340,12 @@ path source (rule-pattern rule))) + (and (typescript-like-language? language) + (scan-typescript-decorated-async-pattern + rule + path + source + (rule-pattern rule))) (and (string=? language "json") (scan-json-key-value-pattern rule --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -78,6 +78,22 @@ (check (finding-start-col finding) => 5) (check (finding-end-col finding) => 10))) +(test-case "scan Python unicode string byte columns" + (let* ([unicode-config + "rules:\n - id: demo.unicode\n languages: [python]\n message: unicode string\n severity: ERROR\n pattern: '\"\x1f385;\"'\n"] + [findings + (scan-config-string unicode-config + "python" + "demo.py" + "\"\x1f385;\"\n\"ZZZZ\"\n")] + [finding (car findings)]) + (check (length findings) => 1) + (check (finding-start-line finding) => 1) + (check (finding-start-col finding) => 1) + (check (finding-end-col finding) => 7) + (check (finding-start-offset finding) => 0) + (check (finding-end-offset finding) => 6))) + (test-case "parse structural pattern rule config" (let* ([rules (parse-config-string structural-config)] [rule (car rules)]) @@ -492,6 +508,36 @@ (check (finding-start-line (cadr findings)) => 3) (check (finding-message (car findings)) => "decorated bar"))) +(test-case "scan TypeScript decorated async methods" + (let* ([ts-config + "rules:\n - id: demo.ts.decorated.async\n languages: [typescript]\n message: decorated async\n severity: ERROR\n pattern-either:\n - pattern: |\n @Query(...)\n async foo\n - pattern: |\n @Query(...)\n async bar(...)\n - pattern: |\n @Query(...)\n async qux(...){\n ...\n }\n"] + [findings + (scan-config-string + ts-config + "typescript" + "demo.ts" + "class Foo {\n @Query()\n async foo(){\n }\n @Query()\n static async foo(){\n }\n @Query()\n async bar(){\n }\n @Query()\n static async bar(){\n }\n @Query()\n async qux(){\n }\n @Query()\n static async qux(){\n }\n}\n")]) + (check (length findings) => 4) + (check (finding-start-line (car findings)) => 8) + (check (finding-start-line (cadr findings)) => 11) + (check (finding-start-line (caddr findings)) => 14) + (check (finding-start-line (cadddr findings)) => 17))) + +(test-case "scan TypeScript decorated static methods" + (let* ([ts-config + "rules:\n - id: demo.ts.decorated.static\n languages: [typescript]\n message: decorated static\n severity: ERROR\n pattern-either:\n - pattern: |\n @Query(...)\n static foo\n - pattern: |\n @Query(...)\n static bar(...)\n - pattern: |\n @Query(...)\n static qux(...){\n ...\n }\n"] + [findings + (scan-config-string + ts-config + "typescript" + "demo.ts" + "class Foo {\n @Query()\n static foo(){\n }\n @Query()\n static async foo(){\n }\n @Query()\n static bar(){\n }\n @Query()\n static async bar(){\n }\n @Query()\n static qux(){\n }\n @Query()\n static async qux(){\n }\n}\n")]) + (check (length findings) => 4) + (check (finding-start-line (car findings)) => 8) + (check (finding-start-line (cadr findings)) => 11) + (check (finding-start-line (caddr findings)) => 14) + (check (finding-start-line (cadddr findings)) => 17))) + (test-case "scan JavaScript metavariable-name imported entities" (let* ([js-config "rules:\n - id: demo.metavar.name.js\n languages: [javascript]\n message: imported name\n severity: WARNING\n pattern-either:\n - patterns:\n - pattern: $X(...)\n - semgrep-internal-metavariable-name:\n metavariable: $X\n fqn: foo3.bar3\n - patterns:\n - pattern: $X.bar(...)\n - semgrep-internal-metavariable-name:\n metavariable: $X\n fqns: [foo1, foo2.default]\n"]