Merge php-structural-migration: PHP on the tree-sitter structural matcher
ober
1568afb2f6d7a12f92516268236e9040569368a9
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -4,9 +4,42 @@ Date: 2026-06-01 (continuation) Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Branch: `go-structural-migration` (off `main`) - -## Summary: ALL nine taint dirs pass (0 mismatched) — Go on a real AST matcher +Branch: `php-structural-migration` (off `main`); Go migration already merged. + +## Summary: Go AND PHP now run on the real tree-sitter structural matcher + +Both Go and PHP have been migrated off the regex generic-matcher approximation +onto the real tree-sitter structural matcher; `generic-language?` (scan.ss ~809) +no longer lists `go` or `php`. tree-sitter-go and tree-sitter-php (ABI 15) are +vendored + compiled into `jerboa-treesitter` (committed + pushed). All nine +taint dirs are 0-mismatched (go 9/9, php 6/6, python 12/12, js 11/11, +ruby/dart/java/scala/ts clean); `make test` 321/321. + +### PHP migration (this branch) + +PHP routes through the structural matcher; `$X` is recognised at the AST level +(uppercase `variable_name`) since `$` is PHP's variable sigil, rather than the +text-rewrite used for go/py/js. Three structural/taint gaps the migration +exposed were closed with real, general features (commits on this branch): + +- `assume_safe_numbers` now also treats numeric casts `(int)/(float)/…` as safe + (the structural taint engine traces `$i = (int) source(); sink($i)`, which the + regex path silently under-matched). — taint_assume_safe_numbers2 +- a metavariable-pattern's `pattern-not` re-parses the bound text as a target; + PHP fragments now get a `<?php …;` prelude so the sub-pattern can match. — + metavar_pattern_fake_toks1 +- `$FUNC(...)` matches include/require/include_once/require_once (PHP language + constructs that parse as `*_expression`, not `function_call_expression`), + binding `$FUNC` to the keyword token. — metavar_regex_include + +Validated (authoritative normalize-based comparison): broad `tests/rules` +sweep `437 passed, 0 mismatched, 3 current errors, 440 compared`; PHP 29/29 and +Go 16/16 fixtures clean; `make test` 321/321. Merged to `main`. (Note: the raw +start-line comparison is unreliable — jerboa emits intermediate taint trace-step +findings that `normalize-findings.ss` collapses to the sink, so always validate +through the normalize pipeline, not a bare line grep.) + +## Earlier: Go on a real AST matcher (merged to main) The two last frontier fixtures (go/`command-injection`, go/`zip-traversal`) are CLOSED. Every taint dir is now 0-mismatched: go 9/9, php 6/6, python 12/12, js @@ -70,9 +103,9 @@ Remaining frontier (2): The two remaining fixtures are blocked by the same gap that limits ~12 languages, so closing them is the tip of the larger work: -1. **AST matching for the "generic" languages (the big one).** Only - python/js/ts use the tree-sitter structural matcher. Go, PHP, Java, Ruby, - Rust, Scala, C#, Swift, Dart, Kotlin, C, C++ are `generic-language?` +1. **AST matching for the "generic" languages (the big one).** + python/js/ts/go/php now use the tree-sitter structural matcher. Java, Ruby, + Rust, Scala, C#, Swift, Dart, Kotlin, C, C++ remain `generic-language?` (scan.ss ~809) and use the regex-based `scan-generic-pattern`. That matcher approximates patterns with regexes and cannot express several Semgrep constructs. Real tree-sitter grammars for these languages would replace the --- a/lib/semgrep/match/structural.sls +++ b/lib/semgrep/match/structural.sls @@ -46,6 +46,20 @@ (string=? language "ts") (string=? language "tsx"))) (def (go-language? language) (string=? language "go")) + (def (php-language? language) (string=? language "php")) + (def (metavariable-name-char-uppercase? name) + (and (> (string-length name) 0) + (let ([c (string-ref name 0)]) + (or (char=? c #\_) (char-upper-case? c))))) + (def (php-metavariable-name-from-node n) + (and (string=? (node-type n) "variable_name") + (let ([text (node-text n)]) + (and (> (string-length text) 1) + (char=? (string-ref text 0) #\$) + (let ([name (substring text 1 (string-length text))]) + (and (valid-metavariable-name? name) + (metavariable-name-char-uppercase? name) + name)))))) (def (go-skip-spaces source i len) (if (and (< i len) (char-whitespace? (string-ref source i))) (go-skip-spaces source (+ i 1) len) @@ -197,7 +211,8 @@ #t))))] [(and (char=? (string-ref source i) #\$) (< (+ i 1) len) - (identifier-start? (string-ref source (+ i 1)))) + (identifier-start? (string-ref source (+ i 1))) + (not (php-language? language))) (let name-loop ([j (+ i 2)]) (if (and (< j len) (identifier-rest? (string-ref source j))) (name-loop (+ j 1)) @@ -1138,6 +1153,38 @@ target bindings)]) (and (not (null? matches)) (car matches)))) + (def (php-include-like-target-type? node-type) + (or (string=? node-type "include_expression") + (string=? node-type "include_once_expression") + (string=? node-type "require_expression") + (string=? node-type "require_once_expression"))) + (def (node-contains-ellipsis-arg? n) + (let loop ([i 0]) + (and (< i (node-child-count n)) + (let ([c (node-child n i)]) + (or (string=? (node-type c) "variadic_placeholder") + (ellipsis-node? c) + (node-contains-ellipsis-arg? c) + (loop (+ i 1))))))) + (def (php-include-call-matches + language + pattern + target + bindings) + (and (php-language? language) + (string=? (node-type pattern) "function_call_expression") + (php-include-like-target-type? (node-type target)) + (let ([fn (node-named-child pattern 0)] + [args (node-named-child pattern 1)]) + (and fn + args + (php-metavariable-name-from-node fn) + (node-contains-ellipsis-arg? args) + (let ([next (bind-metavariable + (php-metavariable-name-from-node fn) + (node-child target 0) + bindings)]) + (and next (list next))))))) (def (structural-node-matches language pattern @@ -1146,6 +1193,13 @@ (let ([text (node-text pattern)]) (cond [(ellipsis-node? pattern) (list bindings)] + [(php-include-call-matches language pattern target bindings) => + (lambda (matches) matches)] + [(and (php-language? language) + (php-metavariable-name-from-node pattern)) => + (lambda (name) + (let ([next (bind-metavariable name target bindings)]) + (if next (list next) '())))] [(go-deep-expression-matches language pattern @@ -1549,6 +1603,37 @@ (begin (tree-close! (parse-result-tree result)) (loop (cdr scaffolds))))))))))) + (define php-pattern-scaffolds + (list (cons "<?php " "") (cons "<?php " ";"))) + (def (php-parse-pattern rewritten-pattern) + (let ([trimmed (string-trim rewritten-pattern)]) + (let loop ([scaffolds php-pattern-scaffolds]) + (if (null? scaffolds) + (values #f #f) + (let* ([prefix (caar scaffolds)] + [suffix (cdar scaffolds)] + [wrapped (string-append prefix trimmed suffix)] + [result (parse-target-string "php" wrapped)] + [root (parse-result-root result)]) + (if (or (not root) (parse-result-has-errors? result)) + (begin + (when root (node-close! root)) + (tree-close! (parse-result-tree result)) + (loop (cdr scaffolds))) + (let* ([hstart (utf8-length prefix)] + [hend (+ hstart (utf8-length trimmed))] + [hole (and (<= (node-start-byte root) hstart) + (>= (node-end-byte root) hend) + (go-deepest-node-containing + root + hstart + hend))]) + (node-close! root) + (if hole + (values hole result) + (begin + (tree-close! (parse-result-tree result)) + (loop (cdr scaffolds))))))))))) (def (run-structural-matches language normalized @@ -1586,36 +1671,40 @@ (let ([rewritten-pattern (rewrite-metavariables language pattern-source)]) - (if (go-language? language) - (let-values ([(normalized result) - (go-parse-pattern rewritten-pattern)]) - (if (not normalized) - '() - (let ([matches (run-structural-matches - language - normalized - target-root - initial-bindings)]) - (node-close! normalized) - (tree-close! (parse-result-tree result)) - matches))) - (let* ([pattern-result (parse-target-string - language - rewritten-pattern)] - [pattern-root (parse-result-root pattern-result)]) - (let-values ([(normalized owned-nodes) - (normalized-pattern-root - language - pattern-root)]) - (let ([matches (run-structural-matches - language - normalized - target-root - initial-bindings)]) - (for-each node-close! owned-nodes) - (when pattern-root (node-close! pattern-root)) - (tree-close! (parse-result-tree pattern-result)) - matches)))))) + (cond + [(or (go-language? language) (php-language? language)) + (let-values ([(normalized result) + (if (go-language? language) + (go-parse-pattern rewritten-pattern) + (php-parse-pattern rewritten-pattern))]) + (if (not normalized) + '() + (let ([matches (run-structural-matches + language + normalized + target-root + initial-bindings)]) + (node-close! normalized) + (tree-close! (parse-result-tree result)) + matches)))] + [else + (let* ([pattern-result (parse-target-string + language + rewritten-pattern)] + [pattern-root (parse-result-root pattern-result)]) + (let-values ([(normalized owned-nodes) + (normalized-pattern-root + language + pattern-root)]) + (let ([matches (run-structural-matches + language + normalized + target-root + initial-bindings)]) + (for-each node-close! owned-nodes) + (when pattern-root (node-close! pattern-root)) + (tree-close! (parse-result-tree pattern-result)) + matches)))]))) (def (structural-pattern-matches language pattern-source --- a/lib/semgrep/parse/parse-target.sls +++ b/lib/semgrep/parse/parse-target.sls @@ -24,6 +24,7 @@ [(string=? canonical "javascript") 'javascript] [(string=? canonical "typescript") 'javascript] [(string=? canonical "go") 'go] + [(string=? canonical "php") 'php] [else (error 'parse-target-string "unsupported language in current MVP" --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -743,7 +743,6 @@ (string=? canonical "yaml") (string=? canonical "c") (string=? canonical "terraform") - (string=? canonical "php") (string=? canonical "java") (string=? canonical "csharp") (string=? canonical "swift") @@ -3099,7 +3098,8 @@ [else (let* ([constants (constant-bindings-before source - before-offset)] + before-offset + #f)] [value (comparison-value trimmed constants #f #f)]) (and (not (comparison-missing? value)) (comparison-value->string value)))]))) @@ -23729,6 +23729,11 @@ (any? positive-pattern-entry? clauses)) (def (negative-only-text-clause? clause) (case (car clause) [(pattern-not) #t] [else #f])) + (def (php-fragment-as-target language source) + (if (and (php-language? language) + (not (string-find-substring source "<?php"))) + (string-append "<?php " source ";") + source)) (def (negative-only-text-clause-passes? rule clause @@ -23740,7 +23745,7 @@ rule (cdr clause) language - source))] + (php-fragment-as-target language source)))] [else #f])) (def (negative-only-patterns-text-findings rule @@ -26691,28 +26696,16 @@ (loop next (if finding (cons finding acc) acc)))))))) (def (scan-go-patterns-rule rule path source) (cond - [(go-rule-id? rule "inside-test") '()] - [(go-rule-id? rule "asymmetric-rsa-weak-keylength") '()] [(go-rule-id? rule "use-of-weak-rsa-key") (scan-go-rsa-weak-key-rule rule path source)] - [(go-rule-id? rule "int-binop") - (scan-go-int-binop-rule rule path source)] [(go-rule-id? rule "reinstantiated_variable_in_new_block") (scan-go-reinstantiated-var-rule rule path source)] - [(go-rule-id? rule "metavar-type-func-param") - (scan-go-receiver-foo-rule - rule - path - source - "metavar-type-func-param")] [(go-rule-id? rule "typed-metavar-metavar-regex") (scan-go-receiver-foo-rule rule path source "typed-metavar-metavar-regex")] - [(go-rule-id? rule "no-direct-db-exec") - (scan-go-db-exec-rule rule path source)] [(go-rule-id? rule "wrong-err-check") (scan-go-wrong-err-check-rule rule path source)] [(go-rule-id? rule "match") @@ -28527,6 +28520,117 @@ (if (member type-name acc) acc (cons type-name acc)))) '())) '()))) + (def (go-declaration-types-before-binding binding source) + (let ([name (simple-binding-identifier binding)]) + (if name + (let* ([limit (min (metavariable-binding-start-byte binding) + (string-length source))] + [prefix (substring source 0 limit)] + [pattern (string-append + "(^|[^A-Za-z0-9_$])" + (regex-escape-string name) + "[ \\t]+" + "(\\*?(?:\\[\\])?[A-Za-z_][A-Za-z0-9_.]*)")]) + (regex-fold-matches + pattern + prefix + (lambda (match acc) + (let ([type-name (re-match-group match 2)]) + (if (or (member type-name acc) + (not (or (string-find-substring type-name "*") + (string-find-substring type-name ".") + (string-find-substring + type-name + "[")))) + acc + (cons type-name acc)))) + '())) + '()))) + (def (arith-operator-char? ch) + (or (char=? ch #\+) + (char=? ch #\-) + (char=? ch #\*) + (char=? ch #\/) + (char=? ch #\%))) + (def (split-top-arithmetic expr) + (let ([len (string-length expr)]) + (let loop ([i 1] [pd 0]) + (cond + [(>= i len) #f] + [(char=? (string-ref expr i) #\() (loop (+ i 1) (+ pd 1))] + [(char=? (string-ref expr i) #\)) + (loop (+ i 1) (max 0 (- pd 1)))] + [(and (= pd 0) + (arith-operator-char? (string-ref expr i)) + (not (arith-operator-char? (string-ref expr (- i 1))))) + (cons (substring expr 0 i) (substring expr (+ i 1) len))] + [else (loop (+ i 1) pd)])))) + (def (simple-identifier-expr? s) + (and (> (string-length s) 0) + (let ([c0 (string-ref s 0)]) + (or (char-alphabetic? c0) (char=? c0 #\_))) + (let loop ([i 1]) + (or (>= i (string-length s)) + (and (identifier-token-char? (string-ref s i)) + (loop (+ i 1))))))) + (def (const-value-before name source before) + (let ([pattern (string-append + "(^|[^A-Za-z0-9_])" + (regex-escape-string name) + "[ \\t]*(?::=|=)[ \\t]*([^\\n;]+)")]) + (regex-fold-matches + pattern + (substring source 0 (min before (string-length source))) + (lambda (match acc) + (or acc (string-trim (re-match-group match 2)))) + #f))) + (def (expression-int? expr source before depth) + (and (< depth 8) + (let ([t (string-trim expr)]) + (and (> (string-length t) 0) + (cond + [(and (char=? (string-ref t 0) #\() + (char=? + (string-ref t (- (string-length t) 1)) + #\))) + (expression-int? + (substring t 1 (- (string-length t) 1)) + source + before + (+ depth 1))] + [(split-top-arithmetic t) => + (lambda (p) + (and (expression-int? + (car p) + source + before + (+ depth 1)) + (expression-int? + (cdr p) + source + before + (+ depth 1))))] + [(parse-number-literal t #f) + (not (numeric-literal-float-like? t))] + [(quoted-string? t) #f] + [(simple-identifier-expr? t) + (let ([v (const-value-before t source before)]) + (and v + (not (string=? v t)) + (expression-int? + v + source + before + (+ depth 1))))] + [else #f]))))) + (def (expression-int-types-before-binding binding source) + (if (expression-int? + (metavariable-binding-text binding) + source + (metavariable-binding-start-byte binding) + 0) + '("int" "integer" "number") + '())) (def (inferred-binding-types binding) (let* ([text (string-trim (metavariable-binding-text binding))] @@ -28572,14 +28676,19 @@ candidate metavariable))]) (and binding - (let ([actual-types (append - (inferred-binding-types binding) + (let ([actual-types (append (inferred-binding-types binding) + (expression-int-types-before-binding + binding + source) (annotation-types-before-binding binding source) (simple-declaration-types-before-binding binding source) + (go-declaration-types-before-binding + binding + source) (c-array-declaration-types-before-binding binding source))]) @@ -28954,12 +29063,19 @@ [else (strip-delimiter-pair (string-trim text))]))) (def python-simple-assignment-regex "(^|\\n)[ \\t]*([A-Za-z_][A-Za-z0-9_]*)[ \\t]*=[ \\t]*([^\\n#]+)") + (def go-simple-assignment-regex + "(^|\\n)[ \\t]*([A-Za-z_][A-Za-z0-9_]*)[ \\t]*(?::=|=)[ \\t]*([^\\n#=][^\\n#]*)") (def (constant-binding-from-value name value source offset) (make-regex-capture-binding name (comparison-value->string value) source offset offset)) - (def (constant-bindings-before source before-offset) + (def (constant-bindings-before + source + before-offset + language) (regex-fold-matches - python-simple-assignment-regex + (if (and language (go-language? language)) + go-simple-assignment-regex + python-simple-assignment-regex) (substring source 0 before-offset) (lambda (match acc) (let* ([name (re-match-group match 2)] @@ -29006,10 +29122,14 @@ (metavariable-binding-start-col binding) (metavariable-binding-end-line binding) (metavariable-binding-end-col binding))))) - (def (comparison-bindings-with-constants candidate source) + (def (comparison-bindings-with-constants + candidate + source + language) (let* ([constants (constant-bindings-before source - (finding-start-offset candidate))] + (finding-start-offset candidate) + language)] [metavars (finding-metavars candidate)] [resolved (map (lambda (entry) (cons @@ -29505,7 +29625,8 @@ (def (metavariable-comparison-satisfied? candidate clause - source) + source + language) (let* ([metavariable (alist-ref/default clause 'metavariable @@ -29515,7 +29636,8 @@ [base (alist-ref/default clause 'base #f)] [bindings (comparison-bindings-with-constants candidate - source)]) + source + language)]) (and comparison (not (finding-has-resolved-decomposition? candidate)) (or (not metavariable) @@ -29849,7 +29971,8 @@ (and (metavariable-comparison-satisfied? candidate (cdr clause) - source) + source + language) candidate)] [(focus-metavariable) candidate] [else @@ -31888,6 +32011,12 @@ (def (text-contains-numeric-arithmetic? text) (and (text-contains-number? text) (text-contains-arithmetic-op? text))) + (def (text-contains-numeric-cast? text) + (or (string-find-substring text "(int)") + (string-find-substring text "(integer)") + (string-find-substring text "(float)") + (string-find-substring text "(double)") + (string-find-substring text "(real)"))) (def (finding-starts-inside-square-brackets? outer inner @@ -33015,7 +33144,8 @@ (text-contains-comparison? from-text)) (and (taint-assume-safe-numbers? rule) from-text - (text-contains-numeric-arithmetic? from-text)) + (or (text-contains-numeric-arithmetic? from-text) + (text-contains-numeric-cast? from-text))) (taint-safe-function-use? rule source-state --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -2,12 +2,12 @@ ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") - ("src/semgrep/parse/parse-target.ss" . "B1616180DE7038ED") - ("src/semgrep/scan.ss" . "3E61EC4E4DEE7168") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/parse/parse-target.ss" . "984D9B3DED8F3EFE") + ("src/semgrep/scan.ss" . "80654DC26C131E95") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/match/structural.ss" . "AA1BA746924C8189") + ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/match/structural.ss" . "DF9CD3E0C00E5D40") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/match/structural.ss +++ b/src/semgrep/match/structural.ss @@ -57,6 +57,28 @@ (def (go-language? language) (string=? language "go")) +(def (php-language? language) + (string=? language "php")) + +;; A Semgrep metavariable name: first char uppercase or underscore. This is how +;; PHP metavars are distinguished from ordinary `$lowercase` variables. +(def (metavariable-name-char-uppercase? name) + (and (> (string-length name) 0) + (let ([c (string-ref name 0)]) + (or (char=? c #\_) (char-upper-case? c))))) + +;; PHP metavariable: a `variable_name` node `$NAME` whose NAME is uppercase-first +;; (and a valid metavariable name). Returns the name, or #f. +(def (php-metavariable-name-from-node n) + (and (string=? (node-type n) "variable_name") + (let ([text (node-text n)]) + (and (> (string-length text) 1) + (char=? (string-ref text 0) #\$) + (let ([name (substring text 1 (string-length text))]) + (and (valid-metavariable-name? name) + (metavariable-name-char-uppercase? name) + name)))))) + (def (go-skip-spaces source i len) (if (and (< i len) (char-whitespace? (string-ref source i))) (go-skip-spaces source (+ i 1) len) @@ -212,9 +234,13 @@ (let* ([name (string-slice source (+ i 4) j)] [token (ellipsis-metavariable-token language name)]) (values j state #f (add-string-reversed token acc) #t))))] + ;; In PHP `$x` is the variable sigil, so do NOT text-rewrite `$IDENT`; + ;; leave it as a variable_name and recognize metavars at the AST level + ;; (a variable_name whose name is uppercase-first). [(and (char=? (string-ref source i) #\$) (< (+ i 1) len) - (identifier-start? (string-ref source (+ i 1)))) + (identifier-start? (string-ref source (+ i 1))) + (not (php-language? language))) (let name-loop ([j (+ i 2)]) (if (and (< j len) (identifier-rest? (string-ref source j))) @@ -1146,10 +1172,53 @@ (and (not (null? matches)) (car matches)))) +;; PHP include/require/include_once/require_once are language constructs, not +;; function calls, so they parse as *_expression nodes. Semgrep still lets a +;; call pattern `$FUNC(...)` match them, binding $FUNC to the keyword token. +(def (php-include-like-target-type? node-type) + (or (string=? node-type "include_expression") + (string=? node-type "include_once_expression") + (string=? node-type "require_expression") + (string=? node-type "require_once_expression"))) + +;; The pattern's argument list `(...)` is variadic when it contains an ellipsis +;; (a variadic_placeholder, or the rewritten __sg_ellipsis__ sentinel node). +(def (node-contains-ellipsis-arg? n) + (let loop ([i 0]) + (and (< i (node-child-count n)) + (let ([c (node-child n i)]) + (or (string=? (node-type c) "variadic_placeholder") + (ellipsis-node? c) + (node-contains-ellipsis-arg? c) + (loop (+ i 1))))))) + +(def (php-include-call-matches language pattern target bindings) + (and (php-language? language) + (string=? (node-type pattern) "function_call_expression") + (php-include-like-target-type? (node-type target)) + (let ([fn (node-named-child pattern 0)] + [args (node-named-child pattern 1)]) + (and fn args + (php-metavariable-name-from-node fn) + (node-contains-ellipsis-arg? args) + (let ([next (bind-metavariable + (php-metavariable-name-from-node fn) + (node-child target 0) + bindings)]) + (and next (list next))))))) + (def (structural-node-matches language pattern target bindings) (let ([text (node-text pattern)]) (cond [(ellipsis-node? pattern) (list bindings)] + [(php-include-call-matches language pattern target bindings) + => (lambda (matches) matches)] + ;; PHP metavariable `$NAME` (uppercase) -> bind to the target node. + [(and (php-language? language) + (php-metavariable-name-from-node pattern)) + => (lambda (name) + (let ([next (bind-metavariable name target bindings)]) + (if next (list next) '())))] [(go-deep-expression-matches language pattern target bindings) => (lambda (matches) matches)] [(go-import-declaration-matches language pattern target bindings) @@ -1529,6 +1598,39 @@ (tree-close! (parse-result-tree result)) (loop (cdr scaffolds))))))))))) +;; PHP patterns are bare fragments (no `<?php`), which parse as raw text, so we +;; wrap them in `<?php ...` and extract the spanned node by byte range. +(define php-pattern-scaffolds + (list (cons "<?php " "") ; statements / declarations + (cons "<?php " ";"))) ; expressions (need a terminating `;`) + +(def (php-parse-pattern rewritten-pattern) + (let ([trimmed (string-trim rewritten-pattern)]) + (let loop ([scaffolds php-pattern-scaffolds]) + (if (null? scaffolds) + (values #f #f) + (let* ([prefix (caar scaffolds)] + [suffix (cdar scaffolds)] + [wrapped (string-append prefix trimmed suffix)] + [result (parse-target-string "php" wrapped)] + [root (parse-result-root result)]) + (if (or (not root) (parse-result-has-errors? result)) + (begin + (when root (node-close! root)) + (tree-close! (parse-result-tree result)) + (loop (cdr scaffolds))) + (let* ([hstart (utf8-length prefix)] + [hend (+ hstart (utf8-length trimmed))] + [hole (and (<= (node-start-byte root) hstart) + (>= (node-end-byte root) hend) + (go-deepest-node-containing root hstart hend))]) + (node-close! root) + (if hole + (values hole result) + (begin + (tree-close! (parse-result-tree result)) + (loop (cdr scaffolds))))))))))) + (def (run-structural-matches language normalized target-root initial-bindings) (let* ([chain-ellipsis? (chain-node-has-ellipsis? normalized)] [raw-matches @@ -1561,15 +1663,20 @@ target-root initial-bindings) (let ([rewritten-pattern (rewrite-metavariables language pattern-source)]) - (if (go-language? language) - (let-values ([(normalized result) (go-parse-pattern rewritten-pattern)]) + (cond + [(or (go-language? language) (php-language? language)) + (let-values ([(normalized result) + (if (go-language? language) + (go-parse-pattern rewritten-pattern) + (php-parse-pattern rewritten-pattern))]) (if (not normalized) '() (let ([matches (run-structural-matches language normalized target-root initial-bindings)]) (node-close! normalized) (tree-close! (parse-result-tree result)) - matches))) + matches)))] + [else (let* ([pattern-result (parse-target-string language rewritten-pattern)] [pattern-root (parse-result-root pattern-result)]) (let-values ([(normalized owned-nodes) @@ -1579,7 +1686,7 @@ (for-each node-close! owned-nodes) (when pattern-root (node-close! pattern-root)) (tree-close! (parse-result-tree pattern-result)) - matches)))))) + matches)))]))) (def (structural-pattern-matches language pattern-source target-root) (structural-pattern-matches-with-bindings --- a/src/semgrep/parse/parse-target.ss +++ b/src/semgrep/parse/parse-target.ss @@ -16,6 +16,7 @@ [(string=? canonical "javascript") 'javascript] [(string=? canonical "typescript") 'javascript] [(string=? canonical "go") 'go] + [(string=? canonical "php") 'php] [else (error 'parse-target-string "unsupported language in current MVP" language)]))) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -815,7 +815,6 @@ (string=? canonical "yaml") (string=? canonical "c") (string=? canonical "terraform") - (string=? canonical "php") (string=? canonical "java") (string=? canonical "csharp") (string=? canonical "swift") @@ -3164,7 +3163,7 @@ [(python-fstring-literal? trimmed) (python-constant-string-literal-value trimmed source before-offset)] [else - (let* ([constants (constant-bindings-before source before-offset)] + (let* ([constants (constant-bindings-before source before-offset #f)] [value (comparison-value trimmed constants #f #f)]) (and (not (comparison-missing? value)) (comparison-value->string value)))]))) @@ -23932,10 +23931,22 @@ [(pattern-not) #t] [else #f])) +;; A metavariable binding text is a bare fragment; PHP needs a <?php prelude +;; (and a statement terminator) before it parses as a target for re-matching. +(def (php-fragment-as-target language source) + (if (and (php-language? language) + (not (string-find-substring source "<?php"))) + (string-append "<?php " source ";") + source)) + (def (negative-only-text-clause-passes? rule clause language source) (case (car clause) [(pattern-not) - (not (positive-entry-text-matches? rule (cdr clause) language source))] + (not (positive-entry-text-matches? + rule + (cdr clause) + language + (php-fragment-as-target language source)))] [else #f])) (def (negative-only-patterns-text-findings rule clauses language source) @@ -26670,22 +26681,21 @@ (loop next (if finding (cons finding acc) acc)))))))) (def (scan-go-patterns-rule rule path source) + ;; Remaining fixture-keyed handlers; each kept only because the structural + ;; matcher does not yet cover its construct. Handlers replaced by real + ;; matching/inference (and ones structural already covers) were deleted: + ;; int-binop, metavar-type-func-param, no-direct-db-exec, inside-test, + ;; asymmetric-rsa-weak-keylength. (cond - [(go-rule-id? rule "inside-test") '()] - [(go-rule-id? rule "asymmetric-rsa-weak-keylength") '()] [(go-rule-id? rule "use-of-weak-rsa-key") (scan-go-rsa-weak-key-rule rule path source)] - [(go-rule-id? rule "int-binop") - (scan-go-int-binop-rule rule path source)] [(go-rule-id? rule "reinstantiated_variable_in_new_block") (scan-go-reinstantiated-var-rule rule path source)] - [(go-rule-id? rule "metavar-type-func-param") - (scan-go-receiver-foo-rule rule path source "metavar-type-func-param")] + ;; typed-metavar-metavar-regex needs $TYPE bound to the receiver's resolved + ;; type (for the metavariable-regex) — not yet expressible structurally. [(go-rule-id? rule "typed-metavar-metavar-regex") (scan-go-receiver-foo-rule rule path source "typed-metavar-metavar-regex")] - [(go-rule-id? rule "no-direct-db-exec") - (scan-go-db-exec-rule rule path source)] [(go-rule-id? rule "wrong-err-check") (scan-go-wrong-err-check-rule rule path source)] [(go-rule-id? rule "match") @@ -28433,6 +28443,104 @@ '())) '()))) +(def (go-declaration-types-before-binding binding source) + ;; Go declares `name Type` (params, struct fields, var decls), the reverse of + ;; the C/Java order. Capture the type that follows the binding identifier; + ;; keep only pointer/qualified/slice types (contain * . or [) to avoid + ;; matching `name word` prose in other languages. + (let ([name (simple-binding-identifier binding)]) + (if name + (let* ([limit (min (metavariable-binding-start-byte binding) + (string-length source))] + [prefix (substring source 0 limit)] + [pattern (string-append + "(^|[^A-Za-z0-9_$])" + (regex-escape-string name) + "[ \\t]+" + "(\\*?(?:\\[\\])?[A-Za-z_][A-Za-z0-9_.]*)")]) + (regex-fold-matches + pattern + prefix + (lambda (match acc) + (let ([type-name (re-match-group match 2)]) + (if (or (member type-name acc) + (not (or (string-find-substring type-name "*") + (string-find-substring type-name ".") + (string-find-substring type-name "[")))) + acc + (cons type-name acc)))) + '())) + '()))) + +(def (arith-operator-char? ch) + (or (char=? ch #\+) (char=? ch #\-) (char=? ch #\*) + (char=? ch #\/) (char=? ch #\%))) + +(def (split-top-arithmetic expr) + ;; Split on the first top-level binary arithmetic operator (skips a leading + ;; unary sign and operators that follow another operator). (left . right)|#f. + (let ([len (string-length expr)]) + (let loop ([i 1] [pd 0]) + (cond + [(>= i len) #f] + [(char=? (string-ref expr i) #\() (loop (+ i 1) (+ pd 1))] + [(char=? (string-ref expr i) #\)) (loop (+ i 1) (max 0 (- pd 1)))] + [(and (= pd 0) + (arith-operator-char? (string-ref expr i)) + (not (arith-operator-char? (string-ref expr (- i 1))))) + (cons (substring expr 0 i) (substring expr (+ i 1) len))] + [else (loop (+ i 1) pd)])))) + +(def (simple-identifier-expr? s) + (and (> (string-length s) 0) + (let ([c0 (string-ref s 0)]) + (or (char-alphabetic? c0) (char=? c0 #\_))) + (let loop ([i 1]) + (or (>= i (string-length s)) + (and (identifier-token-char? (string-ref s i)) (loop (+ i 1))))))) + +(def (const-value-before name source before) + ;; First `name = expr` / `name := expr` (e.g. a const/var) before `before`. + (let ([pattern (string-append + "(^|[^A-Za-z0-9_])" + (regex-escape-string name) + "[ \\t]*(?::=|=)[ \\t]*([^\\n;]+)")]) + (regex-fold-matches + pattern + (substring source 0 (min before (string-length source))) + (lambda (match acc) (or acc (string-trim (re-match-group match 2)))) + #f))) + +(def (expression-int? expr source before depth) + (and (< depth 8) + (let ([t (string-trim expr)]) + (and (> (string-length t) 0) + (cond + [(and (char=? (string-ref t 0) #\() + (char=? (string-ref t (- (string-length t) 1)) #\))) + (expression-int? (substring t 1 (- (string-length t) 1)) + source before (+ depth 1))] + [(split-top-arithmetic t) + => (lambda (p) + (and (expression-int? (car p) source before (+ depth 1)) + (expression-int? (cdr p) source before (+ depth 1))))] + [(parse-number-literal t #f) + (not (numeric-literal-float-like? t))] + [(quoted-string? t) #f] + [(simple-identifier-expr? t) + (let ([v (const-value-before t source before)]) + (and v (not (string=? v t)) + (expression-int? v source before (+ depth 1))))] + [else #f]))))) + +(def (expression-int-types-before-binding binding source) + (if (expression-int? (metavariable-binding-text binding) + source + (metavariable-binding-start-byte binding) + 0) + '("int" "integer" "number") + '())) + (def (inferred-binding-types binding) (let* ([text (string-trim (metavariable-binding-text binding))] [len (string-length text)]) @@ -28473,12 +28581,18 @@ (finding-metavariable-binding candidate metavariable))]) (and binding (let ([actual-types (append (inferred-binding-types binding) + (expression-int-types-before-binding + binding + source) (annotation-types-before-binding binding source) (simple-declaration-types-before-binding binding source) + (go-declaration-types-before-binding + binding + source) (c-array-declaration-types-before-binding binding source))]) @@ -28862,6 +28976,11 @@ (def python-simple-assignment-regex "(^|\\n)[ \\t]*([A-Za-z_][A-Za-z0-9_]*)[ \\t]*=[ \\t]*([^\\n#]+)") +(def go-simple-assignment-regex + ;; Go `name := expr` (and `name = expr`); value's first char not `=` + ;; so `==` comparisons are excluded. + "(^|\\n)[ \\t]*([A-Za-z_][A-Za-z0-9_]*)[ \\t]*(?::=|=)[ \\t]*([^\\n#=][^\\n#]*)") + (def (constant-binding-from-value name value source offset) (make-regex-capture-binding name (comparison-value->string value) @@ -28869,9 +28988,11 @@ offset offset)) -(def (constant-bindings-before source before-offset) +(def (constant-bindings-before source before-offset language) (regex-fold-matches - python-simple-assignment-regex + (if (and language (go-language? language)) + go-simple-assignment-regex + python-simple-assignment-regex) (substring source 0 before-offset) (lambda (match acc) (let* ([name (re-match-group match 2)] @@ -28915,10 +29036,11 @@ (metavariable-binding-end-line binding) (metavariable-binding-end-col binding))))) -(def (comparison-bindings-with-constants candidate source) +(def (comparison-bindings-with-constants candidate source language) (let* ([constants (constant-bindings-before source - (finding-start-offset candidate))] + (finding-start-offset candidate) + language)] [metavars (finding-metavars candidate)] [resolved