Route Java through the tree-sitter structural matcher
ober
6abbe64fbcc3b179ab08abd9a01550f5dffcbaf4
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -6,14 +6,29 @@ Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Branch: `php-structural-migration` (off `main`); Go migration already merged. -## Summary: Go AND PHP now run on the real tree-sitter structural matcher - -Both Go and PHP have been migrated off the regex generic-matcher approximation -onto the real tree-sitter structural matcher; `generic-language?` (scan.ss ~809) -no longer lists `go` or `php`. tree-sitter-go and tree-sitter-php (ABI 15) are -vendored + compiled into `jerboa-treesitter` (committed + pushed). All nine -taint dirs are 0-mismatched (go 9/9, php 6/6, python 12/12, js 11/11, -ruby/dart/java/scala/ts clean); `make test` 321/321. +## Summary: Go, PHP, AND Java now run on the real tree-sitter structural matcher + +Go, PHP, and Java have been migrated off the regex generic-matcher +approximation onto the real tree-sitter structural matcher; `generic-language?` +(scan.ss ~809) no longer lists `go`, `php`, or `java`. tree-sitter-go/php +(ABI 15) and tree-sitter-java (ABI 14, within the runtime's 13-15 window) are +vendored + compiled into `jerboa-treesitter`. `make test` 321/321. + +### Java migration (branch `java-structural-migration`) + +Java flipped to structural with ZERO fixture regressions on the first try +(42/42 `tests/rules` `.java` fixtures still match Semgrep by the normalize +oracle) — unlike Go (which dropped 7/9→3/9 on its first flip and needed many +new structural operators). Two reasons: (1) the structural matcher is now +mature from the Go/PHP work, and (2) the `java-language?`-gated overfit handlers +(`scan-java-patterns-rule`) still fire, so they carry their specific fixtures +while structural handles general matching. Java identifiers also survive the +`$X`→`__sg_mvar_X` / `...`→`__sg_ellipsis__` text rewrite unchanged (no PHP-style +sigil problem). jerboa-treesitter additions: `vendor/grammars/java` (parser-only, +no scanner — like Go), `jt_language_java`, `ffi-language-java`, `known-languages` ++ `language-ref` java clause, smoke "parse Java" (13/13). The +`scan-java-patterns-rule` overfit handlers remain to be deleted as structural +gains their constructs (same incremental plan as Go/PHP). ### PHP migration (this branch) @@ -104,7 +119,7 @@ The two remaining fixtures are blocked by the same gap that limits ~12 languages, so closing them is the tip of the larger work: 1. **AST matching for the "generic" languages (the big one).** - python/js/ts/go/php now use the tree-sitter structural matcher. Java, Ruby, + python/js/ts/go/php/java now use the tree-sitter structural matcher. Ruby, Rust, Scala, C#, Swift, Dart, Kotlin, C, C++ remain `generic-language?` (scan.ss ~809) and use the regex-based `scan-generic-pattern`. That matcher approximates patterns with regexes and cannot express several Semgrep --- a/lib/semgrep/parse/parse-target.sls +++ b/lib/semgrep/parse/parse-target.sls @@ -25,6 +25,7 @@ [(string=? canonical "typescript") 'javascript] [(string=? canonical "go") 'go] [(string=? canonical "php") 'php] + [(string=? canonical "java") 'java] [else (error 'parse-target-string "unsupported language in current MVP" --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -743,7 +743,6 @@ (string=? canonical "yaml") (string=? canonical "c") (string=? canonical "terraform") - (string=? canonical "java") (string=? canonical "csharp") (string=? canonical "swift") (string=? canonical "dart") --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -2,12 +2,12 @@ ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") - ("src/semgrep/parse/parse-target.ss" . "984D9B3DED8F3EFE") - ("src/semgrep/scan.ss" . "80654DC26C131E95") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/parse/parse-target.ss" . "DD1EF1A5E883A90D") + ("src/semgrep/scan.ss" . "4EA92FEDD236FE4F") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "DF9CD3E0C00E5D40") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/parse/parse-target.ss +++ b/src/semgrep/parse/parse-target.ss @@ -17,6 +17,7 @@ [(string=? canonical "typescript") 'javascript] [(string=? canonical "go") 'go] [(string=? canonical "php") 'php] + [(string=? canonical "java") 'java] [else (error 'parse-target-string "unsupported language in current MVP" language)]))) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -815,7 +815,6 @@ (string=? canonical "yaml") (string=? canonical "c") (string=? canonical "terraform") - (string=? canonical "java") (string=? canonical "csharp") (string=? canonical "swift") (string=? canonical "dart")