Update handoff: Scala+Rust migrated; wrapper-root-type? made java/scala/rust structural matching real
ober
cbec9ca4c558c5390e42edc80fc7e9e5fe754b4f
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -4,31 +4,36 @@ Date: 2026-06-01 (continuation) Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Branch: `php-structural-migration` (off `main`); Go migration already merged. - -## Summary: Go, PHP, AND Java now run on the real tree-sitter structural matcher - -Go, PHP, and Java have been migrated off the regex generic-matcher -approximation onto the real tree-sitter structural matcher; `generic-language?` -(scan.ss ~809) no longer lists `go`, `php`, or `java`. tree-sitter-go/php -(ABI 15) and tree-sitter-java (ABI 14, within the runtime's 13-15 window) are -vendored + compiled into `jerboa-treesitter`. `make test` 321/321. - -### Java migration (branch `java-structural-migration`) - -Java flipped to structural with ZERO fixture regressions on the first try -(42/42 `tests/rules` `.java` fixtures still match Semgrep by the normalize -oracle) — unlike Go (which dropped 7/9→3/9 on its first flip and needed many -new structural operators). Two reasons: (1) the structural matcher is now -mature from the Go/PHP work, and (2) the `java-language?`-gated overfit handlers -(`scan-java-patterns-rule`) still fire, so they carry their specific fixtures -while structural handles general matching. Java identifiers also survive the -`$X`→`__sg_mvar_X` / `...`→`__sg_ellipsis__` text rewrite unchanged (no PHP-style -sigil problem). jerboa-treesitter additions: `vendor/grammars/java` (parser-only, -no scanner — like Go), `jt_language_java`, `ffi-language-java`, `known-languages` -+ `language-ref` java clause, smoke "parse Java" (13/13). The -`scan-java-patterns-rule` overfit handlers remain to be deleted as structural -gains their constructs (same incremental plan as Go/PHP). +Branch: `java-structural-migration` (off `main`; carries java+scala+rust); Go +and PHP migrations already merged. + +## Summary: Go, PHP, Java, Scala, Rust all run on the real structural matcher + +Five languages migrated off the regex generic-matcher approximation onto the +real tree-sitter structural matcher; `generic-language?` (scan.ss ~809) no +longer lists `go`, `php`, `java`, `scala`, or `rust`. Grammars vendored + +compiled into `jerboa-treesitter`: go/php/scala/rust (ABI 15), java (ABI 14, +within the runtime's 13-15 window); scala/rust/php carry external scanners, +go/java are parser-only. `make test` 321/321; smoke 15/15 (parse case per lang). +Per-language normalize-oracle fixture results: java 42/42, scala 18/18, +rust 11/11, php 29/29, go 16/16. + +### Java/Scala/Rust migration (branch `java-structural-migration`) + +The decisive fix was `wrapper-root-type?` (structural.ss): it now unwraps the +parse-root wrapper for java (`program`), scala (`compilation_unit`), and rust +(`source_file`) — previously only python/js/json/go were unwrapped. Without it a +bare pattern like `foo($X)` parses to `<wrapper> > … > call` but the matcher +compared the whole wrapper against each target node, so it never matched; the +languages limped along on the rule-id overfit handlers instead. With the unwrap, +`foo($X)`-style patterns match via REAL structural matching (java stayed 42/42, +confirming the prior pass had been overfit-carried). Scala/Rust bindings are +val/var/let definitions, not bare assignments, so `implicit-assignment-patterns` +gained `val/var $L = $R` (scala) and `let [mut] $L = $R` (rust) — this fixed the +Scala `taint_match_*` taint-through-`match` fixtures (source()→val→sink). The +`scan-{go,php,java,scala,rust}-patterns-rule` overfit handlers remain (gated on +the per-language predicate, still firing) and should be deleted incrementally as +structural covers their constructs. ### PHP migration (this branch) @@ -119,8 +124,8 @@ The two remaining fixtures are blocked by the same gap that limits ~12 languages, so closing them is the tip of the larger work: 1. **AST matching for the "generic" languages (the big one).** - python/js/ts/go/php/java now use the tree-sitter structural matcher. Ruby, - Rust, Scala, C#, Swift, Dart, Kotlin, C, C++ remain `generic-language?` + python/js/ts/go/php/java/scala/rust now use the tree-sitter structural + matcher. Ruby, C#, Swift, Dart, Kotlin, C, C++ remain `generic-language?` (scan.ss ~809) and use the regex-based `scan-generic-pattern`. That matcher approximates patterns with regexes and cannot express several Semgrep constructs. Real tree-sitter grammars for these languages would replace the