Update handoff: 7 languages on structural matcher; Dart/remaining documented
ober
a96b12ed2e14c68ba33c1015c6a77d44705a15f9
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,22 +1,31 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-06-01 (continuation) +Date: 2026-06-02 (continuation) Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Branch: `java-structural-migration` (off `main`; carries java+scala+rust); Go -and PHP migrations already merged. - -## Summary: Go, PHP, Java, Scala, Rust, Ruby, Kotlin on the structural matcher - -Five languages migrated off the regex generic-matcher approximation onto the -real tree-sitter structural matcher; `generic-language?` (scan.ss ~809) no -longer lists `go`, `php`, `java`, `scala`, or `rust`. Grammars vendored + -compiled into `jerboa-treesitter`: go/php/scala/rust (ABI 15), java (ABI 14, -within the runtime's 13-15 window); scala/rust/php carry external scanners, -go/java are parser-only. `make test` 321/321; smoke 15/15 (parse case per lang). -Per-language normalize-oracle fixture results: java 42/42, scala 18/18, -rust 11/11, ruby 4/4, kotlin 4/4, php 29/29, go 16/16. +Branch: `main` — all seven migrations below are merged. `dart-wip` branch holds +an unmerged Dart attempt (see Remaining). + +## Summary: 7 languages now on the real tree-sitter structural matcher + +go, php, java, scala, rust, ruby, kotlin have all been migrated off the regex +generic-matcher approximation onto the real tree-sitter structural matcher; +`generic-language?` (scan.ss ~809) no longer lists any of them. Each was gated +by the broad `tests/rules` sweep at `437 passed / 0 mismatched / 0 jerboa errors` +and merged to `main`. Grammars vendored + compiled into `jerboa-treesitter` +(go/php/scala/rust/kotlin/dart ABI 15, java/ruby ABI 14 — runtime window 13-15); +kotlin/dart were generated with tree-sitter-cli 0.26.9 (community grammars ship +no parser.c). `make test` 321/321; smoke 18/18 (parse case per lang, incl. an +unused-by-semgrep dart grammar). Per-language normalize-oracle fixture results: +java 42/42, scala 18/18, rust 11/11, ruby 4/4, kotlin 4/4, php 29/29, go 16/16. + +The decisive enabler was `wrapper-root-type?` (structural.ss): it unwraps the +per-language parse-root wrapper (program / compilation_unit / source_file) so a +bare pattern like `foo($X)` matches via REAL structural matching instead of the +rule-id overfit handlers (java had been entirely overfit-carried before this). +Languages whose bindings are declarations (scala val/var, rust let, kotlin +val/var) needed those forms added to `implicit-assignment-patterns` for taint. ### Java/Scala/Rust migration (branch `java-structural-migration`) @@ -125,9 +134,25 @@ languages, so closing them is the tip of the larger work: 1. **AST matching for the "generic" languages (the big one).** python/js/ts/go/php/java/scala/rust/ruby/kotlin now use the tree-sitter - structural matcher. C#, Swift, Dart, C, C++ remain (Dart grammar is vendored - but deferred: bare patterns parse as top-level declarations, needing Go-style - scaffolding) `generic-language?` + structural matcher. C#, Swift, Dart, Julia, C, C++ remain. + + Remaining migration candidates, by difficulty: + - **Dart** (5 fixtures) — closest. `dart-wip` branch has the working pieces: + Go-style pattern scaffolding (`dart-parse-pattern`, since bare patterns + parse as top-level declarations) and a `;`-trim (`maybe-trim-dart-statement- + semicolon`, since dart has no wrapping call node so an expression pattern + spans the whole expression_statement). All 5 broad fixtures pass there, BUT + it regresses the `tests/smoke.ss` dart taint cases (0 findings): dart taint + was generic-side (assignment propagation + the dead-catch reachability from + commit `06738f3`), and that doesn't run on the structural path yet. Finish + by porting dart taint to structural, then re-gate. + - **C# / Swift** (1 fixture each) — grammars need tree-sitter-cli generation; + low validation value. + - **C / C++** (cpp 23 fixtures) — handled via `c-like-language?` (reachability + filter etc.); migrating loses that the way Dart lost dead-catch. Hardest. + + The original concern below (regex matcher limitations) still applies only to + the languages still on `generic-language?`: `generic-language?` (scan.ss ~809) and use the regex-based `scan-generic-pattern`. That matcher approximates patterns with regexes and cannot express several Semgrep constructs. Real tree-sitter grammars for these languages would replace the