Parse Go via tree-sitter (AST-matching foundation)
ober
19e0c530c0eec6f0616380681683f7bb878da998
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -44,6 +44,32 @@ languages, so closing them is the tip of the larger work: constructs. Real tree-sitter grammars for these languages would replace the approximation wholesale. + **Go AST foundation laid (this run).** `tree-sitter-go` (ABI 15, no external + scanner) is now vendored and compiled into `jerboa-treesitter` + (`vendor/grammars/go`, `native/jerboa_tree_sitter.c` `jt_language_go`, + `Makefile`, `src/tree-sitter/{ffi,tree-sitter}.ss` — `known-languages` now + `(json python javascript go)`; smoke 11/11 incl. "parse Go"). jerboa-semgrep + `parse-target.ss` maps `go -> 'go`, so `parse-target-string` parses Go. The + metavar rewrite (`$X -> __sg_mvar_X`, `... -> __sg_ellipsis__`) yields valid + Go identifiers, so structural matching of Go patterns is mechanically sound. + NOTE: jerboa-treesitter is not a git repo, so those edits are on disk only. + + **But routing Go to structural is NOT a flag flip — it regresses.** Removing + `go` from `generic-language?` (the only switch) routes Go through the + structural matcher everywhere. Measured: the go taint dir drops 7/9 -> 3/9, + because the Go taint machinery is generic-side and the structural path does + not see it — the for-range / `:=` full-RHS propagators and the c-like + reachability filter (continue/goto/make) all operate around the regex + matcher's findings. (Broad go/* could not be measured reliably in isolation; + the real signal is the taint dir.) The flip was reverted; baseline restored + (go taint 7/9, smoke 321/321). **Migration plan:** before flipping, reproduce + on the structural path (a) the Go taint propagators, (b) the c-like + reachability filter, and (c) real structural matching for the constructs the + ~25 `go-rule-id?` handlers fake (int_binop, struct_tags, typed_metavar, …), + keeping the oracle green at each step. Then flip `generic-language?` and + delete the handlers. This is the bulk of the remaining work, now unblocked by + the grammar being available. + 2. **The generic matcher binds a metavar to its FIRST value, then only finds same-value occurrences.** Verified: with `pattern: $R.File` over a file where `asrc.File` and `bsrc.File` both appear, only `asrc.File` matches (the regex --- a/lib/semgrep/parse/parse-target.sls +++ b/lib/semgrep/parse/parse-target.sls @@ -23,6 +23,7 @@ [(string=? canonical "python") 'python] [(string=? canonical "javascript") 'javascript] [(string=? canonical "typescript") 'javascript] + [(string=? canonical "go") 'go] [else (error 'parse-target-string "unsupported language in current MVP" --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -2,12 +2,12 @@ ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") - ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") + ("src/semgrep/parse/parse-target.ss" . "B1616180DE7038ED") ("src/semgrep/scan.ss" . "621FB6C8F6C1151A") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/match/structural.ss" . "6FE77014EE9FDCE4") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/parse/parse-target.ss +++ b/src/semgrep/parse/parse-target.ss @@ -15,6 +15,7 @@ [(string=? canonical "python") 'python] [(string=? canonical "javascript") 'javascript] [(string=? canonical "typescript") 'javascript] + [(string=? canonical "go") 'go] [else (error 'parse-target-string "unsupported language in current MVP" language)])))