Document Semgrep Jerboa implementation plan
ober
c7d0290167a80c1b1d78ebb7346b6e9597035f39
new file mode 100644 --- /dev/null +++ b/SEMGREP_JERBOA_IMPLEMENTATION.md @@ -0,0 +1,844 @@ +# Semgrep in Jerboa: Inventory and Complete Implementation Plan + +Inventory date: 2026-06-03 + +Local repositories inspected: + +- `jerboa-semgrep`: `/Users/user/mine/jerboa-semgrep` +- Upstream Semgrep: `/Users/user/mine/semgrep`, `develop` at `238ad257b` +- Jerboa libraries: `/Users/user/mine/jerboa-*` + +This document records what the current port already contains, what the +Opus-created branches contributed, and the practical path to a complete +Semgrep-compatible implementation in Jerboa. The important premise is that this +is not a greenfield reimplementation. Upstream Semgrep is available locally and +should be treated as the executable specification: the Jerboa implementation +should mirror upstream module boundaries, data types, matching semantics, and +fixture suites wherever that is cheaper than invention. + +## Current Repository State + +The branch consolidation work is done locally. + +- Current branch: `main` +- Remote tracking state: `main` is ahead of `origin/main` by 4 commits. +- `make test`: 321/321 passing after consolidation. +- `make oracle`: 42/42 passing after consolidation. +- All local feature branches inspected are merged into `main`. + +Merged branch inventory: + +- `go-structural-migration`: moved Go toward real tree-sitter structural + matching and grew the taint oracle coverage. +- `go-remove-overfit`: removed two Go fixture-specific handlers after broader + structural behavior was available. +- `php-structural-migration`: added PHP structural matching behavior. +- `java-structural-migration`: added Java structural matching behavior, + including typed metavariable support in previous work. +- `ruby-structural-migration`: added Ruby structural matching behavior. +- `kotlin-dart-structural-migration`: added Kotlin support and started Dart. +- `dart-wip`: finished routing Dart into tree-sitter structural matching. +- `measure-real-engine`: added a measurement scaffold, now merged but disabled + by default. +- `taint-param-source`, `taint-receiver-flow`, `taint-scope-reach`, + `taint-typeclass2`, `taint-value-analysis`: taint/dataflow fixture branches + now absorbed into the combined scanner behavior. + +The current codebase is no longer just a scaffold. It has a working rule parser, +scanner, matcher, CLI, oracle tests, and generated language metadata. The +largest liability is that too much semantic behavior lives in +`src/semgrep/scan.ss`, which is currently around 34k lines. That file contains +useful behavior, but it also mixes target walking, rule formula logic, +language-specific structural matching, text fallback matching, taint heuristics, +output construction, and oracle-era compatibility logic. The completion path is +to preserve the passing behavior while moving semantics into upstream-shaped +modules. + +## Implemented Surface Area + +Current top-level modules: + +- `src/semgrep/main.ss`: entry point. +- `src/semgrep/cli.ss`: command line behavior. +- `src/semgrep/rule.ss`: rule record and core rule fields. +- `src/semgrep/rule/parse-rule.ss`: YAML/JSON Semgrep rule parsing and formula + normalization. +- `src/semgrep/scan.ss`: main scanning and matching engine. +- `src/semgrep/match/structural.ss`: structural pattern matcher. +- `src/semgrep/parse/parse-target.ss`: language routing into tree-sitter. +- `src/semgrep/fix.ss`: autofix support. +- `src/semgrep/result.ss`: result data. +- `src/semgrep/output/*`: output formats. +- `src/semgrep/lang/*`: generated and hand-written language registry data. +- `tests/smoke.ss`, `tests/oracle/*`: local smoke and upstream-derived oracle + checks. + +Current language routing: + +- Tree-sitter structural: `json`, `python`, `javascript`, `typescript` routed + through JavaScript where applicable, `go`, `php`, `java`, `scala`, `rust`, + `ruby`, `kotlin`, `dart`. +- Text/generic fallback: `generic`, `dockerfile`, `html`, `xml`, `yaml`, `c`, + `terraform`, `csharp`, `swift`, `move_on_aptos`, `julia`, `cpp`. + +Current rule/matcher support, based on the repo README, handoff notes, and code +inspection: + +- Rule config loading from YAML and JSON. +- Single `pattern`, `patterns`, `pattern-either`, and negative patterns. +- Metavariables and ellipsis. +- Basic nested formula handling. +- Rule metadata, severity, message, language list, path include/exclude, and + options. +- `focus-metavariable`. +- `metavariable-regex`, `metavariable-comparison`, `metavariable-pattern`, + `metavariable-name`, `metavariable-type`. +- Minimal analysis/taint-style behavior for selected fixtures. +- `.semgrepignore` and recursive target discovery. +- CLI output in multiple formats. +- Basic autofix and `fix-regex`. + +This is enough to keep extending with oracle tests, but not enough to call it a +complete Semgrep port. The missing pieces are not just extra languages. The +major gap is semantic architecture: upstream Semgrep lowers language-specific +parsers into `AST_generic`, normalizes code and patterns, evaluates formulas, +tracks typed/named metavariables, runs dataflow/taint over IL/CFG structures, +and emits stable output through defined interfaces. Jerboa needs those same +layers. + +## Upstream Semgrep Inventory + +The upstream checkout is large but very well organized for a port. Rough scale: + +- Source/interface files inspected under `src`, `languages`, `cli/src/semgrep`, + and `libs`: about 394k lines of OCaml/Python. +- Test files under upstream `tests`: about 5.6k files. +- Pattern fixtures alone: about 2.7k files. +- Rule fixtures: about 879 files. + +Important upstream OCaml directories: + +- `src/rule`: `Rule`, `Pattern`, `Mvar`, `Xpattern`, and rule errors. +- `src/parsing`: rule parsing, pattern parsing, target parsing, rule-file + handling, language routing, and tree-sitter integration. +- `src/ast_generic`: `AST_generic`, helpers, meta AST, and raw-tree support. +- `src/matching`: generic matching, pattern-vs-code, visitors, normalization, + and sub-AST extraction. +- `src/engine`: formula evaluation, rule matching, match environment, + metavariable regex/name/pattern/analysis, xpattern support, ReDoS logic, and + taint spec matching. +- `src/analyzing`: AST-to-IL lowering, CFG construction, constant propagation, + partial evaluation, dataflow value environments, and implicit return logic. +- `src/tainting`: OSS taint dataflow, taint specs, taint lambdas, taint + predicates, and shape-aware taint helpers. +- `src/targeting`: scanning roots, include/exclude filters, target language + guessing, and target selection. +- `src/reporting`: JSON/text output, nosemgrep handling, and metavariable + replacement. +- `src/fixing`: autofix, metavariable replacement, fixed lines, and hybrid + print. +- `src/spacegrep` and `src/aliengrep`: generic and regex-ish pattern engines + that matter for Semgrep's non-AST matching modes. + +Important upstream Python CLI directories: + +- `cli/src/semgrep/commands`: `scan`, `ci`, `login`, `publish`, `install`, + `show`, `mcp`, and related user-facing commands. +- `cli/src/semgrep/config_resolver.py`: local/remote config resolution. +- `cli/src/semgrep/core_runner.py`: orchestration around the core engine. +- `cli/src/semgrep/target_manager.py`: target discovery and filtering. +- `cli/src/semgrep/output.py` and `cli/src/semgrep/formatter/*`: text, JSON, + SARIF, GitLab, JUnit, and other output formatters. +- `cli/src/semgrep/rule*.py`, `nosemgrep.py`, `error.py`, `state.py`, + `metrics.py`, `app/*`: rule models, suppression, auth, telemetry/app + integration, and user experience behavior. + +Important upstream schema/interface files: + +- `cli/src/semgrep/semgrep_interfaces/lang.json` +- `cli/src/semgrep/semgrep_interfaces/rule_schema_v1.yaml` +- `cli/src/semgrep/semgrep_interfaces/rule_schema_v2.atd` +- `cli/src/semgrep/semgrep_interfaces/semgrep_output_v1.atd` +- `cli/src/semgrep/semgrep_interfaces/semgrep_output_v1.jsonschema` +- `interfaces/Rule_options.atd` +- `interfaces/Semgrep_prefilter.atd` +- `interfaces/Parsing_stats.atd` + +These files should be used as generated-data inputs. Hand-maintaining Jerboa +copies of language registries, output record schemas, and rule option structs is +unnecessary churn. + +## Local Jerboa Leverage + +The local Jerboa ecosystem changes the estimate. Several components needed by a +Semgrep port already exist or are close enough to adapt. + +High-value libraries: + +- `jerboa-treesitter`: static tree-sitter grammar bindings. This should remain + the parser bridge for target code and, eventually, for Semgrep pattern code. + The current port already uses it for multiple languages. Completing Semgrep + means adding the remaining grammars and a uniform conversion layer into + `AST_generic`, not hand-writing language scanners. +- `jerboa-pcre2`: PCRE2 compile/search/capture/replacement support. This should + back `metavariable-regex`, Semgrep regex mode, `fix-regex`, Aliengrep regex + fragments, and ReDoS-related checks where compatible. +- `jerboa-https`: native HTTP/TLS client. This should back registry/config/app + interactions once CLI compatibility expands beyond local file scanning. +- `jerboa-sqlite`: useful for optional caching, scan metadata, lockfiles, or + future app/session persistence. It is not required for the first complete core. +- `jerboa-code`: likely useful for source mapping, code-oriented helpers, or + editor-facing APIs, but it should not be a hard dependency until a concrete + integration point is identified. + +Core Jerboa/std leverage visible in the local tree: + +- JSON/YAML parsing for rules and output. +- JSON schema support for rule/output validation. +- Glob/path utilities for target discovery and include/exclude logic. +- Parser/PEG tooling for rule-language fragments where tree-sitter is not + enough. +- Parallel/concurrency utilities for target-level scan parallelism. +- Source/span utilities for exact locations and metavariable ranges. +- Process, filesystem, logging, crypto, and networking utilities for CLI/app + behavior. +- Test/proptest-style support for fixture and property-based validation. + +The implementation should use these instead of recreating ad hoc versions +inside `src/semgrep/scan.ss`. + +## Target Architecture + +The complete Jerboa implementation should follow upstream Semgrep's semantic +boundaries. A practical module layout: + +```text +src/semgrep/ + app/ + auth.ss + registry.ss + metrics.ss + ast/ + generic.ss + helpers.ss + raw-tree.ss + normalize.ss + source.ss + cli/ + main.ss + config.ss + scan-command.ss + ci-command.ss + dataflow/ + il.ss + ast-to-il.ss + cfg.ss + constant-propagation.ss + env.ss + partial-eval.ss + engine/ + eval-formula.ss + eval-generic.ss + match-env.ss + match-rules.ss + metavariable.ss + metavariable-analysis.ss + metavariable-name.ss + metavariable-pattern.ss + metavariable-regex.ss + redos.ss + taint-spec.ss + fix/ + autofix.ss + replacement.ss + printer.ss + lang/ + generated.ss + registry.ss + aliases.ss + match/ + structural.ss + pattern-vs-code.ss + matching-generic.ss + subast.ss + visitor.ss + parse/ + parse-rule.ss + parse-rule-formula.ss + parse-pattern.ss + parse-target.ss + tree-sitter-adapter.ss + reporting/ + json.ss + sarif.ss + text.ss + junit.ss + gitlab.ss + nosemgrep.ss + schema/ + rule-v1.ss + rule-v2.ss + output-v1.ss + options.ss + spacegrep/ + ast.ss + compile.ss + match.ss + aliengrep/ + ast.ss + compile.ss + lexer.ss + parser.ss + match.ss + taint/ + dataflow.ss + lambdas.ss + preds.ss + shape.ss + spec.ss + targeting/ + find-targets.ss + filters.ss + guess-lang.ss + ignore.ss +``` + +The exact file names can vary, but the boundaries should not. The rule parser +should not own target discovery. The structural matcher should not own taint. +The CLI should not know AST internals. This is how the port becomes finishable. + +## Smartest Porting Strategy + +### 1. Treat upstream Semgrep as the spec + +For every semantic module, keep a checked source mapping: + +- Upstream source module path. +- Jerboa target module path. +- Fixtures that cover the module. +- Known intentional deviations. +- Unsupported behavior, if any. + +This prevents vague "Semgrep-like" behavior. The goal is upstream-compatible +behavior unless a documented product decision says otherwise. + +### 2. Generate data-shaped code + +Generate Jerboa definitions from upstream interface files where the source is +schema/data rather than algorithm: + +- `lang.json` -> language registry, aliases, maturity, extensions. +- `rule_schema_v1.yaml` and `rule_schema_v2.atd` -> rule validators and record + constructors. +- `semgrep_output_v1.atd/jsonschema` -> result/output records and JSON/SARIF + compatibility checks. +- `Rule_options.atd` -> options record and defaults. + +This avoids hand-maintaining a second Semgrep schema. + +### 3. Port algorithms by semantic boundary, not by fixture + +The current port has successful fixture-driven migration, especially for the +tree-sitter structural languages. That got the repo moving, but it should not be +the finishing method. The next work should be module-by-module: + +- `AST_generic` and helpers. +- Pattern parsing and normalization. +- Generic matcher. +- Formula evaluator. +- Metavariable constraints. +- Dataflow/taint. +- Reporting and fixes. + +Fixtures should validate each ported module, not define the implementation one +case at a time. + +### 4. Build a real `AST_generic` layer + +Complete Semgrep cannot be a collection of raw tree-sitter node matchers. +Upstream normalizes language-specific syntax into a generic AST. That is where +equivalence across languages becomes possible. + +Jerboa should port: + +- Generic expression, statement, type, definition, literal, call, and pattern + node types. +- Source ranges and token text preservation. +- Helpers for identifiers, dotted names, calls, fields, attributes, indexing, + assignments, and blocks. +- Normalization passes used by matching. +- Raw tree retention for debugging and for features not yet lowered. + +Then each language adapter should convert tree-sitter nodes into this generic +AST. The current structural matchers for Go/PHP/Java/Scala/Rust/Ruby/Kotlin/Dart +can remain as compatibility scaffolding while adapters are added, but the final +engine should match against generic AST values. + +### 5. Use tree-sitter for both targets and patterns + +The current target parser already routes many languages through +`jerboa-treesitter`. Pattern parsing should use the same grammar path whenever +possible. Semgrep pattern syntax is not always a valid full program, so the +parser layer needs the same wrappers upstream uses: + +- Expression wrapper. +- Statement wrapper. +- Type wrapper. +- Field/member wrapper. +- Class/function/body wrapper. +- Language-specific fallback wrapper. + +Port upstream `Parse_pattern` and `Check_pattern` behavior rather than guessing +wrappers per fixture. + +### 6. Port matcher semantics before adding more languages + +Additional grammars are useful, but only after the generic matcher has a stable +semantic contract. The next language work should prioritize languages that +exercise different AST shapes: + +- JavaScript/TypeScript: expressions, object literals, async, imports. +- Python: indentation, decorators, calls, attributes. +- Java/Kotlin/Scala: typed metavariables, class members, generics. +- Go/Rust: blocks, receivers, references, modules, macros where possible. +- PHP/Ruby: dynamic names and language-specific call forms. + +After those are generic-AST based, add remaining languages in batches by grammar +availability and upstream fixture coverage. + +### 7. Use upstream tests as the oracle + +The local oracle suite is small relative to upstream. Completion requires +importing upstream tests systematically. + +Recommended harnesses: + +- Pattern fixture runner: `tests/patterns/**` +- Rule fixture runner: `tests/rules/**` +- Taint fixture runner: `tests/tainting_rules/**` +- Rule schema fixture runner: `tests/rule_formats/**` +- Output fixture runner: `tests/semgrep_output/**` +- CLI behavior runner for selected `osemgrep` and Python CLI fixtures. + +The harness should support: + +- Expected-pass, expected-fail, and expected-difference manifests. +- Per-language filtering. +- Per-feature filtering. +- Differential comparison against upstream Semgrep from + `/Users/user/mine/semgrep`. +- Stable JSON output normalization before comparison. + +## Complete Implementation Phases + +### Phase 0: Stabilize the Current Port + +Goal: keep existing behavior passing while making future work measurable. + +Tasks: + +- Commit the current branch consolidation, if not already pushed. +- Add this implementation plan to the repo. +- Add a branch/feature inventory note to the handoff document or link to this + file. +- Run the broad upstream-derived pattern sweeps again after the Dart merge. +- Add a single `make oracle-upstream` entry point that can run selected upstream + fixture groups by feature/language. +- Split `src/semgrep/scan.ss` without changing behavior: + - target walking and ignore filtering + - rule formula evaluation + - structural matching dispatch + - text/generic matching + - taint/dataflow heuristics + - result construction +- Keep the old public `scan-*` API stable during the split. + +Exit criteria: + +- Existing `make test` and `make oracle` pass. +- Broad fixture runner exists, even with many expected failures. +- No new fixture-specific branches are needed to add features. + +### Phase 1: Schemas, Rule Parsing, and Language Registry + +Goal: make rule/config/output data match upstream definitions. + +Tasks: + +- Generate the language registry from upstream `lang.json`. +- Generate or mechanically port rule option defaults from `Rule_options.atd`. +- Add rule schema validation against upstream `rule_schema_v1.yaml` and v2 ATD + where applicable. +- Separate parse-time rule data from engine-ready compiled rules. +- Implement a compiled rule representation that precompiles regexes, path + filters, pattern formulas, focus metavariables, and fix templates. +- Port upstream rule error categories and locations. +- Add rule-format oracle tests. + +Exit criteria: + +- Upstream rule format fixtures pass or have explicit expected-fail entries. +- Rule parse errors match upstream shape closely enough for CLI use. +- Language aliases/extensions come from generated data, not hand-maintained + lists. + +### Phase 2: Generic AST and Parser Adapters + +Goal: replace raw tree-sitter matching as the semantic center. + +Tasks: + +- Port upstream `AST_generic` core datatypes into Jerboa records. +- Port generic AST helpers needed by matching and dataflow. +- Add source span and token text handling that survives lowering. +- Create tree-sitter-to-generic adapters for existing structural languages: + - JSON + - Python + - JavaScript/TypeScript + - Go + - PHP + - Java + - Scala + - Rust + - Ruby + - Kotlin + - Dart +- Keep raw tree-sitter trees attached for debugging and incomplete language + lowering. +- Port pattern wrapper selection from upstream `Parse_pattern`. +- Add parse stats and parse failure reporting. + +Exit criteria: + +- Existing structural fixtures still pass through the generic AST path. +- Pattern parsing supports expression/statement/type/member/body contexts. +- Parse failures produce upstream-compatible locations and reasons. + +### Phase 3: Matcher and Formula Engine + +Goal: make Semgrep pattern semantics generic and feature-complete. + +Tasks: + +- Port `Pattern_vs_code`. +- Port `Matching_generic`. +- Port matching visitors and sub-AST extraction. +- Port normalization passes from `Normalize_generic`. +- Port formula evaluation from `Eval_generic` and related engine modules: + - `pattern` + - `patterns` + - `pattern-either` + - `pattern-not` + - `pattern-inside` + - `pattern-not-inside` + - `metavariable-*` operators + - `focus-metavariable` +- Port match environments and metavariable binding/range handling. +- Port xpattern support as needed for regex/generic matching interop. +- Use `jerboa-pcre2` for PCRE-compatible constraints. +- Add ReDoS guardrails based on upstream behavior where supported. + +Exit criteria: + +- Pattern fixtures pass for the first batch of generic-AST languages. +- Metavariable constraints match upstream results on rule fixtures. +- Existing heuristic handlers can be deleted or demoted to compatibility shims. + +### Phase 4: Spacegrep, Aliengrep, and Generic Mode + +Goal: complete Semgrep's non-AST pattern modes. + +Tasks: + +- Port `spacegrep` AST, parser/compiler, and matcher. +- Port `aliengrep` pattern AST, lexer/parser/compiler, and matcher. +- Back regex operations with `jerboa-pcre2`. +- Rework current text/generic fallback to use these engines. +- Add generic-language fixtures for Dockerfile, YAML, XML/HTML, Terraform, C, + C++, C#, Swift, Julia, and Move while their full AST adapters are pending. + +Exit criteria: + +- Generic mode behavior is Semgrep-compatible instead of ad hoc text matching. +- `metavariable-regex` and `fix-regex` share the same PCRE backend. + +### Phase 5: Dataflow, Taint, and Typed Analysis + +Goal: replace fixture-level taint handling with upstream-style analysis. + +Tasks: + +- Port IL datatypes. +- Port AST-to-IL lowering for the initial language set. +- Port CFG construction. +- Port constant propagation and partial evaluation. +- Port value environments. +- Port OSS taint dataflow: + - sources + - sinks + - sanitizers + - propagators + - control-flow behavior + - implicit returns + - field/index shape sensitivity where upstream supports it +- Port metavariable analysis constraints that depend on dataflow/typing. +- Preserve current passing taint fixtures during migration. + +Exit criteria: + +- Upstream taint fixtures pass for at least the core language set. +- Current taint branches' behavior is explained by the real dataflow engine, + not by special-case matching. +- Type/name constraints use the same shared environment model as taint and + matcher constraints. + +### Phase 6: Targeting, Ignore Semantics, Output, Fixes + +Goal: match Semgrep user-visible behavior outside the matcher core. + +Tasks: + +- Port target discovery and filtering from `targeting`. +- Match `.semgrepignore`, include/exclude, max-target-size, symlink, and + language-guessing semantics. +- Port nosemgrep suppression behavior. +- Port output data structures from `semgrep_output_v1`. +- Implement output formatters: + - text + - JSON + - SARIF + - JUnit + - GitLab + - any additional upstream-supported stable formats +- Port autofix and metavariable replacement semantics. +- Port fixed-line behavior and hybrid print enough for compatible patches. + +Exit criteria: + +- CLI output compares cleanly against upstream for representative fixtures. +- Autofix fixtures apply expected changes. +- Ignore/nosemgrep behavior is tested with upstream cases. + +### Phase 7: CLI, Registry, App, and Packaging + +Goal: provide a Semgrep-compatible tool, not only a library. + +Tasks: + +- Mirror the core `semgrep scan` command behavior. +- Add config resolution for local files, directories, URLs, and registry + references. +- Use `jerboa-https` for authenticated and unauthenticated registry/app calls. +- Port login/auth/token handling as needed. +- Port CI behavior where practical. +- Add metrics/app integration only after local scan compatibility is stable. +- Add parallel target scanning using Jerboa concurrency primitives. +- Add caching only if profiling shows a need. +- Package the executable and generated grammar/schema assets in a reproducible + way. + +Exit criteria: + +- Local scan behavior is compatible enough to substitute for Semgrep in normal + rule development. +- Registry config resolution works. +- CI behavior is documented and tested. + +## Language Completion Plan + +Language coverage should be added in the order that maximizes semantic coverage, +not raw language count. + +Batch 1: stabilize existing tree-sitter structural languages on generic AST. + +- JSON +- Python +- JavaScript +- TypeScript +- Go +- PHP +- Java +- Scala +- Rust +- Ruby +- Kotlin +- Dart + +Batch 2: turn current generic fallbacks into real adapters where tree-sitter +grammars are available and fixtures are substantial. + +- C +- C++ +- C# +- Swift +- Terraform +- YAML +- HTML/XML +- Dockerfile +- Julia +- Move on Aptos + +Batch 3: add remaining upstream languages by grammar availability and fixture +coverage. + +- Bash +- Cairo +- Circom +- FGA +- Hack +- Jsonnet +- Lisp +- Lua +- Move on Sui +- OCaml +- PromQL +- Protobuf +- Python variants where needed +- QL +- R +- Regexp +- Solidity + +For each language adapter: + +- Parse target code through `jerboa-treesitter`. +- Parse patterns through the Semgrep wrapper strategy. +- Lower target and pattern trees into `AST_generic`. +- Run the shared matcher. +- Run upstream pattern/rule fixtures for that language. +- Add only language-local lowering code, not matcher special cases. + +## Test Strategy + +The current smoke/oracle tests are necessary but too small. The complete port +needs a layered test strategy. + +Unit tests: + +- Rule parsing and validation. +- Language registry generation. +- Pattern wrapper selection. +- Generic AST lowering for small source snippets. +- Matcher primitives. +- Metavariable binding and constraint evaluation. +- Taint/dataflow transfer functions. +- Output serialization. + +Fixture tests: + +- Imported upstream `tests/patterns`. +- Imported upstream `tests/rules`. +- Imported upstream `tests/tainting_rules`. +- Imported upstream `tests/rule_formats`. +- Imported upstream `tests/semgrep_output`. +- Existing local oracle tests. + +Differential tests: + +- Run upstream Semgrep from `/Users/user/mine/semgrep`. +- Run Jerboa Semgrep on the same config/target. +- Normalize nondeterministic fields such as timing, path roots, version strings, + and ordering where upstream allows it. +- Compare result IDs, paths, ranges, metavariables, messages, severities, + metadata, and fixes. + +Regression tests: + +- Every retired special handler should leave behind a fixture proving the + generic engine still covers that behavior. +- Every bug found during porting should become either a direct test or an + upstream fixture inclusion. + +Performance tests: + +- Use representative repositories, not only tiny fixtures. +- Measure parsing, matching, taint, output, and total wall time separately. +- Keep the merged `measure-real-engine` scaffold disabled by default until it is + wired into repeatable benchmarks. + +## Immediate Next Work + +The fastest useful sequence from the current state: + +1. Run the broad upstream pattern sweeps after the Dart merge and record the + current expected-fail baseline. +2. Split `src/semgrep/scan.ss` into behavior-preserving modules. Do not change + semantics during the split. +3. Generate the language registry and rule/output schema records from upstream + interfaces. +4. Port `AST_generic` records and helpers into `src/semgrep/ast`. +5. Add tree-sitter-to-generic lowering for the languages already passing + structural tests. +6. Re-route those languages through generic AST matching behind a feature flag + or per-language switch. +7. Port `Pattern_vs_code`, `Matching_generic`, and normalization until those + languages pass through the shared engine. +8. Port Spacegrep/Aliengrep using `jerboa-pcre2` and replace the current generic + fallback. +9. Port IL/CFG/dataflow and replace current taint heuristics. +10. Expand CLI/output/fix compatibility using generated schema records. + +This path is faster than estimating a from-scratch engine because it reuses: + +- Upstream OCaml/Python as a module-by-module specification. +- Upstream fixtures as a differential oracle. +- `jerboa-treesitter` for parsing. +- `jerboa-pcre2` for regex semantics. +- Jerboa stdlib support for JSON/YAML/schema/glob/source/parallel/networking. +- The current port's working CLI, config parser, matcher scaffolding, and tests. + +## Risks and Controls + +Risk: the current monolithic scanner hides fixture-specific logic. + +Control: split it behavior-preservingly, then delete special cases only after +generic-engine fixtures pass. + +Risk: raw tree-sitter matching keeps diverging from Semgrep. + +Control: build `AST_generic` and make tree-sitter adapters language-local. + +Risk: hand-maintained schemas drift from upstream. + +Control: generate schema/language/output code from upstream files. + +Risk: CLI compatibility consumes time before core correctness. + +Control: keep CLI work behind core matcher/dataflow milestones, except for +output needed by differential tests. + +Risk: full language coverage delays correctness. + +Control: finish shared semantics on a small diverse language set before adding +new adapters. + +Risk: upstream Semgrep changes while the port progresses. + +Control: pin the inspected upstream revision in test manifests and periodically +refresh with an explicit compatibility update. + +## Definition of Complete + +The Jerboa implementation can be called a complete Semgrep implementation when: + +- Rule schema parsing and validation are generated from or demonstrably aligned + with upstream schemas. +- Language aliases/extensions are generated from upstream language data. +- Target parsing lowers through a generic AST layer for the supported languages. +- Pattern parsing uses upstream-equivalent wrapper/context behavior. +- The matcher implements Semgrep formula and metavariable semantics through + shared generic modules. +- Generic/regex modes use Spacegrep/Aliengrep-compatible engines. +- Taint/dataflow behavior is implemented through IL/CFG/dataflow modules, not + fixture-specific handlers. +- Target discovery, ignore rules, nosemgrep, output formats, and autofix are + covered by upstream-derived tests. +- Differential tests against the local upstream Semgrep checkout pass for the + selected compatibility suite. +- Unsupported upstream features are listed in a manifest with exact reasons, + not silently approximated. + +The current repo has a valuable working base. The smart move is not to discard +it, and not to keep growing it as a monolith. Preserve its passing behavior, +split the responsibilities, generate schema/data from upstream, port the core +upstream semantic modules into Jerboa, and let upstream fixtures drive the rest.