Remove obsolete Semgrep port plan
ober
d9526de33d82aa5bb236662c029cd4cc5386d33f
deleted file mode 100644 --- a/what.md +++ /dev/null @@ -1,1597 +0,0 @@ -# Semgrep to Pure Jerboa Port Plan - -## Goal - -Replace Semgrep's OCaml and Python implementation with pure Jerboa `.ss` -modules. - -Constraints: - -- No OCaml implementation remains. -- No Python implementation remains. -- No Chez `.sls` files. -- Jerboa source should be ordinary `.ss` modules. -- Existing Semgrep rules must continue to work unchanged. -- Existing Semgrep output formats must remain compatible. -- The result should be easy to embed in other Jerboa applications. - -The port should not be a scanner inspired by Semgrep. It should be a -compatibility-preserving Semgrep implementation in Jerboa. - -The hardest requirement is rule compatibility. Semgrep has a large existing -rule corpus, and the value of the port depends on those rules continuing to -work. That means parser behavior, matcher behavior, rule validation, taint -semantics, target selection, output schemas, and autofix behavior all need -compatibility gates. - -## Repository Review Summary - -The current Semgrep repository is a large mixed OCaml/Python monorepo. - -The important implementation areas are: - -- `src/`: OCaml core engine, matching, parsing, tainting, targeting, reporting, - `osemgrep`, CLI internals, and scan orchestration. -- `languages/`: language parser integrations and AST adapters. -- `libs/`: shared OCaml libraries such as commons, globbing, gitignore, - Jsonnet, networking, parallelism, and telemetry. -- `cli/src/semgrep`: Python CLI, config resolution, target manager, scan - runner, formatters, app integration, MCP, metrics, and auth. -- `cli/src/semgrep/semgrep_interfaces`: generated schemas shared by Python, - OCaml, TypeScript, JSON Schema, and protobuf. -- `tests/`: the main compatibility corpus. -- `perf/`: performance-oriented rules and benchmarks. - -Approximate implementation size across the major OCaml/Python areas is several -hundred thousand lines. The largest functional areas are the Python CLI, OCaml -engine, language parsers, `osemgrep`, shared OCaml libraries, matching, -tainting, and parsing. - -The important conclusion is that this is not mainly a CLI rewrite. The CLI is -large, but the core risk is preserving language parsing, rule parsing, matching, -taint/dataflow, target discovery, and output compatibility. - -## Reviewed Checkout - -This plan is grounded in the sibling Semgrep checkout at -`/Users/user/mine/semgrep`, branch `develop`, revision `238ad257b`. - -Observed implementation surface: - -- `src/`: 360 OCaml `.ml`/`.mli` files. -- `languages/`: 342 files, including language ASTs, tree-sitter adapters, - legacy Menhir parsers, and generic AST conversion modules. -- `cli/src/semgrep`: 108 Python files. -- `tests/`: 5596 files across rule, parser, taint, output, target-selection, - and CLI behavior. - -The primary generated schema inputs are in -`cli/src/semgrep/semgrep_interfaces/`: - -- `semgrep_output_v1.atd` -- `rule_schema_v1.yaml` -- `rule_schema_v2.atd` -- `lang.json` - -The top-level `interfaces/` directory contains additional ATD files such as -`Parsing_stats.atd` and `Rule_options.atd`, but it is not the main home for the -rule and output schemas. - -Existing test entry points worth preserving as oracle drivers: - -- `make core-test` from the Semgrep root for semgrep-core tests. -- `./test -s PATTERN` after `make build-core-test` for focused core tests. -- `cd cli && make test` for Python CLI unit and end-to-end snapshot tests. -- `cd cli && pytest --run-only-snapshots ...` for focused CLI snapshot oracle - runs. - -## Current Jerboa Port Status - -The current repository now contains a pure Jerboa `.ss` MVP with a thin CLI, -generated Chez `.sls` output under `lib/`, smoke tests, and a Semgrep oracle -harness. - -Implemented compatibility surface: - -- YAML rule config parsing for files and config directories, including - Semgrep-style aligned block sequences and their continuation blocks under - mapping keys, comments between keys and sequences, and plain scalar - continuation lines, including quoted continuations that contain colons and - quoted scalar continuations inside sequence items. -- Generated language registry data from upstream `lang.json`. -- Known-language validation with runtime aliases for Python and JavaScript. -- Search mode for `pattern`, `pattern-regex`, `pattern-either`, and nested - `patterns`. -- Python, JavaScript, and JSON target parsing through the current - Jerboa/tree-sitter bridge. -- TypeScript and TSX targets use the JavaScript parser as a compatibility - fallback for JS-compatible syntax. This removes hard failures for basic TS - scans but is not full TypeScript parser parity. -- Generic targets have a small text matcher for `pattern`, nested `patterns`, - plain ellipsis, `$X`, `$...X`, repeated metavariable consistency, and - comment-tolerant whitespace separators. This supports the current upstream - generic/txt hard-failure slice, but it is not full Spacegrep/Aliengrep - parity. -- Structural metavariables, repeated metavariable equality, ellipsis, and - metavariable ellipsis matching in the supported languages. -- Numeric-looking pattern scalars from YAML are normalized to strings before - matching. -- Formula filters for `pattern-not`, `pattern-inside`, - `pattern-not-inside`, `metavariable-regex`, `metavariable-pattern`, - `metavariable-type`, `metavariable-name`, `metavariable-analysis`, and - `metavariable-comparison`. -- `metavariable-type` parses `type`, `types`, and `language`, with literal - binding inference for strings, numbers, booleans, `None`/`null`, arrays, and - objects in supported parsed languages. It also recognizes simple Python and - TypeScript annotation sites such as `a: str` and `res: Response`, which is - enough for the current upstream `metavariable-type` smoke slice. Full - semantic type analysis remains a future parser/analyzer task. -- `metavariable-name` and `semgrep-internal-metavariable-name` parse - `module`, `modules`, `fqn`, and `fqns`, with local import and `require()` - resolution for Python and JavaScript targets. Pro-only `kind` constraints and - full semantic name resolution remain future analyzer work. -- `metavariable-analysis` supports the OSS-style `entropy` analyzer and a - practical ReDoS nested-quantifier check for string metavariables. -- `metavariable-comparison` supports comparisons across multiple bound - metavariables, optional `metavariable`, `strip`, numeric parsing, string - membership, boolean `and`/`or`/`not`, and common conversion/case functions - such as `str`, `int`, `float`, `lower`, `upper`, and `len`. -- `metavariable-comparison` also supports common arithmetic and bitwise - operators used in upstream rules, including `+`, `-`, `*`, `/`, `%`, `**`, - `&`, `|`, `^`, and unary `~`. Numeric equality treats integer and floating - results with the same numeric value as equal. -- `metavariable-pattern` supports direct `pattern`, `pattern-regex`, - `pattern-either`, and nested `patterns` formulas against the bound text, - including negative-only regex filters and string-literal content matching - without enclosing quotes. -- `metavariable-pattern` now preserves relevant outer metavariable bindings - when an inner formula references them, so inner `pattern-not` clauses can - distinguish `$A + $B` from `$A + $A` and can share bindings through - `pattern-either`/nested formulas. -- `metavariable-pattern` keeps inner structural and regex capture bindings - local to the bound-text formula. Inner structural bindings can satisfy - sibling inner clauses, but they are not exported into later outer formula - clauses or focused findings; regex captures from inner `pattern-regex` - filters are likewise treated as local filter state for Semgrep OSS parity. -- `metavariable-pattern` can persist selected inner structural bindings for a - later `focus-metavariable` when no intervening outer filter references those - names. This covers focused persistent-metavariable-pattern behavior while - preserving local-only inner bindings for later filtering clauses. -- Regex capture bindings from `pattern-regex`, including named captures and - numeric `$1`-style captures. -- `pattern-regex` treats `^` and `$` as line anchors for Semgrep-style - multiline matching. -- Regex-only rules can run without an AST parser, including `generic` and - `regex` `pattern-regex` formulas with overlapping `pattern-not-regex` - filters. -- Regex capture bindings from `metavariable-regex`, including propagation into - later formula clauses. -- Leading `(?i)` case-insensitive regex flags are accepted for Semgrep-style - `pattern-regex` and metavariable regex compatibility. -- Formula-local `focus-metavariable` for capture-focused findings. -- Formula-local multi-focus behavior for single-statement unions and - multi-statement range intersections. -- `pattern-either` supports formula-style alternatives such as - `pattern-inside` and nested `patterns`, and seed selection prefers ordinary - positive clauses when an either clause is purely contextual. -- `pattern-inside` formula clauses propagate bindings from the smallest - enclosing positive match into later clauses and focus handling. -- `patterns` formulas can use `pattern-inside` as the seed when no ordinary - positive pattern is present, matching focused source/sanitizer specs. -- `semgrep-internal-pattern-anywhere` is supported as a file-wide formula - precondition with compatible metavariable binding merge. -- `as` metavariable bindings on positive pattern entries. -- Old `match: {all, where}` rules are mapped into `patterns` formulas for the - common `as` plus metavariable-pattern shape. -- Rule-level `focus-metavariable` fields parse, while range narrowing is - currently applied only by formula-local focus clauses. This matches the - current OSS oracle slice, where top-level focus fields do not narrow the - normalized finding range. -- Basic taint mode with sources, sinks, sanitizers, formula entries, label - `requires` expressions, and direct `pattern-propagators` from one - metavariable binding to another, including output labels and - `replace-labels`. -- Taint source `requires` can derive new labels from prior compatible taint, - and sink `requires` is evaluated against the compatible label set reaching - that sink. -- Taint labels from token-propagated values can feed enclosing argument - expressions, allowing calls such as `q(a)` to combine argument taint with - call-source labels. -- Taint sinks honor explicit `exact: false` so nested tainted expressions can - trigger wider sink patterns such as `sink(...)`. -- Taint source and sanitizer specs parse `exact`, `by-side-effect`, and - `control`; `exact` is enforced for sources and sanitizers so whole-expression - taint does not implicitly taint or sanitize nested metavariables. -- Focused by-side-effect sanitizers use token/source compatibility so - sanitizer variable names do not have to match source metavariable names. -- Taint mode includes built-in assignment propagation for simple Python - assignments and JavaScript `=`, `const`, `let`, and `var` declarations. -- Focused `by-side-effect` taint sources and assignment-propagated values can - reach wildcard sinks such as `sink(...)` through token-level variable - matching. -- Token-level taint propagation is constrained by simple Python `def` function - scopes to avoid leaking same-named locals across functions. -- Taint propagators honor explicit `by-side-effect: false` as expression-local - propagation, including label preconditions and focused `from`/`to` bindings. -- Path include/exclude filters, `.semgrepignore` negation handling, and - `nosemgrep` suppression. -- JSON, SARIF, and text output. -- `fix` and `fix-regex` rendering, including count-limited regex fixes. -- CLI scanning for files, directories, and stdin, including fallback language - inference from single-language configs for extensionless regex/generic-style - targets. -- Additional upstream-compatibility fallbacks cover currently observed rule - idioms for JSON key metavariable unification, JavaScript package dependency - fields, JavaScript constructor-parameter type matching in decorated classes, - `new Function(...); ... func();` negation, constant string concatenation for - metavariable-regex, TypeScript object intersection type properties, - TypeScript request-param-to-`$where` deep metavariable flow, constant - template-string suppression, Python `and/or` string constant propagation, - Python loop-`else` constant propagation, Python bare `$X` expression/range - pruning, Python constant f-string calls, JSX `style={...}` attributes, - JavaScript object/record patterns with explicit ellipsis and spread syntax, - and one jQuery callback taint sink shape from the upstream regression corpus. - -Compatibility gates in this repository: - -- `make test` runs the Jerboa smoke suite. It currently passes: 162 tests, - 162 passed, 0 failed. -- `make oracle` compares normalized findings against a current Semgrep binary - when `SEMGREP_CURRENT` is set or `/Users/user/mine/semgrep/bin/semgrep` - exists. -- The local oracle currently passes against packaged Semgrep 1.164.0 installed - at `/Users/user/.local/bin/semgrep`: 42 cases passed, 0 failed. The sibling - Semgrep checkout could not be built locally because `dune`/`opam` are not - installed, so this is a packaged-Semgrep compatibility baseline rather than - a fresh build of `/Users/user/mine/semgrep`. -- The first 120 upstream rule/target pairs from the sibling Semgrep checkout - now have no Jerboa mismatches: 118 passed, 0 mismatched, 0 Jerboa errors, - and 2 current-Semgrep errors from `/Users/user/.local/bin/semgrep`. This is - the strongest current parity gate and should be expanded in batches. -- Local upstream hard-failure sweeps currently parse 440 YAML rule configs - under `/Users/user/mine/semgrep/tests/rules/`, skipping 4 `.test.yaml` - target fixtures, with 0 parse failures. They also scan the currently wired - upstream target pairs without hard scanner failures: 134 YAML/Python pairs, - 83 YAML/JavaScript pairs, 14 YAML/TypeScript pairs using the JavaScript - fallback, 1 YAML/JSON pair, and 9 generic/txt pairs. These sweeps prove - crash/parser coverage only; finding parity still requires the Semgrep oracle - binary. -- Oracle cases currently cover baseline Python/JavaScript matching, regex - matching, structural metavariables, nested formulas, `fix-regex`, regex - capture groups, `metavariable-regex` capture propagation, richer - `metavariable-comparison` expressions, `metavariable-name`, - `metavariable-analysis`, focus behavior, and taint labels, exact matching, - non-exact sinks, source `requires`, by-side-effect sources, propagators, - assignment propagation, and `pattern-not-regex`. - -Known next parity gaps: - -- The JSON output shape is still a normalized MVP, not full Semgrep JSON v1. -- Many rule operators are still missing or partial, including full semantic - metavariable type/name checks, analyzers, extract mode, join mode, - dependency/SCA mode, advanced taint side-effect sanitizers, precise source - `only` behavior, control-flow behavior, reassignment invalidation, - field/index-sensitive aliases, and propagator options beyond labels. -- Target discovery is not yet Semgrep-equivalent for git-aware scans, baseline - scans, binary detection, symlinks, large-file behavior, and full ignore - precedence. -- Language support is limited to the parsers currently wired through - Jerboa/tree-sitter. TypeScript currently uses a JavaScript-parser fallback, - and the generic matcher is currently a regex-backed textual subset rather - than Semgrep's full OCaml generic AST, Spacegrep, or Aliengrep behavior. -- CLI parity is still shallow: config resolution, registry/app integration, - auth, metrics, timeout handling, and snapshot-compatible error reporting - remain open. - -## Existing Architecture to Preserve - -### Rule Layer - -The central rule model lives around `src/rule/Rule.ml`. - -Important rule capabilities: - -- Search mode. -- Taint mode. -- Extract mode. -- Steps mode. -- SCA/dependency mode. -- Join mode. -- `pattern`. -- `patterns`. -- `pattern-either`. -- `pattern-not`. -- `pattern-inside`. -- `pattern-not-inside`. -- `pattern-regex`. -- `metavariable-regex`. -- `metavariable-pattern`. -- `metavariable-comparison`. -- `metavariable-type`. -- `metavariable-name`. -- `metavariable-analysis`. -- `focus-metavariable`. -- `as` metavariables. -- `fix`. -- `fix-regex`. -- Validators. -- Paths include/exclude filters. -- Metadata. -- Minimum and maximum Semgrep versions. - -Taint mode includes: - -- Sources. -- Sinks. -- Sanitizers. -- Propagators. -- Built-in assignment propagation for common local variable assignments. -- Labels. -- `requires`. -- `by-side-effect`. -- `exact`. -- `control`. -- Sink-at-exit behavior. -- Preconditions. - -The port needs a Jerboa rule IR that is intentionally close to the existing -Semgrep rule IR. This is not the right layer to simplify semantics. - -### Rule Parsing - -Rule parsing is centered around `src/parsing/Parse_rule.ml`. - -The parser accepts YAML, JSON, and Jsonnet rule files. It preserves source -positions for useful error reporting and for downstream behavior. It also -supports old rule syntax. The Jerboa port must preserve this, because existing -rule packs use a mix of old and newer syntax. - -Jsonnet support matters because some Semgrep configs are generated or import -other files. The current implementation has special handling for imports, -registry-style configs, and YAML imports converted into Jsonnet-compatible AST -forms. - -### Language Parsing - -Target parsing is centered around `src/parsing/Parse_target.ml`, -`src/parsing/Parse_pattern.ml`, and `src/parsing/Pfff_or_tree_sitter.ml`. - -Semgrep supports many languages. The current implementation uses a combination -of tree-sitter grammars, legacy parser infrastructure, and language-specific AST -normalization. - -The Jerboa port should not begin by handwriting every parser from scratch. That -would make compatibility unrealistic. The practical plan is to preserve -tree-sitter grammar behavior where possible and replace the OCaml adapter and -normalization layers with Jerboa `.ss` modules. - -The current parser stack is mixed and should be migrated in tiers: - -- Direct tree-sitter-to-generic languages: many newer languages already parse - directly to the generic AST or a very thin intermediate form. These are good - early candidates once the tree-sitter FFI and generic AST constructors exist. -- Tree-sitter plus language-specific generic conversion: Java, JavaScript, - TypeScript, Ruby, Bash, Dockerfile, Jsonnet, Terraform, QL, and similar - languages still have important conversion modules that must be ported, not - skipped. -- Legacy parser fallback languages: Python, Go, Scala, OCaml, PHP, and parts of - C/C++ still preserve legacy parser behavior or pattern parser behavior. These - need explicit fallback and mismatch tests. -- External/proprietary parser hooks: Apex, Elixir, Gosu, Powershell, and - optional C# parser behavior need extension points even if the OSS Jerboa port - cannot implement the proprietary parser itself. - -The port needs equivalent handling for: - -- Target parsing. -- Pattern parsing. -- Partial parsing. -- Parser fallback. -- Parse recovery. -- Serious parse errors. -- Language-specific normalization. -- Mapping parser nodes into the generic AST. -- Mapping source ranges and tokens correctly. - -### Generic AST - -The generic AST is one of the main compatibility surfaces. - -It carries: - -- Expressions. -- Statements. -- Declarations. -- Types. -- Patterns. -- Names. -- Attributes. -- Fields. -- XML/YAML/JSON-like nodes. -- Tokens. -- Ranges. -- Origin information. -- Language-specific details. -- Name and type annotations. - -The matcher depends on the exact shape of this tree. The Jerboa port should -therefore recreate a compatible generic AST first, then optimize or refactor -later only after compatibility tests are strong. - -### Matching Engine - -The core matcher is around: - -- `src/engine/Match_rules.ml` -- `src/engine/Match_search_mode.ml` -- `src/matching/Match_patterns.ml` -- `src/matching/Matching_generic.mli` -- `src/matching/Pattern_vs_code.mli` - -Important matcher behavior: - -- Pattern-vs-code matching. -- Metavariable binding. -- Bound metavariable equality. -- Ellipsis matching. -- Metavariable ellipsis matching. -- List matching. -- Associative/commutative matching. -- Deep expression matching. -- Statement matching. -- Type matching. -- Name matching. -- Attribute matching. -- XML/YAML/JSON matching. -- Regex matching. -- Formula evaluation. -- Match set intersection, union, and subtraction. -- `pattern-inside`. -- `pattern-not`. -- `pattern-not-inside`. -- `focus-metavariable`. -- `as` metavariables. -- Range tracking. -- Token tracking. -- Per-rule timeout behavior. -- Rule grouping and prefiltering. - -This layer is the critical path for existing rule corpus compatibility. - -### Semantic Analysis and Taint - -Semgrep's taint and dataflow path uses: - -- AST-to-IL lowering. -- Control-flow graph construction. -- Generic dataflow fixpoint analysis. -- Name analysis. -- Type propagation. -- Constant propagation. -- Taint source/sink/sanitizer/propagator logic. - -The Jerboa port should preserve the current layering: - -1. Parse source to generic AST. -2. Normalize and annotate. -3. Lower to IL where needed. -4. Build CFG. -5. Run dataflow. -6. Feed findings back through source origins. - -Taint compatibility is especially important because many security rules depend -on subtle behavior around sanitizers, side effects, labels, propagators, and -field sensitivity. - -### Target Selection - -Targeting is not incidental. Many Semgrep users depend on exact include/exclude -behavior. - -The Jerboa port needs equivalent behavior for: - -- Language guessing by extension, shebang, and content. -- Explicit target paths. -- Directory traversal. -- Symlink handling. -- `.semgrepignore`. -- Gitignore-compatible matching. -- Built-in ignore defaults. -- Include/exclude flags. -- Max target size. -- Baseline scans. -- Diff scans. -- Project roots. -- Git repository discovery. -- Stdin scanning. -- Dependency and lockfile discovery. - -Target selection should be ported before claiming rule parity, because a rule -that matches correctly on one file is still useless if the scanner chooses the -wrong file set. - -### Output and Schemas - -The key schema is `semgrep_output_v1.atd`. - -The Jerboa implementation must preserve: - -- JSON v1 output. -- SARIF output. -- Text output. -- GitLab output. -- JUnit output. -- Skipped target reporting. -- Parse error reporting. -- Match ordering. -- Severity filtering. -- Exit codes. -- Metrics fields where applicable. -- Autofix output. -- `nosemgrep` behavior. - -Downstream tools often depend on Semgrep output shape. Output compatibility -should be tested with snapshots. - -### Python CLI Functionality - -The Python CLI still contains important behavior even though `osemgrep` has -already moved substantial scan orchestration into OCaml. - -Important Python areas include: - -- `run_scan.py` -- `core_runner.py` -- `config_resolver.py` -- `target_manager.py` -- `output.py` -- `commands/scan.py` -- `commands/ci.py` -- `mcp/server.py` -- formatters -- metrics -- auth -- app integration - -The Jerboa port should use `osemgrep` as the primary architecture guide, but it -must still audit Python for behavior that has not yet been ported into OCaml. - -## Porting Surface Map - -The port should track Semgrep source-of-truth areas explicitly. This table is a -working map from current implementation surface to the proposed Jerboa modules. - -| Semgrep surface | Main current files/directories | Jerboa destination | -| --- | --- | --- | -| Rule schema and output schema | `cli/src/semgrep/semgrep_interfaces/*.atd`, `rule_schema_v1.yaml`, `lang.json` | `semgrep/schema/*.ss` | -| Rule IR | `src/rule/Rule.ml`, `src/rule/Xpattern.ml`, `src/rule/Mini_rule.ml` | `semgrep/rule/rule.ss`, `xpattern.ss` | -| Rule parsing and validation | `src/parsing/Parse_rule.ml`, `Parse_rule_helpers.ml`, `src/metachecking/` | `semgrep/rule/parse-rule.ss`, `validate-rule.ss` | -| Config resolution | `src/osemgrep/core/Rules_config.ml`, `Rules_source.ml`, `cli/src/semgrep/config_resolver.py` | `semgrep/app/config-fetch.ss`, `semgrep/cli/config.ss` | -| Language registry | `lang.json`, `cli/src/semgrep/semgrep_interfaces/Language.ml`, `Lang.ml`, `src/targeting/Guess_lang.ml` | `semgrep/lang/language.ss`, `guess-lang.ss`, `language-registry.ss` | -| Target parsing | `src/parsing/Parse_target.ml`, `Pfff_or_tree_sitter.ml`, `languages/*/tree-sitter/` | `semgrep/parse/parse-target.ss`, `tree-sitter-adapter.ss` | -| Pattern parsing | `src/parsing/Parse_pattern.ml`, `Check_pattern.ml`, language `*_to_generic` modules | `semgrep/parse/parse-pattern.ss` | -| Generic AST | `src/ast_generic/`, `src/matching/Normalize_generic.ml` | `semgrep/ast/*.ss` | -| Search matcher | `src/engine/Match_search_mode.ml`, `src/matching/Match_patterns.ml`, `Pattern_vs_code.ml` | `semgrep/match/*.ss`, `semgrep/engine/search-mode.ss` | -| Taint and dataflow | `src/analyzing/`, `src/il/`, `src/tainting/`, `src/engine/Match_tainting_mode.ml` | `semgrep/analysis/*.ss`, `semgrep/engine/taint-mode.ss` | -| Target discovery | `src/targeting/`, `src/target/`, `libs/gitignore/`, `libs/glob/`, `libs/paths/` | `semgrep/target/*.ss` | -| Core scan orchestration | `src/core_scan/`, `src/osemgrep/core_runner/` | `semgrep/engine/scan-core.ss` | -| Output and post-processing | `src/reporting/`, `src/osemgrep/reporting/`, `src/fixing/` | `semgrep/output/*.ss` | -| CLI and app behavior | `src/osemgrep/cli_*`, `src/osemgrep/networking/`, `cli/src/semgrep/commands/` | `semgrep/cli/*.ss`, `semgrep/app/*.ss` | -| MCP | `src/osemgrep/cli_mcp/`, `cli/src/semgrep/mcp/` | `semgrep/cli/mcp.ss` | - -This map should live with the port and be updated when a Semgrep source file is -declared fully replaced. A file is not replaced just because an analogous -Jerboa module exists; it is replaced when the compatibility oracle covers the -behavior and the Jerboa implementation passes that slice. - -## Proposed Jerboa Module Layout - -Use `.ss` modules only. - -Suggested layout: - -```text -semgrep/ - schema/ - output-v1.ss - rule-schema-v1.ss - rule-schema-v2.ss - lang.ss - rule/ - rule.ss - xpattern.ss - parse-rule.ss - validate-rule.ss - rule-error.ss - lang/ - analyzer.ss - language.ss - guess-lang.ss - language-registry.ss - ast/ - token.ss - range.ss - generic.ss - visitors.ss - normalize.ss - parse/ - parse-target.ss - parse-pattern.ss - tree-sitter-adapter.ss - jsonnet.ss - yaml.ss - match/ - generic-matcher.ss - pattern-vs-code.ss - match-patterns.ss - formula.ss - metavariable.ss - engine/ - scan-core.ss - match-rules.ss - search-mode.ss - taint-mode.ss - rule-prefilter.ss - timeouts.ss - analysis/ - naming.ss - typing.ss - constant-propagation.ss - ast-to-il.ss - il.ss - cfg.ss - dataflow.ss - taint.ss - target/ - find-targets.ss - semgrepignore.ss - gitignore.ss - glob.ss - baseline.ss - dependency-discovery.ss - output/ - json.ss - text.ss - sarif.ss - gitlab.ss - junit.ss - autofix.ss - nosemgrep.ss - app/ - auth.ss - registry.ss - config-fetch.ss - metrics.ss - telemetry.ss - cli/ - main.ss - scan.ss - ci.ss - test.ss - validate.ss - show.ss - lsp.ss - mcp.ss -``` - -This layout keeps the embeddable library separate from command-line concerns. - -## Embedding API - -The public API should be small and stable. - -Example shape: - -```scheme -(compile-rules source) -(compile-rules/files paths) -(compile-rules/config config-source) - -(parse-target language source) -(parse-pattern language pattern) - -(match-rule rule ast) -(match-rules rules ast) - -(scan config targets) -(scan-stream config targets callbacks) - -(apply-fixes findings) -(format-findings format result) -``` - -Important embedding properties: - -- No shelling out. -- No global process state required. -- Configurable filesystem, network, and git providers. -- Structured errors instead of process exits. -- Streaming findings for long scans. -- Cancellation support. -- Timeouts controlled by the host app. -- Metrics returned as data. -- Output formatting optional. - -The CLI should call this API rather than owning scan behavior itself. - -## Phased Port Plan - -### Phase 0: Compatibility Oracle - -Before porting, freeze behavior from the current implementation. - -Artifacts to capture: - -- Rule parsing success/failure snapshots. -- Rule validation diagnostics. -- Target parsing snapshots. -- Pattern parsing snapshots. -- Match result snapshots. -- Taint result snapshots. -- Autofix snapshots. -- `nosemgrep` snapshots. -- Target selection snapshots. -- CLI output snapshots. -- JSON v1 output snapshots. -- SARIF output snapshots. -- Performance baselines. - -This oracle should run against the current Semgrep implementation and then -against the Jerboa implementation. - -Without this step, the port will drift. - -The oracle should be checked in as a small harness, not as an informal list of -manual commands. It should have two interchangeable backends: - -- `semgrep-current`: invokes the checked-out Semgrep implementation. -- `semgrep-jerboa`: invokes the Jerboa implementation through the embedding API - or thin CLI. - -The harness should normalize only fields that are deliberately unstable: - -- Absolute workspace paths. -- Temporary directory names. -- Wall-clock timings. -- Process IDs. -- Network timestamps and request IDs. -- Version banners when testing behavior unrelated to version reporting. - -It should not normalize: - -- Match locations. -- Byte offsets. -- Metavariable bindings. -- Rule IDs. -- Severity values. -- Skipped target reasons. -- Parse error categories. -- Output field presence or absence. -- Match ordering. - -Initial oracle slices: - -1. Rule parse matrix: - - Inputs from `tests/rule_formats/`, `tests/errors/`, - `tests/rules_error_recovery/`, and representative registry configs. - - Expected output includes success/failure, normalized rule IR, and diagnostic - locations. -2. Pattern parse matrix: - - Inputs from `tests/patterns/`, `tests/parsing_patterns/`, and common - patterns embedded in `tests/rules/`. - - Expected output includes normalized generic AST shape and parser errors. -3. Target parse matrix: - - Inputs from `tests/parsing/`, `tests/parsing_partial/`, - `tests/parsing_errors/`, and `tests/parsing_missing/`. - - Expected output includes generic AST snapshots, skipped/inserted tokens, - and partial parse diagnostics. -4. Search match matrix: - - Inputs from `tests/rules/`, `tests/irrelevant_rules/`, - `tests/metachecks/`, and `tests/explanations/`. - - Expected output includes findings, ranges, metavariables, explanations - where enabled, and errors. -5. Taint matrix: - - Inputs from `tests/tainting_rules/` and `tests/taint_maturity/`. - - Expected output includes source/sink traces, labels, requirements, - propagator effects, and sanitizer behavior. -6. Targeting matrix: - - Inputs from `src/targeting/tests/`, CLI e2e path tests, synthetic git repos, - and symlink fixtures. - - Expected output includes selected files, skipped files, project roots, and - skip reasons. -7. Output matrix: - - Inputs from CLI snapshot tests for JSON, SARIF, text, GitLab, JUnit, - autofix, and `nosemgrep`. - - Expected output is the emitted artifact after the narrow normalization - rules above. - -Every later phase should add oracle cases before or alongside implementation. - -### Phase 1: Runtime Foundation - -Build or confirm the Jerboa libraries needed by the port: - -- JSON. -- YAML. -- Jsonnet. -- Regex. -- Paths. -- Filesystem traversal. -- Glob. -- Gitignore. -- HTTP/TLS. -- Git access. -- Parallel execution. -- Timeouts. -- Logging. -- Metrics. -- Structured errors. - -This phase should also establish project conventions for `.ss` module names, -exports, tests, generated files, and package boundaries. - -### Phase 2: Schema and Code Generation - -Port generated interfaces first. - -Inputs: - -- `semgrep_output_v1.atd` -- `rule_schema_v1.yaml` -- `rule_schema_v2.atd` -- `lang.json` - -Outputs: - -- Jerboa data definitions. -- JSON encoders. -- JSON decoders. -- Validators. -- Default values. -- Compatibility aliases. -- Snapshot tests. - -This reduces ambiguity in later phases. - -### Phase 3: Rule Parser and Rule IR - -Port the Semgrep rule language. - -This phase must support: - -- YAML rules. -- JSON rules. -- Jsonnet rules. -- Registry-style configs. -- Local config files. -- Local config directories. -- Pattern command mode. -- Old syntax. -- New syntax. -- Full source-position errors. - -This phase can be validated independently before target parsing is complete. - -### Phase 4: Language Registry and Parser Adapters - -Port language metadata and parsing dispatch. - -Tasks: - -- Generate language definitions from `lang.json`. -- Port analyzer definitions. -- Port language guessing. -- Implement parser adapter interfaces. -- Integrate tree-sitter or equivalent parsing. -- Map parser output into generic AST. -- Preserve pattern parsing behavior. -- Preserve partial parsing and recovery behavior. - -This is one of the highest-risk phases. - -### Phase 5: Generic AST - -Port the generic AST and supporting utilities. - -Tasks: - -- Define tokens and ranges. -- Define AST node variants. -- Define wrappers and origin tracking. -- Define visitors, mappers, and folders. -- Define equality and hashing. -- Define pretty/debug output. -- Define normalization passes. - -This phase should be driven by parser and matcher needs, not by aesthetic -cleanup. - -### Phase 6: Matcher - -Port the pattern matcher. - -Tasks: - -- Port generic matching monad/state. -- Port metavariable binding. -- Port bound-metavariable equality. -- Port ellipsis. -- Port metavariable ellipsis. -- Port list matching. -- Port associative/commutative matching. -- Port formula composition. -- Port pattern-inside/not-inside. -- Port focus metavariables. -- Port regex patterns. -- Port range and token propagation. -- Port rule prefiltering. - -Acceptance gate: non-taint search rules should match the current implementation -across the existing test corpus. - -### Phase 7: Semantic Analysis and Taint - -Port analysis layers. - -Tasks: - -- Name analysis. -- Type propagation. -- Constant propagation. -- AST-to-IL lowering. -- IL representation. -- CFG construction. -- Dataflow framework. -- Taint engine. -- Taint labels and requirements. -- Taint propagators. -- Sanitizers. -- By-side-effect behavior. -- Field sensitivity. -- Lambda and function effects. -- Timeouts. - -Acceptance gate: taint rules should match the current implementation across the -existing taint corpus. - -### Phase 8: Target Selection - -Port target discovery and filtering. - -Tasks: - -- Directory traversal. -- Explicit paths. -- Stdin. -- Language guessing. -- Include/exclude filtering. -- `.semgrepignore`. -- Gitignore compatibility. -- Built-in ignore defaults. -- Symlink behavior. -- Max file size. -- Baseline scans. -- Diff scans. -- Project root logic. -- Dependency file discovery. - -Acceptance gate: for a fixed config and repository, the Jerboa scanner should -choose the same targets and skipped-target reasons as current Semgrep. - -### Phase 9: Output, Autofix, and Post-processing - -Port result handling. - -Tasks: - -- JSON v1. -- SARIF. -- Text. -- GitLab. -- JUnit. -- Match sorting. -- Duplicate handling. -- Severity filtering. -- `nosemgrep`. -- Autofix. -- Skipped target reporting.