λgitsite

~ober/jerboa-semgrep

Imported from ~/mine/jerboa-semgrep

download snapshot

about

# jerboa-semgrep

Pure Jerboa Semgrep port scaffold.

This repository is intentionally library-first. The current executable slice
supports a compatibility-oriented Semgrep subset:

- YAML/JSON-compatible Semgrep config parsing for `rules:`, including config
  directories.
- Search rules with `pattern`, `pattern-regex`, `pattern-either`, and
  `patterns`.
- Structural metavariables like `$X`, repeated metavariable consistency, and
  Semgrep-style `...` and `$...ARGS` wildcard matching for unfielded child
  sequences.
- A limited regex-backed generic matcher for `generic` text targets, including
  plain ellipsis, `$X`, `$...X`, repeated metavariable consistency, and
  comment-tolerant whitespace separators.
- `as: $X` captures for positive `patterns` entries.
- `patterns` filters for `pattern-not`, `pattern-not-regex`,
  `pattern-inside`, `pattern-not-inside`, `metavariable-regex`, and
  `metavariable-pattern`.
- Safe `metavariable-comparison` support for direct string comparisons and
  numeric `int($X)`/`float($X)` comparisons.
- A minimal taint mode for ordered source/sink/sanitizer pattern lists with
  shared-metavariable compatibility.
- Language aliases for the supported languages, including `python3`, `python2`,
  `py`, `js`, `jsx`, `ts`, and `tsx`.
- Rule metadata in JSON output, rendered metavariable messages, rendered
  `fix:` and `fix-regex`, string-level autofix application, and formula-local
  `focus-metavariable` range focusing.
- A generated Semgrep language registry from upstream `lang.json`, with
  parser support still limited to the wired tree-sitter grammars.
- Target parsing via `jerboa-treesitter` for JSON, Python, JavaScript, JSX,
  and TypeScript/TSX parsed through the JavaScript grammar as a fallback.
- Recursive directory targets with conservative default source-file filtering.
- Descriptor-relative target/config walks that skip symlinks, reject hard
  links, pin directory identities, enforce cumulative traversal budgets, and
  commit autofixes only if the originally read inode is still in place.
- A first `.semgrepignore` pass with `!` negation for scanned directory roots,
  plus line-level `nosemgrep` suppression, including rule-specific comments.
- CLI compatibility for `semgrep scan --json|--sarif|--text --config RULES TARGET`,
  plus `--include`, `--exclude`, `--severity`, `--autofix`, and stdin target
  `-` with `--lang`.
- Semgrep-like finding records and JSON/SARIF/text output, including
  `extra.metavars` for structural captures.

Unsupported Semgrep rule forms fail explicitly instead of pretending to be
compatible. Tree-sitter grammars are provided by `jerboa-treesitter`, which
uses statically linked grammars only.

## Build, Test, And Release Gates

```sh
make test
```

`JERBOA_TREESITTER_HOME` defaults to `../jerboa-treesitter`. That sibling
checkout must be built first because parser FFI is isolated there.
CI instead fetches the exact commit and tree in `dependency-lock.tsv`.
The network Jerbuild bootstrap is deliberately fail-closed until a producer
signing identity, signed manifest, and asset digest are provisioned in
`support/jerbuild-bootstrap.lock` and `support/jerboa-release-signers`; CI
never executes a merely versioned, same-origin download.

Run the release/security gates:

```sh
make security
make malformed-corpus
make verify
make target-evidence
make release-evidence
```

`make verify` runs static security checks, dependency audit, import checks, the
512-case hostile rule/source corpus, and the smoke/regression test suite.
`make release-evidence` writes
`dist/release-evidence`, including Semgrep SBOM/source hashes, refreshed sibling
`jerboa-treesitter` grammar/runtime evidence, and a repeated-transpile
reproducibility report for tracked generated Scheme artifacts, normalized
Jerbuild hash metadata, repeated corpus output, and target scanner proof
status.

Production support requires marker-complete target proof through
`JSEMGREP_TARGET_PROOF_FILE` with `JSEMGREP_REQUIRE_TARGET_PROOF=1`; without it,
target-only oracle, compatibility, review, autofix, and output-redaction gates
remain blocked.

Input size caps are enforced before rule configs and targets are fully read:

- `JSEMGREP_MAX_CONFIG_CHARS`, default `4194304`
- `JSEMGREP_MAX_TARGET_CHARS`, default `8388608`

Untrusted `pattern-regex`, `pattern-not-regex`, nested regex formulae, and
`metavariable-regex` constraints use the bundled Rust finite-automata engine.
General look-around and backreferences fail closed; the bounded compatibility
layer supports only a leading capture-free `(?!literal)` metavariable filter.
Patterns are capped at 8192 characters, capture groups at 128, and findings at
50000 per rule scan. `fix-regex` also uses the linear engine with a 50000
replacement/16 MiB output cap; replacement text is currently literal, and
capture-template `$` syntax fails closed. `make adversarial-fs` covers external-file links,
directory cycles, hard links, linked configs, and autofix link rejection.

This repository is expected to remain network-free, process-execution-free, and
free of direct Scheme FFI. Native parser linkage belongs in `jerboa-treesitter`
and is audited by `make audit`.

Regenerate the Semgrep language registry from the sibling checkout:

```sh
make generate
```

Run the focused compatibility oracle against a current Semgrep binary:

```sh
make oracle
```

`make oracle` looks for a sibling Semgrep checkout binary, then `semgrep` on
`PATH`. Set `SEMGREP_CURRENT=/path/to/semgrep` to override it. If no current
Semgrep binary is available, the oracle skips cleanly. The current local
baseline passes 42/42 normalized oracle cases against packaged Semgrep 1.164.0.

recent commits