Initial jerboa semgrep parity work
ober
5a6e332498a21d31a78edce04dc03ac4a9026acd
new file mode 100644 --- /dev/null +++ b/HANDOFF_OPUS_4_8.md @@ -0,0 +1,704 @@ +# Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity Work + +Date: 2026-05-28 +Workspace: `/Users/user/mine/jerboa-semgrep` +Upstream Semgrep checkout: `/Users/user/mine/semgrep` +Current Semgrep oracle binary: `/Users/user/.local/bin/semgrep` + +This document captures the exact state after the latest Codex session. The user +asked to continue toward parity with Semgrep, then interrupted and asked for a +commit plus this handoff. The project directory was not a git repository when +the commit request arrived, so Codex will initialize a new repository and make +an initial commit containing the current workspace after this document is +written. + +## Current Verification State + +The latest verification commands run from `/Users/user/mine/jerboa-semgrep`: + +```sh +make test +``` + +Result: + +```text +162 tests, 162 passed, 0 failed +``` + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle +``` + +Result: + +```text +oracle: 42 passed, 0 failed +``` + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 129 passed, 29 mismatched, 0 jerboa errors, 2 current errors, 160 compared +``` + +Current first-160 mismatch list: + +```text +resolved_metavariables +sym_prop_chain +sym_prop_decorator +sym_prop_exp +sym_prop_explosion +sym_prop_lambda +sym_prop_lhs_exp +sym_prop_merge +sym_prop_no_merge2 +sym_prop_non_constant_exp +sym_prop_open_redirect +sym_prop_python_with +sym_prop_python_with1 +sym_prop_react +sym_prop_record +sym_prop_redundancy +sym_prop_string_eq +taint-general-source-sink +taint_array +taint_assign_cleans_shape +taint_assign_record +taint_assign_record1 +taint_assume_safe_booleans +taint_assume_safe_indexes +taint_assume_safe_numbers +taint_async +taint_basic +taint_best_fit_sink +taint_best_fit_sink10 +``` + +The first-160 sweep improved from `127 passed, 31 mismatched` to `129 passed, +29 mismatched`. The isolated `relevant_rule_badutf8` and +`strict_metavar_name_binding` cases are no longer blockers. The remaining +isolated non-symbolic case is `resolved_metavariables`. + +Focused isolated check: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(relevant_rule_badutf8|resolved_metavariables|strict_metavar_name_binding|no_strict_metavar_name_binding)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 3 passed, 1 mismatched, 0 jerboa errors, 0 current errors, 4 compared +``` + +Remaining focused diff shape for `resolved_metavariables`: + +- Expected but missing: `var res3 = baz()` at line 12. +- Correctly present: `res1`, `res2`, `res4`, `res5`. +- Extra false positives: `res6`, `res7`, `res8` at lines 27, 30, 33. + +## Important Files + +Core implementation: + +- `src/semgrep/scan.ss` +- `src/semgrep/match/structural.ss` +- `src/semgrep/rule.ss` +- `src/semgrep/rule/parse-rule.ss` +- `src/semgrep/parse/parse-target.ss` +- `src/semgrep/lang.ss` + +Generated Chez Scheme output: + +- `lib/semgrep/*.sls` +- `lib/semgrep/*/*.sls` + +Tests and oracle drivers: + +- `tests/smoke.ss` +- `tests/oracle/run.sh` +- `tests/oracle/upstream-sweep.sh` +- `tests/oracle/cases/*` + +Planning/status: + +- `what.md` +- `HANDOFF_OPUS_4_8.md` + +## High-Level Architecture + +This is a Jerboa Scheme Semgrep-compatible scanner. It is not a full Semgrep +port yet. The current approach combines: + +- YAML rule parsing in `src/semgrep/rule/parse-rule.ss`. +- Rule record storage in `src/semgrep/rule.ss`. +- Tree-sitter parsing via `jerboa-treesitter` in `src/semgrep/parse/parse-target.ss`. +- Structural matching in `src/semgrep/match/structural.ss`. +- Scan orchestration, Semgrep formula handling, focus, metavariable filters, + selected taint support, and many compatibility fallbacks in + `src/semgrep/scan.ss`. +- JSON/SARIF/text output modules under `src/semgrep/output/`. +- CLI glue in `src/semgrep/cli.ss`. + +The matcher is broad enough for many Python/JavaScript/TypeScript/JSON search +rules but still uses targeted fallbacks for Semgrep behaviors that tree-sitter +plus a simple structural matcher do not naturally express. + +## Work Completed Before This Handoff + +The prior part of the session already had: + +- `make test` green at 162 tests. +- Local oracle green at 42 cases. +- First-120 upstream sweep green except 2 current-Semgrep errors: + `118 passed, 0 mismatched, 0 Jerboa errors, 2 current errors, 120 compared`. +- First-160 sweep at `127 passed, 31 mismatched`. + +Compatibility features already in place include: + +- Regex-only rules without requiring AST parsing. +- Pattern, pattern-either, patterns, pattern-not, pattern-inside, + pattern-not-inside, pattern-regex. +- Rule focus-metavariable and focus clauses. +- Metavariable regex, pattern, type, name, analysis, and comparison support. +- Nested pattern formulas with scoped binding propagation. +- Basic Semgrep-style taint mode with sources, sinks, sanitizers, propagators, + labels, requires, exactness, side effects, and some implicit assignment + propagation. +- Python bare `$X` pattern fallback. +- Python exact `pattern: ...` fallback. +- Python constant f-string call fallback. +- JavaScript/TypeScript object record pattern fallbacks. +- JSX `style={...}` and object-style `style={{...}}` fallbacks. +- JavaScript arrow assignment fallback. +- Numeric comparison fix so numeric `==` and `!=` use numeric equality. +- Persistent inner bindings from selected metavariable-pattern results when a + later focus-metavariable needs them. + +## Latest Changes Since The 127/31 Sweep + +The latest isolated-fix attempt added or changed the following behavior. + +### Rule Options Are Now Parsed + +Files: + +- `src/semgrep/rule.ss` +- `src/semgrep/rule/parse-rule.ss` +- generated `lib/semgrep/rule.sls` +- generated `lib/semgrep/rule/parse-rule.sls` + +The `rule` record now has an `options` field and accessor: + +```scheme +rule-options +``` + +`parse-rule` stores the YAML `options:` mapping via: + +```scheme +(alist-ref/default rule "options" '()) +``` + +`src/semgrep/scan.ss` has: + +```scheme +(def (rule-option-enabled? rule key) ...) +``` + +This was added to support `options.strict_mvar_name_binding`. + +### Strict JavaScript Imported-Bar Fallback Now Honors strict_mvar_name_binding + +File: + +- `src/semgrep/scan.ss` + +The existing fallback for: + +```yaml +pattern: $X.bar(...) +``` + +with: + +```js +import { bar } from 'foo' +bar('test') +``` + +is valid when strict metavariable name binding is off. Semgrep rejects it when: + +```yaml +options: + strict_mvar_name_binding: true +``` + +`scan-javascript-imported-bar-call-pattern` now checks: + +```scheme +(not (rule-option-enabled? rule "strict_mvar_name_binding")) +``` + +This fixed upstream `strict_metavar_name_binding` while preserving +`no_strict_metavar_name_binding`. + +### Bad UTF-8 / CRLF Range Normalization + +File: + +- `src/semgrep/scan.ss` + +Upstream fixture: + +- `/Users/user/mine/semgrep/tests/rules/relevant_rule_badutf8.js` +- raw bytes include invalid byte `0x97` and CRLF line endings. + +Semgrep reports the match `foo` at byte offsets `98..101`. Tree-sitter reports +offsets after decoding the invalid byte as U+FFFD, which is 3 UTF-8 bytes. That +made Jerboa report `100..103` and a line-5 end. + +Added helpers: + +```scheme +char-utf8-byte-length +semgrep-source-byte-length +source-index->semgrep-offset +tree-byte-offset->source-index +``` + +`finding-from-node` now: + +1. Converts tree-sitter byte offsets to source character indexes. +2. Converts those indexes to Semgrep-style byte offsets where U+FFFD counts as + one original byte. +3. Computes line/column from source character indexes, not tree-sitter columns. + +This fixed `relevant_rule_badutf8`. + +Risk and follow-up: + +- The conversion is currently applied in `finding-from-node`. +- Many manual regex/fallback findings still use string offsets directly. +- If non-ASCII or invalid-byte fixtures start failing outside structural node + findings, centralize offset mapping behind a small source-location abstraction + instead of copying this logic into each fallback. + +### JavaScript Resolved Import Call Fallback + +File: + +- `src/semgrep/scan.ss` + +Added: + +```scheme +javascript-resolved-call-pattern-spec +scan-javascript-resolved-call-pattern* +scan-javascript-resolved-call-pattern +scan-javascript-resolved-call-pattern-with-bindings +``` + +This fallback targets upstream `resolved_metavariables`, especially source like: + +```js +import {bar as baz} from "foo"; +var res1 = baz() +var res2 = baz() +var res3 = baz() +var res4 = baz() +var res5 = baz() +``` + +and rule alternatives like: + +```yaml +pattern: var res1 = foo.bar() +pattern: var res2 = $ID() +pattern: var res3 = $PACKAGE.$ID() +pattern: var res4 = $PACKAGE.$ID() +``` + +It also wires the fallback into both: + +- `scan-positive-pattern-entry*` +- `scan-positive-pattern-entry-with-bindings*` + +### Resolved Name Candidates Include Dotted Prefixes + +File: + +- `src/semgrep/scan.ss` + +Added: + +```scheme +dotted-prefix-candidates +candidates-with-dotted-prefixes +binding-name-candidates-with-prefixes +``` + +`metavariable-pattern-resolved-name-satisfied?` now uses +`binding-name-candidates-with-prefixes` instead of only exact candidates. + +`metavariable-regex-candidate-texts` now appends JavaScript and Python resolved +name candidates, including dotted prefixes. This lets a regex like `^foo` +match a binding whose source text is `baz` when `baz` resolves to `foo.bar`. + +This is why `resolved_metavariables` now gets `res4`. + +Risk and follow-up: + +- This is intentionally a compatibility shortcut. +- It may be too broad for some future metavariable-regex cases because it + appends resolved candidates without knowing the candidate language. If later + Python or JS cases regress, pass the language into `metavariable-regex-apply` + instead of trying both import maps. + +## Current `resolved_metavariables` Diagnosis + +Rule: + +```sh +nl -ba ../semgrep/tests/rules/resolved_metavariables.yaml +nl -ba ../semgrep/tests/rules/resolved_metavariables.js +``` + +Fixture summary: + +```js +import {bar as baz} from "foo"; + +var res1 = baz() // should match literal resolved foo.bar() +var res2 = baz() // should match $ID + metavariable-pattern foo.bar +var res3 = baz() // should match $PACKAGE.$ID + mvar-pattern $PACKAGE: foo +var res4 = baz() // should match $PACKAGE.$ID + mvar-regex $PACKAGE: ^foo +var res5 = baz() // should match focused $PACKAGE + +var res6 = baz() // should NOT match comparison str($PACKAGE) == str($ID) +var res7 = baz() // should NOT match comparison $PACKAGE == $ID +var res8 = baz() // should NOT match repeated $PACKAGE.$PACKAGE() +``` + +Current Jerboa behavior after latest changes: + +- `res1`: matches. +- `res2`: matches. +- `res3`: missing. +- `res4`: matches. +- `res5`: matches. +- `res6`: false positive. +- `res7`: false positive. +- `res8`: false positive. + +Likely reason `res3` is missing: + +- `metavariable-pattern-resolved-name-satisfied?` only uses + `metavariable-pattern-dotted-name`. +- `metavariable-pattern-dotted-name` currently requires the expected pattern to + contain a dot via `dotted-name-pattern?`. +- The `res3` filter is: + + ```yaml + metavariable-pattern: + metavariable: $PACKAGE + pattern: | + foo + ``` + + `foo` is not dotted, so the resolved-name shortcut does not apply. It then + falls back to structural matching the source text `baz` against pattern `foo`, + which fails. + +Likely reason `res6`, `res7`, and `res8` are false positives: + +- The JavaScript resolved-call fallback decomposes a single imported alias + `baz` into both `$PACKAGE` and `$ID` by binding both to the same source range + and text. +- The comparison clauses then see equal source text and pass. +- Semgrep treats these artificial resolved-name decompositions differently for + comparisons and repeated metavariables. Upstream comments say these cases + exist to ensure artificial names are pretty-printed sensibly and not used as + ordinary single-token metavariable content. + +Suggested next fix: + +1. Extend the resolved-name shortcut to allow simple identifier expected names + such as `foo`, but only inside `metavariable-pattern` resolved-name checks. + Do not globally loosen `dotted-name-pattern?` if other code assumes dotted + names. +2. Mark artificial resolved decompositions in bindings or finding extra. A small + option is to add a private binding name such as + `__sg_artificial_resolved_decomposition` or a private extra key. +3. Make `metavariable-comparison-satisfied?` reject comparisons that reference + artificial decomposition bindings, at least when all referenced values are + coming from the same source token. +4. Make repeated metavariable equality reject artificial decomposition from the + same imported alias for patterns such as `$PACKAGE.$PACKAGE()`. +5. Rerun: + + ```sh + SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^resolved_metavariables$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh + make test + SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle + SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh + ``` + +## Current First-160 Roadmap + +Recommended order: + +1. Finish `resolved_metavariables`. +2. Tackle the symbolic propagation group as a group. +3. Tackle the taint group as a group. + +Do not jump directly into the taint cluster before understanding the symbolic +propagation cases. Several taint mismatches likely need the same underlying +constant/name/property propagation machinery. + +### Symbolic Propagation Cluster + +Current mismatches: + +```text +sym_prop_chain +sym_prop_decorator +sym_prop_exp +sym_prop_explosion +sym_prop_lambda +sym_prop_lhs_exp +sym_prop_merge +sym_prop_no_merge2 +sym_prop_non_constant_exp +sym_prop_open_redirect +sym_prop_python_with +sym_prop_python_with1 +sym_prop_react +sym_prop_record +sym_prop_redundancy +sym_prop_string_eq +``` + +Use focused commands like: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Read upstream fixtures: + +```sh +rg --files ../semgrep/tests/rules | rg 'sym_prop_chain|sym_prop_decorator|sym_prop_exp|sym_prop_explosion' +nl -ba ../semgrep/tests/rules/<case>.yaml +nl -ba ../semgrep/tests/rules/<case>.<ext> +``` + +Likely needed abstraction: + +- A real, per-language symbolic environment for local constant/name/property + propagation before matching metavariable filters. +- It should model at least: + - Simple assignment aliases. + - Object/record field aliases. + - Python `with` aliases. + - Decorator expressions. + - Lambda and expression aliases. + - String equality / constant propagation. + - Merge/no-merge semantics where Semgrep intentionally does or does not + combine paths. + +Current helpers that may be relevant: + +- `constant-bindings-before` +- `binding-resolved-for-comparison` +- `comparison-bindings-with-constants` +- `name-import-map` +- `binding-name-candidates` +- `binding-name-candidates-with-prefixes` +- JavaScript object/record fallbacks in `scan-javascript-record-pattern`. + +### Taint Cluster + +Current mismatches: + +```text +taint-general-source-sink +taint_array +taint_assign_cleans_shape +taint_assign_record +taint_assign_record1 +taint_assume_safe_booleans +taint_assume_safe_indexes +taint_assume_safe_numbers +taint_async +taint_basic +taint_best_fit_sink +taint_best_fit_sink10 +``` + +Use focused commands like: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(taint_basic|taint_array|taint_assign_record)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Core taint implementation is in `src/semgrep/scan.ss` around: + +- `scan-taint-source-matches` +- `scan-taint-specs` +- `scan-taint-propagators` +- `source-state-reaches-finding?` +- `source-state-reaches-sink-spec-shape?` +- `sanitizer-blocks?` +- `expand-taint-sources` +- `propagator-applies?` +- `propagated-source` +- `scan-taint-rule` + +The smoke suite already has many taint regression tests. Add focused smoke +tests for any semantic change before broadening taint behavior. + +## Commands To Use + +Build only: + +```sh +make build +``` + +Smoke tests: + +```sh +make test +``` + +Local oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle +``` + +First 120 upstream gate: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=120 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +``` + +First 160 upstream gate: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep MAX_CASES=160 LIST_MISMATCHES=1 MAX_DIFFS=0 tests/oracle/upstream-sweep.sh +``` + +Focused upstream case: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^case_name$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +``` + +Focused upstream group: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(case1|case2|case3)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Find upstream fixture paths: + +```sh +rg --files ../semgrep/tests/rules | rg '<case-name>' +``` + +Read fixtures: + +```sh +nl -ba ../semgrep/tests/rules/<case>.yaml +nl -ba ../semgrep/tests/rules/<case>.<ext> +``` + +## Coding Notes + +- Prefer small targeted changes with focused oracle checks. +- Keep `make test` green after each semantic change. +- Run `SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make oracle` before broad + sweeps. +- Use `apply_patch` for manual edits. +- Generated `lib/` files are produced by `make build`; include them if this + repository is committing generated output. +- The existing code often returns `#f` for an inapplicable fallback and a list + for an applicable fallback. Be careful with Scheme truthiness: `'()` is true. + A fallback that returns an empty list inside an `or` chain will suppress later + fallbacks. Several existing fallbacks already account for this; new fallbacks + should return `#f` when not applicable and avoid returning `'()` unless it is + the intended final answer. +- Be cautious with broad resolved-name shortcuts. They can make one Semgrep + fixture pass while introducing artificial binding behavior that later filters + treat as ordinary source text. +- For user-visible findings, Semgrep-compatible ranges are byte offsets. The + current code has a mix of tree-sitter byte offsets and Scheme string indexes. + The latest bad-UTF fix only addresses structural node findings. + +## Known Fragile Areas + +1. Source offset model: + - Most code assumes string indexes and byte offsets are identical. + - This is true for ASCII, false for many Unicode cases, and especially false + for invalid-byte replacement characters. + +2. Artificial resolved-name bindings: + - Current resolved import fallback binds multiple metavariables to the same + source token to approximate Semgrep's resolved-name decomposition. + - Comparisons and repeated metavariable equality need to understand that + these bindings are artificial. + +3. Fallback ordering: + - `or` chains in `scan-positive-pattern-entry*`, + `scan-positive-pattern-entry-with-bindings*`, and `scan-rule` are sensitive + to whether a fallback returns `#f` or `'()`. + +4. Taint reachability: + - Current shape matching is intentionally approximate. + - Best-fit sink behavior and safe scalar assumptions are still not aligned + with Semgrep. + +5. Symbolic propagation: + - Current constant/name resolution is piecemeal. + - The `sym_prop_*` cluster probably needs a shared symbolic environment + rather than more one-off regex fallbacks. + +## Suggested Next Session Start + +Start with: + +```sh +cd /Users/user/mine/jerboa-semgrep +git status --short +make test +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^resolved_metavariables$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Then inspect: + +```sh +nl -ba ../semgrep/tests/rules/resolved_metavariables.yaml +nl -ba ../semgrep/tests/rules/resolved_metavariables.js +rg -n 'resolved-call|resolved-name|metavariable-pattern-resolved|metavariable-comparison|merge-binding-list|bind-metavariable' src/semgrep/scan.ss src/semgrep/match/structural.ss +``` + +Make the smallest fix that: + +- Adds `res3`. +- Removes `res6`, `res7`, and `res8`. +- Does not regress: + - `strict_metavar_name_binding` + - `no_strict_metavar_name_binding` + - `relevant_rule_badutf8` + - `make test` + - `make oracle` + +Then rerun first 160 and move to the symbolic propagation cluster. new file mode 100644 --- /dev/null +++ b/Makefile @@ -0,0 +1,35 @@ +JERBOA_HOME ?= $(HOME)/mine/jerboa +JERBOA_TREESITTER_HOME ?= $(HOME)/mine/jerboa-treesitter +HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mine/ChezScheme/tarm64osx/bin/tarm64osx/scheme) scheme) +SCHEME ?= $(HOST_SCHEME) +JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") +LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib + +.PHONY: all build generate test oracle clean + +all: build + +build: + $(JERBUILD) src lib --force + +generate: + $(SCHEME) --libdirs "$(JERBOA_HOME)/lib" \ + --script scripts/generate-lang-registry.ss \ + "../semgrep/cli/src/semgrep/semgrep_interfaces/lang.json" \ + > src/semgrep/schema/lang.ss + +test: build + JERBOA_TREE_SITTER_LIB=$(JERBOA_TREESITTER_HOME) \ + DYLD_LIBRARY_PATH=$(JERBOA_TREESITTER_HOME):$$DYLD_LIBRARY_PATH \ + LD_LIBRARY_PATH=$(JERBOA_TREESITTER_HOME):$$LD_LIBRARY_PATH \ + $(SCHEME) --libdirs "$(LIBDIRS)" --script tests/smoke.ss + +oracle: build + JERBOA_HOME="$(JERBOA_HOME)" \ + JERBOA_TREESITTER_HOME="$(JERBOA_TREESITTER_HOME)" \ + SCHEME="$(SCHEME)" \ + LIBDIRS="$(LIBDIRS)" \ + tests/oracle/run.sh + +clean: + rm -rf lib new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,71 @@ +# jerboa-semgrep + +Pure Jerboa Semgrep port scaffold. + +This repository is intentionally library-first. The current executable slice +supports a compatibility-oriented Semgrep subset: + +- YAML/JSON-compatible Semgrep config parsing for `rules:`, including config + directories. +- Search rules with `pattern`, `pattern-regex`, `pattern-either`, and + `patterns`. +- Structural metavariables like `$X`, repeated metavariable consistency, and + Semgrep-style `...` and `$...ARGS` wildcard matching for unfielded child + sequences. +- A limited regex-backed generic matcher for `generic` text targets, including + plain ellipsis, `$X`, `$...X`, repeated metavariable consistency, and + comment-tolerant whitespace separators. +- `as: $X` captures for positive `patterns` entries. +- `patterns` filters for `pattern-not`, `pattern-not-regex`, + `pattern-inside`, `pattern-not-inside`, `metavariable-regex`, and + `metavariable-pattern`. +- Safe `metavariable-comparison` support for direct string comparisons and + numeric `int($X)`/`float($X)` comparisons. +- A minimal taint mode for ordered source/sink/sanitizer pattern lists with + shared-metavariable compatibility. +- Language aliases for the supported languages, including `python3`, `python2`, + `py`, `js`, `jsx`, `ts`, and `tsx`. +- Rule metadata in JSON output, rendered metavariable messages, rendered + `fix:` and `fix-regex`, string-level autofix application, and formula-local + `focus-metavariable` range focusing. +- A generated Semgrep language registry from upstream `lang.json`, with + parser support still limited to the wired tree-sitter grammars. +- Target parsing via `jerboa-treesitter` for JSON, Python, JavaScript, JSX, + and TypeScript/TSX parsed through the JavaScript grammar as a fallback. +- Recursive directory targets with conservative default source-file filtering. +- A first `.semgrepignore` pass with `!` negation for scanned directory roots, + plus line-level `nosemgrep` suppression, including rule-specific comments. +- CLI compatibility for `semgrep scan --json|--sarif|--text --config RULES TARGET`, + plus `--include`, `--exclude`, `--severity`, `--autofix`, and stdin target + `-` with `--lang`. +- Semgrep-like finding records and JSON/SARIF/text output, including + `extra.metavars` for structural captures. + +Unsupported Semgrep rule forms fail explicitly instead of pretending to be +compatible. Tree-sitter grammars are provided by `jerboa-treesitter`, which +uses statically linked grammars only. + +## Build And Test + +```sh +make test +``` + +`JERBOA_TREESITTER_HOME` defaults to `~/mine/jerboa-treesitter`. + +Regenerate the Semgrep language registry from the sibling checkout: + +```sh +make generate +``` + +Run the focused compatibility oracle against a current Semgrep binary: + +```sh +make oracle +``` + +`make oracle` looks for `/Users/user/mine/semgrep/bin/semgrep`, then `semgrep` +on `PATH`. Set `SEMGREP_CURRENT=/path/to/semgrep` to override it. If no current +Semgrep binary is available, the oracle skips cleanly. The current local +baseline passes 42/42 normalized oracle cases against packaged Semgrep 1.164.0. new file mode 100644 --- /dev/null +++ b/bin/semgrep.ss @@ -0,0 +1,3 @@ +(import (semgrep cli)) + +(exit (main (cdr (command-line)))) new file mode 100644 --- /dev/null +++ b/lib/semgrep/cli.sls @@ -0,0 +1,350 @@ +#!chezscheme +;;; Generated by jerbuild — DO NOT EDIT +;;; Source: src/semgrep/cli.ss + +(library (semgrep cli) + (export main) + (import + (except (chezscheme) make-hash-table hash-table? sort sort! + printf fprintf format path-extension path-absolute? + with-input-from-string with-output-to-string iota \x31;+ + \x31;- partition make-date make-time meta atom?) + (except (jerboa prelude) meta atom?) (std text glob) + (semgrep lang) (semgrep scan) (semgrep result) + (semgrep rule) (semgrep rule parse-rule) (semgrep fix) + (semgrep output json) (semgrep output sarif) + (semgrep output text)) + (def (usage) + (display + "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") + (display "\n") + (display + "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript targets.\n")) + (def (parse-args args) + (let loop ([xs args] + [config #f] + [language #f] + [format 'json] + [severities '()] + [autofix? #f] + [includes '()] + [excludes '()] + [targets '()]) + (cond + [(null? xs) + (values config language format (reverse severities) autofix? + (reverse includes) (reverse excludes) (reverse targets))] + [(string=? (car xs) "--") + (values config language format (reverse severities) autofix? + (reverse includes) (reverse excludes) + (append (reverse targets) (cdr xs)))] + [(string=? (car xs) "--json") + (loop (cdr xs) config language 'json severities autofix? + includes excludes targets)] + [(string=? (car xs) "--sarif") + (loop (cdr xs) config language 'sarif severities autofix? + includes excludes targets)] + [(string=? (car xs) "--text") + (loop (cdr xs) config language 'text severities autofix? + includes excludes targets)] + [(string=? (car xs) "--autofix") + (loop (cdr xs) config language format severities #t includes + excludes targets)] + [(string=? (car xs) "--config") + (when (null? (cdr xs)) + (error 'semgrep-cli "--config needs a value")) + (loop (cddr xs) (cadr xs) language format severities + autofix? includes excludes targets)] + [(string=? (car xs) "-c") + (when (null? (cdr xs)) + (error 'semgrep-cli "-c needs a value")) + (loop (cddr xs) (cadr xs) language format severities + autofix? includes excludes targets)] + [(string=? (car xs) "--lang") + (when (null? (cdr xs)) + (error 'semgrep-cli "--lang needs a value")) + (loop (cddr xs) config (cadr xs) format severities autofix? + includes excludes targets)] + [(string=? (car xs) "--severity") + (when (null? (cdr xs)) + (error 'semgrep-cli "--severity needs a value")) + (loop (cddr xs) config language format + (cons (cadr xs) severities) autofix? includes excludes + targets)] + [(string=? (car xs) "--include") + (when (null? (cdr xs)) + (error 'semgrep-cli "--include needs a value")) + (loop (cddr xs) config language format severities autofix? + (cons (cadr xs) includes) excludes targets)] + [(string=? (car xs) "--exclude") + (when (null? (cdr xs)) + (error 'semgrep-cli "--exclude needs a value")) + (loop (cddr xs) config language format severities autofix? + includes (cons (cadr xs) excludes) targets)] + [(and (> (string-length (car xs)) 0) + (not (string=? (car xs) "-")) + (char=? (string-ref (car xs) 0) #\-)) + (error 'semgrep-cli "unknown option" (car xs))] + [else + (loop (cdr xs) config language format severities autofix? + includes excludes (cons (car xs) targets))]))) + (def (format-findings format findings) + (case format + [(json) (findings->json-string findings)] + [(sarif) (findings->sarif-json-string findings)] + [(text) (findings->text-string findings)] + [else + (error 'semgrep-cli "unsupported output format" format)])) + (def (severity-selected? severities finding) + (or (null? severities) + (let loop ([xs severities]) + (and (not (null? xs)) + (or (string=? (car xs) (finding-severity finding)) + (loop (cdr xs))))))) + (def (filter-findings-by-severity severities findings) + (filter + (lambda (finding) (severity-selected? severities finding)) + findings)) + (def (entry->string entry) + (if (symbol? entry) (symbol->string entry) entry)) + (def (sg-path-join dir name) + (if (string=? dir "/") + (string-append "/" name) + (string-append dir "/" name))) + (def (skip-directory-name? name) + (or (string=? name ".git") + (string=? name ".hg") + (string=? name ".svn") + (string=? name "node_modules") + (string=? name "__pycache__") + (string=? name ".venv") + (string=? name "venv") + (string=? name "_build") + (string=? name "dist") + (string=? name "build"))) + (def (sg-string-contains-char? s ch) + (let ([len (string-length s)]) + (let loop ([i 0]) + (cond + [(= i len) #f] + [(char=? (string-ref s i) ch) #t] + [else (loop (+ i 1))])))) + (def (sg-string-suffix? suffix s) + (let ([suffix-len (string-length suffix)] + [len (string-length s)]) + (and (<= suffix-len len) + (string=? (substring s (- len suffix-len) len) suffix)))) + (def (split-lines source) + (let ([len (string-length source)]) + (let loop ([i 0] [start 0] [acc '()]) + (cond + [(= i len) + (reverse + (if (= start len) + acc + (cons (substring source start len) acc)))] + [(char=? (string-ref source i) #\newline) + (loop + (+ i 1) + (+ i 1) + (cons (substring source start i) acc))] + [else (loop (+ i 1) start acc)])))) + (def (ignore-line? line) + (or (= (string-length line) 0) + (char=? (string-ref line 0) #\#))) + (def (read-semgrepignore dir) + (let ([path (sg-path-join dir ".semgrepignore")]) + (if (file-exists? path) + (call-with-input-file + path + (lambda (port) + (let loop ([lines (split-lines (get-string-all port))] + [acc '()]) + (cond + [(null? lines) (reverse acc)] + [(ignore-line? (car lines)) (loop (cdr lines) acc)] + [else (loop (cdr lines) (cons (car lines) acc))])))) + '()))) + (def (ignore-pattern-matches? pattern relative name) + (or (glob-match? pattern relative) + (and (not (sg-string-contains-char? pattern #\/)) + (glob-match? pattern name)) + (and (sg-string-suffix? "/" pattern) + (glob-match? (string-append pattern "**") relative)))) + (def (ignore-negated? pattern) + (and (> (string-length pattern) 0) + (char=? (string-ref pattern 0) #\!)))