Add upstream pattern fixture oracle
ober
9962fe1502f143ef0e3cfe71f5cea476377a8550
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,13 +1,13 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 12:08 MDT +Date: 2026-05-29 12:37 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`07dc9aa Cover PHP upstream fixtures` -Latest implementation commit before this Java checkpoint: -`07dc9aa Cover PHP upstream fixtures` +`efbfd7a Cover Java upstream fixtures` +Latest implementation commit before this pattern-oracle checkpoint: +`efbfd7a Cover Java upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -34,15 +34,12 @@ language parser coverage. The recent code checkpoint includes: ```text +Makefile HANDOFF_OPUS_4_8.md src/semgrep/scan.ss -src/semgrep/lang.ss -src/semgrep/cli.ss lib/semgrep/scan.sls -lib/semgrep/lang.sls -lib/semgrep/cli.sls src/.jerbuild-hashes -tests/oracle/upstream-sweep.sh +tests/oracle/patterns-sweep.sh tests/smoke.ss ``` @@ -63,7 +60,7 @@ make test Result: ```text -272 tests, 272 passed, 0 failed +273 tests, 273 passed, 0 failed ``` Local oracle: @@ -78,6 +75,30 @@ Result: oracle: 42 passed, 0 failed ``` +Focused upstream pattern-fixture oracle through the new Make target: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep make patterns-oracle +``` + +Result: + +```text +patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared +``` + +Direct focused upstream pattern-fixture oracle: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep PATTERN_LANGS=python CASE_REGEX='^(dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/patterns-sweep.sh +``` + +Result: + +```text +patterns-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 0 skipped, 5 compared +``` + Focused generic Gemfile regex guardrail: ```sh @@ -384,9 +405,95 @@ fixture YAML. Packaged Semgrep 1.164.0 exits with does not produce mismatches for these fixtures; the current-error classification comes from the packaged oracle failing before Jerboa comparison. +Whitespace check: + +```sh +git diff --check +``` + +Result: no output and exit code 0. + ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.java` +This checkpoint opens a second upstream oracle surface based on Semgrep's +`tests/patterns` fixtures. The existing `tests/oracle/upstream-sweep.sh` only +compares same-basename rule/target pairs under `tests/rules`; that surface is +now exhausted for the currently supported text-backed suffix set. The new +`tests/oracle/patterns-sweep.sh` generates temporary YAML rules from upstream +`.sgrep` files, runs packaged Semgrep and Jerboa against the sibling target +fixture, normalizes JSON findings with the existing normalizer, sorts them, and +diffs exact finding ranges/messages/check IDs. + +The new harness supports: + +- `SEMGREP_ROOT`, `SEMGREP_CURRENT`, `SCHEME`, `JERBOA_TREESITTER_HOME`, + `LIBDIRS`, and `PATTERN_ROOT` overrides. +- `PATTERN_LANGS` for selecting upstream pattern directories. `python` and + `js` are mapped initially; this checkpoint verifies only the Python slice. +- `CASE_REGEX`, `MAX_CASES`, `LIST_MISMATCHES`, and `MAX_DIFFS`, matching the + style of `upstream-sweep.sh`. +- stable generated rule IDs under `oracle.patterns.*`, so temporary config + paths do not produce false check-id diffs. +- no-target/empty-pattern skip accounting and separate packaged-current, + Jerboa, and normalized-output mismatch counts. + +`Makefile` now has a `patterns-oracle` target that builds the tracked Jerboa +libraries and then runs the curated passing pattern slice by default through +`PATTERN_CASE_REGEX`. Use `make patterns-oracle PATTERN_CASE_REGEX=...` to grow +the green slice or `tests/oracle/patterns-sweep.sh` directly for exploratory +sweeps. Running all Python `.sgrep` fixtures is intentionally still an +exploratory parity sweep, not a passing target. + +The exact Python upstream pattern cases covered now are: + +```text +dots_stmts +import_metavar +metavar_equality_var +metavar_stmt +misc_dots_stmts +``` + +This checkpoint also fixes one real Python structural-range parity gap exposed +by `tests/patterns/python/import_metavar.sgrep`. When the structural matcher +returns a Python module-root finding, Jerboa now trims leading comment/blank +fixture lines and trailing final newlines from the reported range. That matches +packaged Semgrep for the import-alias fixture where the target begins with +`#ERROR: match`. The trim is applied only to Python-like structural findings +without an explicit internal match-range binding, preserving focused and +range-bound findings. A smoke test locks the exact range: + +```text +start line/col/offset: 2 / 1 / 14 +end line/col/offset: 4 / 34 / 76 +``` + +Known next Python pattern gaps from the quick probe: + +```text +deep_expr_operator packaged current 1 / Jerboa 0 +dict_ellipsis packaged current 1 / Jerboa 0 +dots_nested_stmts packaged current 1 / Jerboa 0 +cp_strings packaged current 3 / Jerboa 0 +dots_list packaged current 2 / Jerboa 1 +``` + +Recommended next work: + +- Grow `patterns-sweep.sh` from the five exact Python fixtures, keeping every + newly green case in a documented `CASE_REGEX` until the entire directory + passes. +- Implement Python deep-expression operator support for `<... ... ...>` style + matching. +- Improve Python dict/list ellipsis matching; `dict_ellipsis` and `dots_list` + are the most direct fixtures. +- Add the missing string constant-propagation behavior needed by `cp_strings`. +- After the Python pattern frontier is stable, enable and verify the `js` + pattern directory mapping already present in the harness. + +## Recent Prior Checkpoint: Java Same-Basename Fixtures + +The previous code checkpoint broadened the current sorted upstream sweep by adding `.java` targets and a narrow Java text-backed compatibility path. Java is not parser-backed in this port yet. The added support covers the forty-two current same-basename upstream Java fixtures without claiming general Java parser or --- a/Makefile +++ b/Makefile @@ -4,8 +4,9 @@ HOST_SCHEME ?= $(firstword $(wildcard $(JERBOA_HOME)/.chez/bin/scheme $(HOME)/mi SCHEME ?= $(HOST_SCHEME) JERBUILD ?= $(if $(wildcard $(JERBOA_HOME)/jerbuild),$(JERBOA_HOME)/jerbuild,$(SCHEME) --libdirs "$(JERBOA_HOME)/lib" --script "$(JERBOA_HOME)/jerbuild.ss") LIBDIRS = lib:$(JERBOA_TREESITTER_HOME)/lib:$(JERBOA_HOME)/lib +PATTERN_CASE_REGEX ?= ^(dots_stmts|import_metavar|metavar_equality_var|metavar_stmt|misc_dots_stmts)$$ -.PHONY: all build generate test oracle clean +.PHONY: all build generate test oracle patterns-oracle clean all: build @@ -31,5 +32,13 @@ oracle: build LIBDIRS="$(LIBDIRS)" \ tests/oracle/run.sh +patterns-oracle: build + JERBOA_HOME="$(JERBOA_HOME)" \ + JERBOA_TREESITTER_HOME="$(JERBOA_TREESITTER_HOME)" \ + SCHEME="$(SCHEME)" \ + LIBDIRS="$(LIBDIRS)" \ + CASE_REGEX="$(PATTERN_CASE_REGEX)" \ + tests/oracle/patterns-sweep.sh + clean: rm -rf lib --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -4598,7 +4598,12 @@ (scan-structural-pattern rule language path source target-root candidate)) patterns))]) - (and (not (null? findings)) findings)))))) + (and (not (null? findings)) + (map (lambda (finding) + (trim-python-module-finding + finding + source)) + findings))))))) (def (scan-python-import-local-pattern-with-bindings rule language path source target-root pattern initial-bindings) (and (symbolic-python-like-language? language) @@ -4613,7 +4618,37 @@ target-root candidate initial-bindings)) patterns))]) - (and (not (null? findings)) findings)))))) + (and (not (null? findings)) + (map (lambda (finding) + (trim-python-module-finding + finding + source)) + findings))))))) + (def (python-leading-code-offset source) + (let ([len (string-length source)]) + (let loop ([line-start 0]) + (if (>= line-start len) + len + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [blank? (= first line-end)] + [comment? (and (< first line-end) + (char=? + (string-ref source first) + #\#))] + [next (if (< line-end len) (+ line-end 1) len)]) + (if (or blank? comment?) (loop next) first)))))) + (def (trim-python-module-finding finding source) + (let* ([start (python-leading-code-offset source)] + [end (source-content-end source)]) + (if (and (= (finding-start-offset finding) 0) + (>= (finding-end-offset finding) end) + (< start end)) + (finding-with-range finding source start end) + finding))) (def (symbolic-assignment-line-offset source start) (if (and (< start (string-length source)) (char=? (string-ref source start) #\newline)) @@ -5833,10 +5868,17 @@ extra message) (finding-from-node rule path source node extra message))] + [finding1 (if (and (not range-binding) + (symbolic-python-like-language? + language)) + (trim-python-module-finding + finding0 + source) + finding0)] [finding (if range-binding - finding0 + finding1 (maybe-extend-expression-semicolon rule pattern source node - finding0))] + finding1))] [focused (apply-rule-focus rule finding)]) (node-close! node) (loop (cdr xs) (append (reverse focused) acc)))))))) @@ -6390,6 +6432,14 @@ (finding-end-line finding) (finding-end-col finding) (finding-start-offset finding) (finding-end-offset finding) (finding-message finding) (finding-severity finding) extra)) + (def (finding-with-range finding source start end) + (let-values ([(start-line start-col) + (offset->line-col source start)] + [(end-line end-col) (offset->line-col source end)]) + (make-finding (finding-rule-id finding) (finding-path finding) start-line + start-col end-line end-col start end + (finding-message finding) (finding-severity finding) + (finding-extra finding)))) (def (finding-with-bindings rule finding bindings source) (let* ([match-text (source-slice source --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "806069F744F692AC") + ("src/semgrep/scan.ss" . "C754F524EE6C647C") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -4694,7 +4694,10 @@ target-root candidate)) patterns))]) - (and (not (null? findings)) findings)))))) + (and (not (null? findings)) + (map (lambda (finding) + (trim-python-module-finding finding source)) + findings))))))) (def (scan-python-import-local-pattern-with-bindings rule @@ -4719,7 +4722,36 @@ candidate initial-bindings)) patterns))]) - (and (not (null? findings)) findings)))))) + (and (not (null? findings)) + (map (lambda (finding) + (trim-python-module-finding finding source)) + findings))))))) + +(def (python-leading-code-offset source) + (let ([len (string-length source)]) + (let loop ([line-start 0]) + (if (>= line-start len) + len + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [blank? (= first line-end)] + [comment? (and (< first line-end) + (char=? (string-ref source first) #\#))] + [next (if (< line-end len) + (+ line-end 1) + len)]) + (if (or blank? comment?) + (loop next) + first)))))) + +(def (trim-python-module-finding finding source) + (let* ([start (python-leading-code-offset source)] + [end (source-content-end source)]) + (if (and (= (finding-start-offset finding) 0) + (>= (finding-end-offset finding) end) + (< start end)) + (finding-with-range finding source start end) + finding))) (def (symbolic-assignment-line-offset source start) (if (and (< start (string-length source)) @@ -5938,14 +5970,21 @@ node extra message))] + [finding1 (if (and (not range-binding) + (symbolic-python-like-language? + language)) + (trim-python-module-finding + finding0 + source) + finding0)] [finding (if range-binding - finding0 + finding1 (maybe-extend-expression-semicolon rule pattern source node - finding0))] + finding1))] [focused (apply-rule-focus rule finding)]) (node-close! node) (loop (cdr xs) (append (reverse focused) acc)))))))) @@ -6687,6 +6726,22 @@ (finding-severity finding) extra)) +(def (finding-with-range finding source start end) + (let-values ([(start-line start-col) (offset->line-col source start)] + [(end-line end-col) (offset->line-col source end)]) + (make-finding + (finding-rule-id finding) + (finding-path finding) + start-line + start-col + end-line + end-col + start + end + (finding-message finding) + (finding-severity finding) + (finding-extra finding)))) + (def (finding-with-bindings rule finding bindings source) (let* ([match-text (source-slice source (finding-start-offset finding) new file mode 100755 --- /dev/null +++ b/tests/oracle/patterns-sweep.sh @@ -0,0 +1,208 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SEMGREP_ROOT="${SEMGREP_ROOT:-/Users/user/mine/semgrep}" +CURRENT_DEFAULT="$SEMGREP_ROOT/bin/semgrep" + +if [[ -z "${SEMGREP_CURRENT:-}" ]]; then + if [[ -x "$CURRENT_DEFAULT" ]]; then + SEMGREP_CURRENT="$CURRENT_DEFAULT" + elif command -v semgrep >/dev/null 2>&1; then + SEMGREP_CURRENT="$(command -v semgrep)" + else + echo "patterns-sweep: skipped; set SEMGREP_CURRENT or build $CURRENT_DEFAULT" >&2 + exit 0 + fi +fi + +if [[ -z "${SCHEME:-}" ]]; then + if [[ -x "$HOME/mine/jerboa/.chez/bin/scheme" ]]; then + SCHEME="$HOME/mine/jerboa/.chez/bin/scheme" + else + SCHEME="$HOME/mine/ChezScheme/tarm64osx/bin/tarm64osx/scheme" + fi +fi + +JERBOA_TREESITTER_HOME="${JERBOA_TREESITTER_HOME:-$HOME/mine/jerboa-treesitter}" +LIBDIRS="${LIBDIRS:-$ROOT/lib:$JERBOA_TREESITTER_HOME/lib:$HOME/mine/jerboa/lib}" +NORMALIZE="$ROOT/tests/oracle/normalize-findings.ss" +PATTERN_ROOT="${PATTERN_ROOT:-$SEMGREP_ROOT/tests/patterns}" +PATTERN_LANGS="${PATTERN_LANGS:-python}" +MAX_CASES="${MAX_CASES:-0}" +MAX_DIFFS="${MAX_DIFFS:-20}" +CASE_REGEX="${CASE_REGEX:-}" +LIST_MISMATCHES="${LIST_MISMATCHES:-0}" +TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/jerboa-semgrep-patterns.XXXXXX")" +trap 'rm -rf "$TMPDIR"' EXIT + +normalize_json() { + "$SCHEME" --libdirs "$LIBDIRS" --script "$NORMALIZE" | sort +} + +run_json() { + local name="$1" + local outfile="$2" + shift 2 + local rc=0 + "$@" >"$outfile" 2>"$TMPDIR/$name.stderr" || rc=$? + if [[ "$rc" -eq 0 || "$rc" -eq 1 ]]; then + return 0 + fi + return "$rc" +} + +rule_language_for_dir() { + case "$1" in + python) printf '%s\n' "python" ;; + js) printf '%s\n' "javascript" ;; + *) return 1 ;; + esac +} + +target_ext_for_dir() { + case "$1" in + python) printf '%s\n' "py" ;; + js) printf '%s\n' "js" ;; + *) return 1 ;; + esac +} + +write_rule_config() { + local language="$1" + local case_id="$2" + local pattern_file="$3" + local config="$4" + + { + printf '%s\n' "rules:" + printf '%s\n' " - id: oracle.patterns.${case_id//\//.}" + printf '%s\n' " languages: [$language]" + printf '%s\n' " message: pattern $case_id" + printf '%s\n' " severity: WARNING" + printf '%s\n' " pattern: |" + while IFS= read -r line || [[ -n "$line" ]]; do + printf ' %s\n' "$line" + done <"$pattern_file" + } >"$config" +} + +total=0 +pass=0 +mismatch=0 +skipped=0 +current_error=0 +jerboa_error=0 +diffs_shown=0 + +IFS=' ' read -r -a lang_dirs <<<"$PATTERN_LANGS" +for lang_dir in "${lang_dirs[@]}"; do + pattern_dir="$PATTERN_ROOT/$lang_dir" + if [[ ! -d "$pattern_dir" ]]; then + skipped=$((skipped + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "SKIP missing-dir $lang_dir" + fi + continue + fi + + if ! rule_language="$(rule_language_for_dir "$lang_dir")"; then + skipped=$((skipped + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "SKIP unsupported-lang $lang_dir" + fi + continue + fi + if ! target_ext="$(target_ext_for_dir "$lang_dir")"; then + skipped=$((skipped + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "SKIP unsupported-ext $lang_dir" + fi + continue + fi + + while IFS= read -r pattern; do + base="$(basename "$pattern" .sgrep)" + case_id="$lang_dir/$base" + if [[ -n "$CASE_REGEX" ]]; then + if [[ ! "$base" =~ $CASE_REGEX && ! "$case_id" =~ $CASE_REGEX ]]; then + continue + fi + fi + + target="$pattern_dir/$base.$target_ext" + if [[ ! -s "$pattern" || ! -f "$target" ]]; then + skipped=$((skipped + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "SKIP no-target-or-empty-pattern $case_id" + fi + continue + fi + + total=$((total + 1)) + if [[ "$MAX_CASES" -gt 0 && "$total" -gt "$MAX_CASES" ]]; then + total=$((total - 1)) + break 2 + fi + + config="$TMPDIR/${lang_dir}.${base}.yaml" + current_json="$TMPDIR/${lang_dir}.${base}.current.json" + jerboa_json="$TMPDIR/${lang_dir}.${base}.jerboa.json" + current_norm="$TMPDIR/${lang_dir}.${base}.current.norm" + jerboa_norm="$TMPDIR/${lang_dir}.${base}.jerboa.norm" + + write_rule_config "$rule_language" "$case_id" "$pattern" "$config" + + if ! SEMGREP_SEND_METRICS=off run_json "$lang_dir.$base.current" "$current_json" \ + "$SEMGREP_CURRENT" scan --quiet --json --config "$config" "$target"; then + current_error=$((current_error + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "ERROR current $case_id" + fi + if [[ "$diffs_shown" -lt "$MAX_DIFFS" ]]; then + echo "ERROR current $case_id" >&2 + sed 's/^/ /' "$TMPDIR/$lang_dir.$base.current.stderr" >&2 + diffs_shown=$((diffs_shown + 1)) + fi + continue + fi + + if ! JERBOA_TREE_SITTER_LIB="$JERBOA_TREESITTER_HOME" \ + DYLD_LIBRARY_PATH="$JERBOA_TREESITTER_HOME:${DYLD_LIBRARY_PATH:-}" \ + LD_LIBRARY_PATH="$JERBOA_TREESITTER_HOME:${LD_LIBRARY_PATH:-}" \ + run_json "$lang_dir.$base.jerboa" "$jerboa_json" \ + "$SCHEME" --libdirs "$LIBDIRS" --script "$ROOT/bin/semgrep.ss" \ + scan --json --config "$config" "$target"; then + jerboa_error=$((jerboa_error + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "ERROR jerboa $case_id" + fi + if [[ "$diffs_shown" -lt "$MAX_DIFFS" ]]; then + echo "ERROR jerboa $case_id" >&2 + sed 's/^/ /' "$TMPDIR/$lang_dir.$base.jerboa.stderr" >&2 + diffs_shown=$((diffs_shown + 1)) + fi + continue + fi + + normalize_json <"$current_json" >"$current_norm" + normalize_json <"$jerboa_json" >"$jerboa_norm" + + if diff -u "$current_norm" "$jerboa_norm" >"$TMPDIR/$lang_dir.$base.diff"; then + pass=$((pass + 1)) + else + mismatch=$((mismatch + 1)) + if [[ "$LIST_MISMATCHES" != "0" ]]; then + echo "MISMATCH $case_id" + fi + if [[ "$diffs_shown" -lt "$MAX_DIFFS" ]]; then + echo "MISMATCH $case_id" >&2 + cat "$TMPDIR/$lang_dir.$base.diff" >&2 + diffs_shown=$((diffs_shown + 1)) + fi + fi + done < <(find "$pattern_dir" -maxdepth 1 -type f -name '*.sgrep' | sort) +done + +echo "patterns-sweep: $pass passed, $mismatch mismatched, $jerboa_error jerboa errors, $current_error current errors, $skipped skipped, $total compared" +[[ "$mismatch" -eq 0 && "$jerboa_error" -eq 0 && "$current_error" -eq 0 ]] --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -224,6 +224,23 @@ (check (finding-start-line (car findings)) => 1) (check (finding-start-line (cadr findings)) => 2))) +(test-case "scan Python imported alias pattern trims fixture comments" + (let* ([alias-config + "rules:\n - id: demo.import.alias\n languages: [python]\n message: imported alias\n severity: WARNING\n pattern: |\n import os as $OS\n ...\n $OS.system(...)\n"] + [findings + (scan-config-string alias-config + "python" + "demo.py" + "#ERROR: match\nimport os as totally_not_os\n\ntotally_not_os.system('rm -rf /')\n")] + [finding (car findings)]) + (check (length findings) => 1) + (check (finding-start-line finding) => 2) + (check (finding-start-col finding) => 1) + (check (finding-start-offset finding) => 14) + (check (finding-end-line finding) => 4) + (check (finding-end-col finding) => 34) + (check (finding-end-offset finding) => 76))) + (test-case "scan Python ellipsis call arguments" (let* ([ellipsis-config "rules:\n - id: demo.eval.ellipsis\n languages: [python]\n message: eval call\n severity: WARNING\n pattern: eval(...)\n"]