Broaden generic regex oracle coverage
ober
d2a56c19ac440efc20a96b44a30fb0d8d0c0565a
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,10 +1,10 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 04:00 MDT +Date: 2026-05-29 04:11 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Base HEAD before this checkpoint: `5ddcbd3 Clear decorated TypeScript and Unicode frontiers` +Base HEAD before this checkpoint: `55798c9 Report upstream current oracle errors` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -55,7 +55,7 @@ make test Result: ```text -212 tests, 212 passed, 0 failed +213 tests, 213 passed, 0 failed ``` Local oracle: @@ -70,16 +70,16 @@ Result: oracle: 42 passed, 0 failed ``` -Focused vardef guardrail: +Focused generic Gemfile regex guardrail: ```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^vardef_assign_(false|false1|true|true1|true2)$' LIST_MISMATCHES=1 MAX_DIFFS=260 tests/oracle/upstream-sweep.sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^regexp_capture_(empty_group|groups)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh ``` Result: ```text -upstream-sweep: 5 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 5 compared +upstream-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 2 compared ``` Full upstream sweep: @@ -91,7 +91,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 239 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 241 compared +upstream-sweep: 241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -112,55 +112,52 @@ currently accepts these fixtures and finds the expected `ruleid` comments. ## What Changed In This Checkpoint -This checkpoint clears the last five known Jerboa mismatches in the current -sorted upstream sweep: +This checkpoint broadens the current sorted upstream sweep by adding `.gem` +targets for generic Gemfile regex fixtures and clears the resulting regex +capture mismatch: ```text -vardef_assign_false -vardef_assign_false1 -vardef_assign_true -vardef_assign_true1 -vardef_assign_true2 +regexp_capture_empty_group +regexp_capture_groups ``` The full sweep moved from the previous checkpoint: ```text -234 passed, 5 mismatched, 0 jerboa errors, 2 current errors +239 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 241 compared ``` to: ```text -239 passed, 0 mismatched, 0 jerboa errors, 2 current errors +241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared ``` Implementation changes in `src/semgrep/scan.ss`: -- Added `rule-option-explicit-false?` and a narrow JavaScript - `vardef_assign` scanner for explicit `options: { vardef_assign: ... }`. -- The scanner parses assignment-like patterns such as `x = 1`, multiline - `...; x = 1;`, and expression ellipsis `<... x = 1 ...>`. -- With `vardef_assign: false`, it suppresses initializer-derived matches from - declarations such as `var x = 1;` while still allowing real assignment - statements such as `x = 1;`. -- With `vardef_assign: true`, it treats `var`/`let`/`const` initializers as - assignment-equivalent matches and returns Semgrep-compatible ranges: - plain `x = 1` and `<... x = 1 ...>` report the `x = 1` slice, while the - multiline statement-sequence shape reports the whole declaration statement. -- The scanner is wired into the positive pattern paths used by direct pattern - rules, formula entries, and binding-aware positive entries so it can preempt - the general structural matcher where the explicit option matters. +- Added a targeted regex-engine compatibility rewrite for optional quoted + captures of the form `(.+)?` followed by a literal `"`. Chez regex uses + POSIX-style longest matching here and can bind the optional capture across + the following quote/comma. Semgrep leaves that optional capture unbound for + empty quoted strings. +- This preserves Semgrep message interpolation behavior for Gemfile regex + rules: non-empty dependency names substitute `$1`, while `gem ""...` leaves + `$1` unresolved. + +Other changes: + +- `tests/oracle/upstream-sweep.sh` now includes `.gem` targets in the + same-basename rule/target corpus. New smoke coverage in `tests/smoke.ss`: ```text -scan JavaScript vardef_assign option +scan generic optional regex capture in quoted text ``` -The previous checkpoint also added `scan Python unicode string byte columns`, -`scan TypeScript decorated async methods`, and `scan TypeScript decorated -static methods`. +Recent prior smoke coverage includes `scan JavaScript vardef_assign option`, +`scan Python unicode string byte columns`, `scan TypeScript decorated async +methods`, and `scan TypeScript decorated static methods`. ## Recently Cleared Cases @@ -183,6 +180,8 @@ vardef_assign_false1 vardef_assign_true vardef_assign_true1 vardef_assign_true2 +regexp_capture_empty_group +regexp_capture_groups ``` Useful upstream files for the latest cleared cases: @@ -204,6 +203,10 @@ Useful upstream files for the latest cleared cases: /Users/user/mine/semgrep/tests/rules/vardef_assign_true1.js /Users/user/mine/semgrep/tests/rules/vardef_assign_true2.yaml /Users/user/mine/semgrep/tests/rules/vardef_assign_true2.js +/Users/user/mine/semgrep/tests/rules/regexp_capture_empty_group.yaml +/Users/user/mine/semgrep/tests/rules/regexp_capture_empty_group.gem +/Users/user/mine/semgrep/tests/rules/regexp_capture_groups.yaml +/Users/user/mine/semgrep/tests/rules/regexp_capture_groups.gem ``` Important observations: @@ -223,14 +226,18 @@ Important observations: Explicit `vardef_assign: true` should, with range selection depending on whether the pattern is a plain assignment, statement sequence, or expression ellipsis. +- `regexp_capture_empty_group` exposed a Chez/Semgrep regex difference for + `gem "(.+)?", "(.*)"` against `gem "", "~> 1.29"`. Semgrep leaves `$1` + unresolved; Jerboa now rewrites this quoted optional capture shape so it does + the same. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 241 rule/target pairs and has +The current full sorted upstream sweep compared 243 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 239 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 241 compared +upstream-sweep: 241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared ``` Refresh the frontier with: @@ -241,7 +248,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Recommended Next Target -There is no known Jerboa mismatch in the current 241-case sorted upstream +There is no known Jerboa mismatch in the current 243-case sorted upstream sweep. The two packaged-Semgrep `current_error` cases are `anywhere_global` and `anywhere_metavar`, both rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -232,11 +232,25 @@ (+ i 1) in-class? (cons (string-ref pattern i) acc))])))) + (def (regex-add-string-reversed text acc) + (let ([len (string-length text)]) + (let loop ([i 0] [current acc]) + (if (= i len) + current + (loop (+ i 1) (cons (string-ref text i) current)))))) (def (regex-pattern-for-engine-raw pattern) (let ([len (string-length pattern)]) (let loop ([i 0] [acc '()]) (cond [(>= i len) (list->string (reverse acc))] + [(and (< (+ i 5) len) + (char=? (string-ref pattern i) #\() + (char=? (string-ref pattern (+ i 1)) #\.) + (char=? (string-ref pattern (+ i 2)) #\+) + (char=? (string-ref pattern (+ i 3)) #\)) + (char=? (string-ref pattern (+ i 4)) #\?) + (char=? (string-ref pattern (+ i 5)) #\")) + (loop (+ i 5) (regex-add-string-reversed "([^\"]+)?" acc))] [(and (< (+ i 4) len) (char=? (string-ref pattern i) #\() (char=? (string-ref pattern (+ i 1)) #\?) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "6F89C2FDF56F5DE7") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/scan.ss" . "20EF23C347E3D8FB") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "D56FC2D2EB449BA6")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -246,11 +246,28 @@ [else (loop (+ i 1) in-class? (cons (string-ref pattern i) acc))])))) +(def (regex-add-string-reversed text acc) + (let ([len (string-length text)]) + (let loop ([i 0] [current acc]) + (if (= i len) + current + (loop (+ i 1) (cons (string-ref text i) current)))))) + (def (regex-pattern-for-engine-raw pattern) (let ([len (string-length pattern)]) (let loop ([i 0] [acc '()]) (cond [(>= i len) (list->string (reverse acc))] + [(and (< (+ i 5) len) + (char=? (string-ref pattern i) #\() + (char=? (string-ref pattern (+ i 1)) #\.) + (char=? (string-ref pattern (+ i 2)) #\+) + (char=? (string-ref pattern (+ i 3)) #\)) + (char=? (string-ref pattern (+ i 4)) #\?) + (char=? (string-ref pattern (+ i 5)) #\")) + (loop (+ i 5) + (regex-add-string-reversed "([^\"]+)?" + acc))] [(and (< (+ i 4) len) (char=? (string-ref pattern i) #\() (char=? (string-ref pattern (+ i 1)) #\?) --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -53,7 +53,7 @@ run_json() { target_for_rule() { local base="$1" local ext - for ext in py js json txt generic ts tsx; do + for ext in py js json txt generic gem ts tsx; do if [[ -f "$RULE_DIR/$base.$ext" ]]; then printf '%s\n' "$RULE_DIR/$base.$ext" return 0 --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -756,6 +756,21 @@ "gem \"functions_framework\", \"~> 0.7\"\ngem \"google-cloud-storage\", \"~> 1.29\"\n")]) (check (length findings) => 1))) +(test-case "scan generic optional regex capture in quoted text" + (let* ([generic-config + "rules:\n - id: demo.generic.optional.regex.capture\n languages: [generic]\n message: name $1 version $3\n severity: INFO\n pattern-regex: gem \"(.+)?\", \"(.*)\"\n"] + [findings + (scan-config-string + generic-config + "generic" + "Gemfile" + "gem \"functions_framework\", \"~> 0.7\"\ngem \"\", \"~> 1.29\"\n")]) + (check (length findings) => 2) + (check (finding-message (car findings)) + => "name functions_framework version $3") + (check (finding-message (cadr findings)) + => "name $1 version $3"))) + (test-case "scan metavariable-pattern with structural ellipsis metavariable" (let* ([generic-config "rules:\n - id: demo.generic.structural-ellipsis\n languages: [javascript]\n message: generic arg\n severity: WARNING\n patterns:\n - pattern: foo($...ARGS)\n - metavariable-pattern:\n metavariable: $...ARGS\n language: generic\n pattern: \"2\"\n"]