Cover generic template upstream fixtures
ober
75d278dd5a0c72c2723917b60c2b2663223d8d1e
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,11 +1,11 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 05:23 MDT +Date: 2026-05-29 05:38 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`638f904 Cover XML upstream fixtures` +`7e821c9 Cover YAML upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -61,7 +61,7 @@ make test Result: ```text -221 tests, 221 passed, 0 failed +222 tests, 222 passed, 0 failed ``` Local oracle: @@ -136,6 +136,18 @@ Result: upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` +Focused generic template guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(inside|and_inside)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 2 compared +``` + Full upstream sweep: ```sh @@ -145,7 +157,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared +upstream-sweep: 254 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 256 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -166,65 +178,68 @@ currently accepts these fixtures and finds the expected `ruleid` comments. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.yl` -targets and a YAML text fallback. The three upstream `.yl` fixture files are -symlinks to corresponding `.yaml` files; packaged Semgrep refuses symlink -scanning roots, so the harness now resolves symlink targets before invoking -both scanners. +This checkpoint broadens the current sorted upstream sweep by adding `.ejs` +and `.mustache` targets for same-basename upstream template fixtures. Both +current cases use `languages: [generic]`, so they exercise the generic text +matcher rather than adding new parser-backed languages. -The scanner now treats YAML as a supported text-backed language that reuses the -generic matcher where possible and adds targeted compatibility fallbacks for -the current upstream YAML fixtures: duplicate rule IDs and JavaScript language -lists that omit TypeScript. +The scanner now treats whitespace adjacent to punctuation in generic patterns +as optional. This matches Semgrep's generic behavior for patterns such as +`src = "..."` against compact template markup such as `src="..."`, while still +requiring separators between word-like tokens such as `EXPOSE $PORT`. Newly covered upstream cases: ```text -inception -inception2 -js_no_ts +inside +and_inside ``` The full sweep moved from the previous checkpoint: ```text -249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared +252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared ``` to: ```text -252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared +254 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 256 compared ``` Recent prior checkpoint context: the previous code checkpoint broadened the -current sorted upstream sweep by adding `.xml` targets and an XML text -fallback. The scanner treats XML as a supported text-backed language that -reuses the generic matcher instead of requiring a tree-sitter parser. It also -adds two focused XML compatibility paths for the current upstream XML fixtures: -self-closing attribute metavariables with XML whitespace tolerance, and -`xml_children_ordered: false` matching for plugin children in either order. +current sorted upstream sweep by adding `.yl` targets and a YAML text fallback. +The three upstream `.yl` fixture files are symlinks to corresponding `.yaml` +files; packaged Semgrep refuses symlink scanning roots, so the harness resolves +symlink targets before invoking both scanners. ```text -option_xml_children_ordered -xml_metavar_comp -xml_metavar_comp2 +inception +inception2 +js_no_ts ``` That earlier full sweep moved from: ```text -246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared +249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared ``` to: ```text -249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared +252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared ``` Latest implementation changes: +- `src/semgrep/scan.ss` adds `generic-optional-separator-regex` and uses it + when pattern whitespace is adjacent to punctuation. +- `tests/oracle/upstream-sweep.sh` now includes `.ejs` and `.mustache` targets + in the same-basename rule/target corpus. + +Recent prior implementation changes: + - `src/semgrep/lang.ss` adds YAML to the non-parser supported language set so language inference can select it for `.yaml`/`.yl` targets. - `src/semgrep/scan.ss` treats YAML as a generic text-matching language. @@ -235,11 +250,8 @@ Latest implementation changes: it finds real `languages: [javascript]` entries while ignoring block scalar pattern text inside the rule YAML, and it reports the Semgrep-compatible range ending before the closing `]`. -- `tests/oracle/upstream-sweep.sh` now includes `.yl` targets in the +- `tests/oracle/upstream-sweep.sh` includes `.yl` targets in the same-basename rule/target corpus and resolves symlink targets before scanning. - -Recent prior implementation changes: - - `src/semgrep/lang.ss` adds XML to the non-parser supported language set so language inference can select it for `.xml` targets. - `src/semgrep/scan.ss` treats XML as a generic text-matching language. @@ -270,6 +282,7 @@ Other changes: New smoke coverage in `tests/smoke.ss`: ```text +scan generic punctuation-adjacent spaces are optional scan YAML duplicate id range scan YAML duplicate id focused lines scan YAML javascript language missing typescript @@ -319,6 +332,8 @@ xml_metavar_comp2 inception inception2 js_no_ts +inside +and_inside ``` Useful upstream files for the latest cleared cases: @@ -366,6 +381,10 @@ Useful upstream files for the latest cleared cases: /Users/user/mine/semgrep/tests/rules/inception2.yl /Users/user/mine/semgrep/tests/rules/js_no_ts.yaml /Users/user/mine/semgrep/tests/rules/js_no_ts.yl +/Users/user/mine/semgrep/tests/rules/inside.yaml +/Users/user/mine/semgrep/tests/rules/inside.ejs +/Users/user/mine/semgrep/tests/rules/and_inside.yaml +/Users/user/mine/semgrep/tests/rules/and_inside.mustache ``` Important observations: @@ -418,14 +437,21 @@ Important observations: - `js_no_ts` expects only the real `languages: [javascript]` key to match, not the rule's own block scalar pattern text. The finding range ends before the closing `]`. +- `inside` expects one EJS template finding for + `<%=reactDemoIndexBundleJs %>` inside a compact `src="..."` attribute. The + rule pattern writes this as `src = "..."`, so generic punctuation-adjacent + pattern spaces must be optional. +- `and_inside` is a current zero-finding parity case despite the inline + `ruleid` comment in the upstream fixture. Keep the oracle result authoritative + unless Semgrep current changes. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 254 rule/target pairs and has +The current full sorted upstream sweep compared 256 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared +upstream-sweep: 254 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 256 compared ``` Refresh the frontier with: @@ -437,14 +463,14 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Unsupported Upstream Frontier The upstream `tests/rules` same-basename corpus currently contains 444 rule -configs. The sweep can compare 254 because `target_for_rule` only considers +configs. The sweep can compare 256 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx dockerfile html xml yl +py js json txt generic gem ts tsx dockerfile html xml yl ejs mustache ``` -That leaves 190 upstream configs outside the current oracle surface. These are +That leaves 188 upstream configs outside the current oracle surface. These are not known Jerboa mismatches; most are skipped because the static `jerboa-treesitter` bridge does not ship those language parsers yet, or because the harness does not yet map that suffix. @@ -467,8 +493,6 @@ Skipped target-extension buckets: 4 kt 1 tpl 1 swift - 1 mustache - 1 ejs 1 cs ``` @@ -506,8 +530,7 @@ rs: macro_arg_taint, metavar_type_not_rust, misc_macro_call, Lower-count skipped buckets are still important for parity because they exercise target language behavior that generic matching cannot cover: Terraform (`tf`), -Julia (`jl`), C, Move, Dart, Ruby, Kotlin, template/PHP (`tpl`), Swift, -Mustache, EJS, and C#. +Julia (`jl`), C, Move, Dart, Ruby, Kotlin, template/PHP (`tpl`), Swift, and C#. Parser coverage constraints: @@ -523,6 +546,9 @@ Parser coverage constraints: - YAML is supported as a text-backed fallback using the generic matcher plus targeted fallbacks for the current YAML upstream fixtures; it is not backed by a YAML parser. +- EJS and Mustache are included in the same-basename upstream sweep only for + `languages: [generic]` fixtures. They are not recognized as standalone target + languages. - TypeScript and TSX use the JavaScript parser fallback; decorated members are handled by a targeted scanner fallback, not a full TypeScript grammar. - `/Users/user/mine/jerboa-treesitter` currently vendors only JSON, Python, and @@ -534,7 +560,7 @@ Parser coverage constraints: ## Recommended Next Target -There is no known Jerboa mismatch in the current 254-case sorted upstream +There is no known Jerboa mismatch in the current 256-case sorted upstream sweep. The two packaged-Semgrep `current_error` cases are `anywhere_global` and `anywhere_metavar`, both rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -754,6 +754,8 @@ (string=? language "tsx"))) (def generic-separator-regex "(?:\\s+|//[^\\n]*(?:\\n|$)|/\\*(?:.|\\n)*?\\*/)+") + (def generic-optional-separator-regex + "(?:\\s+|//[^\\n]*(?:\\n|$)|/\\*(?:.|\\n)*?\\*/)*") (def (regex-special-char? ch) (let loop ([xs '(#\\ #\. #\^ #\$ #\| #\? #\* #\+ #\( #\) #\[ #\] #\{ #\})]) @@ -804,6 +806,17 @@ (or (char-alphabetic? ch) (char-numeric? ch) (char=? ch #\_))) + (def (generic-space-punctuation? ch) + (and (not (char-whitespace? ch)) + (not (generic-word-char? ch)) + (not (char=? ch #\$)))) + (def (generic-optional-space-context? source start end) + (let ([prev (and (> start 0) + (string-ref source (- start 1)))] + [next (and (< end (string-length source)) + (string-ref source end))]) + (or (and prev (generic-space-punctuation? prev)) + (and next (generic-space-punctuation? next))))) (def (generic-capture-name source marker-start @@ -837,7 +850,14 @@ (loop j parts captures) (loop j - (cons generic-separator-regex parts) + (cons + (if (generic-optional-space-context? + source + i + j) + generic-optional-separator-regex + generic-separator-regex) + parts) captures))))] [(and (char=? (string-ref source i) #\$) (< (+ i 4) len) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -3,11 +3,11 @@ ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") ("src/semgrep/lang.ss" . "4BE7F333D7C9DC94") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "776015E81972DACC") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/scan.ss" . "5F40795058799B3D") ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") ("src/semgrep/cli.ss" . "F4EC2626A953778F")) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -831,6 +831,9 @@ (def generic-separator-regex "(?:\\s+|//[^\\n]*(?:\\n|$)|/\\*(?:.|\\n)*?\\*/)+") +(def generic-optional-separator-regex + "(?:\\s+|//[^\\n]*(?:\\n|$)|/\\*(?:.|\\n)*?\\*/)*") + (def (regex-special-char? ch) (let loop ([xs '(#\\ #\. #\^ #\$ #\| #\? #\* #\+ #\( #\) #\[ #\] #\{ #\})]) (and (not (null? xs)) @@ -892,6 +895,17 @@ (char-numeric? ch) (char=? ch #\_))) +(def (generic-space-punctuation? ch) + (and (not (char-whitespace? ch)) + (not (generic-word-char? ch)) + (not (char=? ch #\$)))) + +(def (generic-optional-space-context? source start end) + (let ([prev (and (> start 0) (string-ref source (- start 1)))] + [next (and (< end (string-length source)) (string-ref source end))]) + (or (and prev (generic-space-punctuation? prev)) + (and next (generic-space-punctuation? next))))) + (def (generic-capture-name source marker-start name-start end) (if (and (< (+ marker-start 3) name-start) (char=? (string-ref source (+ marker-start 1)) #\.) @@ -919,7 +933,15 @@ (and (pair? parts) (string=? (car parts) "((?:.|\\n)*?)"))) (loop j parts captures) - (loop j (cons generic-separator-regex parts) captures))))] + (loop j + (cons (if (generic-optional-space-context? + source + i + j) + generic-optional-separator-regex + generic-separator-regex) + parts) + captures))))] [(and (char=? (string-ref source i) #\$) (< (+ i 4) len) (char=? (string-ref source (+ i 1)) #\.) --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -55,7 +55,7 @@ target_for_rule() { local ext local candidate local link - for ext in py js json txt generic gem ts tsx dockerfile html xml yl; do + for ext in py js json txt generic gem ts tsx dockerfile html xml yl ejs mustache; do candidate="$RULE_DIR/$base.$ext" if [[ -f "$candidate" ]]; then if [[ -L "$candidate" ]]; then --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -746,6 +746,22 @@ "a // ignore me\nb\n")]) (check (length findings) => 1))) +(test-case "scan generic punctuation-adjacent spaces are optional" + (let* ([generic-config + "rules:\n - id: demo.generic.template\n languages: [generic]\n message: template src\n severity: WARNING\n patterns:\n - pattern-inside: <script ...>\n - pattern-either:\n - pattern-inside: src = '...'\n - pattern-inside: src = \"...\"\n - pattern: <% ... >\n"] + [source + "<script src=\"./<%=reactDemoIndexBundleJs %>\" crossorigin=\"anonymous\"></script>\n"] + [findings + (scan-config-string generic-config + "generic" + "demo.ejs" + source)]) + (check (length findings) => 1) + (check (substring source + (finding-start-offset (car findings)) + (finding-end-offset (car findings))) + => "<%=reactDemoIndexBundleJs %>"))) + (test-case "scan Dockerfile patterns with generic text fallback" (let* ([dockerfile-config "rules:\n - id: demo.dockerfile.port\n languages: [dockerfile]\n message: invalid port\n severity: ERROR\n patterns:\n - pattern: EXPOSE $PORT\n - metavariable-comparison:\n metavariable: $PORT\n comparison: $PORT > 65535\n strip: true\n"]