Cover HTML upstream fixtures
ober
0f4a6ab0f197f4e66f61cd5a5b2430824abfc140
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,11 +1,11 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 04:34 MDT +Date: 2026-05-29 04:47 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`d0ef0bd Document parity handoff frontier` +`0f578cf Cover Dockerfile upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -61,7 +61,7 @@ make test Result: ```text -214 tests, 214 passed, 0 failed +216 tests, 216 passed, 0 failed ``` Local oracle: @@ -100,6 +100,18 @@ Result: upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` +Focused HTML guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(metavar_ellipsis_xmls|pattern_not_regex)$' LIST_MISMATCHES=1 MAX_DIFFS=160 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 2 compared +``` + Full upstream sweep: ```sh @@ -109,7 +121,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared +upstream-sweep: 246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -130,63 +142,64 @@ currently accepts these fixtures and finds the expected `ruleid` comments. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding -`.dockerfile` targets and a Dockerfile text fallback. The scanner now treats -Dockerfile as a supported text-backed language that reuses the generic matcher -instead of requiring a tree-sitter parser. It also allows generic word -metavariables to bind numeric atoms, which is needed for rules such as -`EXPOSE $PORT`. +This checkpoint broadens the current sorted upstream sweep by adding `.html` +targets and an HTML text fallback. The scanner now treats HTML as a supported +text-backed language that reuses the generic matcher instead of requiring a +tree-sitter parser. The CLI also now keeps single-language `generic` or `regex` +configs in that text mode before falling back to target-extension inference, so +generic rules still scan `.html` targets as generic. Newly covered upstream cases: ```text -compare-exposed-port -option_implicit_exprstmt -spacegrep_metavarbug +metavar_ellipsis_xmls +pattern_not_regex ``` The full sweep moved from the previous checkpoint: ```text -241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared +244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared ``` to: ```text -244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared +246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared ``` Recent prior checkpoint context: the previous code checkpoint broadened the -current sorted upstream sweep by adding `.gem` targets for generic Gemfile regex -fixtures and cleared the resulting regex capture mismatch: +current sorted upstream sweep by adding `.dockerfile` targets and a Dockerfile +text fallback. It also allowed generic word metavariables to bind numeric atoms, +which was needed for rules such as `EXPOSE $PORT`. ```text -regexp_capture_empty_group -regexp_capture_groups +compare-exposed-port +option_implicit_exprstmt +spacegrep_metavarbug ``` That earlier full sweep moved from: ```text -239 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 241 compared +241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared ``` to: ```text -241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared +244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared ``` Latest implementation changes: -- `src/semgrep/lang.ss` adds Dockerfile to the non-parser supported language - set so language inference can select it for `Dockerfile`/`.dockerfile` - targets. -- `src/semgrep/scan.ss` treats Dockerfile as a generic text-matching language. -- Generic `$X` captures now accept numeric word atoms as well as identifier-like - atoms. -- `tests/oracle/upstream-sweep.sh` now includes `.dockerfile` targets in the +- `src/semgrep/lang.ss` adds HTML to the non-parser supported language set so + language inference can select it for `.html` targets. +- `src/semgrep/scan.ss` treats HTML as a generic text-matching language. +- `src/semgrep/cli.ss` prefers a single `generic` or `regex` config language + before target-extension inference, preserving generic scans over known + extensions such as `.html`. +- `tests/oracle/upstream-sweep.sh` now includes `.html` targets in the same-basename rule/target corpus. Earlier implementation changes in `src/semgrep/scan.ss`: @@ -208,6 +221,8 @@ Other changes: New smoke coverage in `tests/smoke.ss`: ```text +scan HTML script body with generic text fallback +cli keeps single generic config for html target scan Dockerfile patterns with generic text fallback scan generic optional regex capture in quoted text ``` @@ -242,6 +257,8 @@ regexp_capture_groups compare-exposed-port option_implicit_exprstmt spacegrep_metavarbug +metavar_ellipsis_xmls +pattern_not_regex ``` Useful upstream files for the latest cleared cases: @@ -273,6 +290,10 @@ Useful upstream files for the latest cleared cases: /Users/user/mine/semgrep/tests/rules/option_implicit_exprstmt.dockerfile /Users/user/mine/semgrep/tests/rules/spacegrep_metavarbug.yaml /Users/user/mine/semgrep/tests/rules/spacegrep_metavarbug.dockerfile +/Users/user/mine/semgrep/tests/rules/metavar_ellipsis_xmls.yaml +/Users/user/mine/semgrep/tests/rules/metavar_ellipsis_xmls.html +/Users/user/mine/semgrep/tests/rules/pattern_not_regex.yaml +/Users/user/mine/semgrep/tests/rules/pattern_not_regex.html ``` Important observations: @@ -302,14 +323,19 @@ Important observations: matches `RUN b` without matching `RUN a && b`. - `spacegrep_metavarbug` is a generic rule over a `.dockerfile` target; it remains a zero-finding parity case. +- `metavar_ellipsis_xmls` captures a `<script ...>...</script>` body in an HTML + rule, then applies a JavaScript `metavariable-pattern` to the captured text. +- `pattern_not_regex` is a generic rule over an `.html` target; CLI language + selection must keep the generic config language instead of switching to HTML + by extension. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 246 rule/target pairs and has +The current full sorted upstream sweep compared 248 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared +upstream-sweep: 246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared ``` Refresh the frontier with: @@ -321,14 +347,14 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Unsupported Upstream Frontier The upstream `tests/rules` same-basename corpus currently contains 444 rule -configs. The sweep can compare 246 because `target_for_rule` only considers +configs. The sweep can compare 248 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx dockerfile +py js json txt generic gem ts tsx dockerfile html ``` -That leaves 198 upstream configs outside the current oracle surface. These are +That leaves 196 upstream configs outside the current oracle surface. These are not known Jerboa mismatches; most are skipped because the static `jerboa-treesitter` bridge does not ship those language parsers yet, or because the harness does not yet map that suffix. @@ -351,7 +377,6 @@ Skipped target-extension buckets: 4 kt 3 yl 3 xml - 2 html 1 tpl 1 swift 1 mustache @@ -393,8 +418,8 @@ rs: macro_arg_taint, metavar_type_not_rust, misc_macro_call, Lower-count skipped buckets are still important for parity because they exercise target language behavior that generic matching cannot cover: Terraform (`tf`), -Julia (`jl`), C, Move, Dart, Ruby, Kotlin, YAML (`yl`), XML, HTML, -template/PHP (`tpl`), Swift, Mustache, EJS, and C#. +Julia (`jl`), C, Move, Dart, Ruby, Kotlin, YAML (`yl`), XML, template/PHP +(`tpl`), Swift, Mustache, EJS, and C#. Parser coverage constraints: @@ -402,6 +427,8 @@ Parser coverage constraints: `json`, `python`, `js`, and `ts`. - Dockerfile is supported as a text-backed fallback using the generic matcher; it is not backed by a Dockerfile parser. +- HTML is supported as a text-backed fallback using the generic matcher; it is + not backed by an HTML parser. - TypeScript and TSX use the JavaScript parser fallback; decorated members are handled by a targeted scanner fallback, not a full TypeScript grammar. - `/Users/user/mine/jerboa-treesitter` currently vendors only JSON, Python, and @@ -413,7 +440,7 @@ Parser coverage constraints: ## Recommended Next Target -There is no known Jerboa mismatch in the current 246-case sorted upstream +There is no known Jerboa mismatch in the current 248-case sorted upstream sweep. The two packaged-Semgrep `current_error` cases are `anywhere_global` and `anywhere_metavar`, both rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. --- a/lib/semgrep/cli.sls +++ b/lib/semgrep/cli.sls @@ -274,9 +274,18 @@ (and (not (null? languages)) (null? (cdr languages)) (car languages)))) + (def (single-text-config-language rules) + (let ([language (single-config-language rules)]) + (and language + (let ([canonical (or (canonical-language language) + language)]) + (and (or (string=? canonical "generic") + (string=? canonical "regex")) + language))))) (def (scan-one config-path language-opt target-path) (let* ([rules (parse-config-file config-path)] [language (or language-opt + (single-text-config-language rules) (and (not (string=? target-path "-")) (guess-language-from-path target-path)) (single-config-language rules))]) --- a/lib/semgrep/lang.sls +++ b/lib/semgrep/lang.sls @@ -12,7 +12,7 @@ (prefix (semgrep schema lang) schema-)) (def parse-supported-language-ids '("json" "python" "js" "ts")) - (def text-supported-language-ids '("dockerfile")) + (def text-supported-language-ids '("dockerfile" "html")) (def (string-member? needle xs) (and (not (null? xs)) (or (string=? needle (car xs)) --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -729,7 +729,8 @@ (let ([canonical (or (canonical-language language) language)]) (or (string=? canonical "generic") - (string=? canonical "dockerfile")))) + (string=? canonical "dockerfile") + (string=? canonical "html")))) (def (javascript-like-language? language) (or (string=? language "javascript") (string=? language "js") --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -1,13 +1,13 @@ (("src/semgrep/output/sarif.ss" . "E935456E4B1921FB") ("src/semgrep/rule/parse-rule.ss" . "EC5BDBE8CB185021") ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") - ("src/semgrep/lang.ss" . "8076C5561FECD75D") + ("src/semgrep/lang.ss" . "2C1880C5B4121698") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "535A03F3724981B1") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/scan.ss" . "E80A335AFC65DC8") ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") - ("src/semgrep/cli.ss" . "84899658C448ADBF")) + ("src/semgrep/cli.ss" . "BA1521EF48C3ED13")) --- a/src/semgrep/cli.ss +++ b/src/semgrep/cli.ss @@ -310,9 +310,18 @@ (null? (cdr languages)) (car languages)))) +(def (single-text-config-language rules) + (let ([language (single-config-language rules)]) + (and language + (let ([canonical (or (canonical-language language) language)]) + (and (or (string=? canonical "generic") + (string=? canonical "regex")) + language))))) + (def (scan-one config-path language-opt target-path) (let* ([rules (parse-config-file config-path)] [language (or language-opt + (single-text-config-language rules) (and (not (string=? target-path "-")) (guess-language-from-path target-path)) (single-config-language rules))]) --- a/src/semgrep/lang.ss +++ b/src/semgrep/lang.ss @@ -13,7 +13,7 @@ '("json" "python" "js" "ts")) (def text-supported-language-ids - '("dockerfile")) + '("dockerfile" "html")) (def (string-member? needle xs) (and (not (null? xs)) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -802,7 +802,8 @@ (def (generic-language? language) (let ([canonical (or (canonical-language language) language)]) (or (string=? canonical "generic") - (string=? canonical "dockerfile")))) + (string=? canonical "dockerfile") + (string=? canonical "html")))) (def (javascript-like-language? language) (or (string=? language "javascript") --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -53,7 +53,7 @@ run_json() { target_for_rule() { local base="$1" local ext - for ext in py js json txt generic gem ts tsx dockerfile; do + for ext in py js json txt generic gem ts tsx dockerfile html; do if [[ -f "$RULE_DIR/$base.$ext" ]]; then printf '%s\n' "$RULE_DIR/$base.$ext" return 0 --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -762,6 +762,30 @@ (finding-end-offset (car findings))) => "EXPOSE 65536"))) +(test-case "scan HTML script body with generic text fallback" + (let* ([html-config + "rules:\n - id: demo.html.script\n languages: [html]\n message: script console\n severity: WARNING\n patterns:\n - pattern: |\n <script ...>$...JS</script>\n - metavariable-pattern:\n language: javascript\n metavariable: $...JS\n patterns:\n - pattern: console.log(...)\n"] + [findings + (scan-config-string html-config + "html" + "demo.html" + "<script type=\"text/javascript\">\n console.log(foobar)\n</script>\n")]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 1))) + +(test-case "cli keeps single generic config for html target" + (let ([rule-path "/tmp/jerboa-semgrep-generic-html.yml"] + [target-path "/tmp/jerboa-semgrep-generic-html.html"]) + (write-file-string + rule-path + "rules:\n - id: demo.generic.html\n languages: [generic]\n message: pre tag\n severity: ERROR\n patterns:\n - pattern-inside: <$TAG ... >\n - pattern-not-regex: 'div'\n") + (write-file-string + target-path + "<div class=\"email-text\">\n <pre>{{ body }}</pre>\n</div>\n") + (check (main (list "scan" "--config" rule-path target-path)) => 1) + (delete-file rule-path) + (delete-file target-path))) + (test-case "scan metavariable-pattern with generic language" (let* ([generic-config "rules:\n - id: demo.generic.metavariable-pattern\n languages: [generic]\n message: generic dependency\n severity: WARNING\n patterns:\n - pattern-regex: \"gem \\\"(?P<ONE>.*)\\\", \\\"(.*)\\\"\"\n - metavariable-pattern:\n metavariable: $ONE\n language: generic\n pattern: google-cloud-storage\n"] @@ -2246,6 +2270,7 @@ (check (guess-language-from-path "demo.json") => "json") (check (guess-language-from-path "Dockerfile") => "dockerfile") (check (guess-language-from-path "demo.dockerfile") => "dockerfile") + (check (guess-language-from-path "demo.html") => "html") (check (guess-language-from-path "demo.rb") => #f)) (test-case "generated language registry"