Cover Dockerfile upstream fixtures
ober
0f578cfcc0ce3485dd66c79fdf9a28b1b7a61991
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,11 +1,11 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 04:16 MDT +Date: 2026-05-29 04:34 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` -Base HEAD before this final handoff update: -`d2a56c1 Broaden generic regex oracle coverage` +Base HEAD before this checkpoint: +`d0ef0bd Document parity handoff frontier` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -33,13 +33,18 @@ The recent code checkpoint includes: ```text HANDOFF_OPUS_4_8.md +src/semgrep/lang.ss +src/semgrep/cli.ss src/semgrep/scan.ss +lib/semgrep/lang.sls +lib/semgrep/cli.sls lib/semgrep/scan.sls src/.jerbuild-hashes +tests/oracle/upstream-sweep.sh tests/smoke.ss ``` -`lib/semgrep/scan.sls` and `src/.jerbuild-hashes` are generated by +The `lib/semgrep/*.sls` files and `src/.jerbuild-hashes` are generated by `make test`/`make oracle`; they are tracked and must be committed with source changes. @@ -56,7 +61,7 @@ make test Result: ```text -213 tests, 213 passed, 0 failed +214 tests, 214 passed, 0 failed ``` Local oracle: @@ -83,6 +88,18 @@ Result: upstream-sweep: 2 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 2 compared ``` +Focused Dockerfile guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(compare-exposed-port|option_implicit_exprstmt|spacegrep_metavarbug)$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared +``` + Full upstream sweep: ```sh @@ -92,7 +109,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared +upstream-sweep: 244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -113,16 +130,43 @@ currently accepts these fixtures and finds the expected `ruleid` comments. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.gem` -targets for generic Gemfile regex fixtures and clears the resulting regex -capture mismatch: +This checkpoint broadens the current sorted upstream sweep by adding +`.dockerfile` targets and a Dockerfile text fallback. The scanner now treats +Dockerfile as a supported text-backed language that reuses the generic matcher +instead of requiring a tree-sitter parser. It also allows generic word +metavariables to bind numeric atoms, which is needed for rules such as +`EXPOSE $PORT`. + +Newly covered upstream cases: + +```text +compare-exposed-port +option_implicit_exprstmt +spacegrep_metavarbug +``` + +The full sweep moved from the previous checkpoint: + +```text +241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared +``` + +to: + +```text +244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared +``` + +Recent prior checkpoint context: the previous code checkpoint broadened the +current sorted upstream sweep by adding `.gem` targets for generic Gemfile regex +fixtures and cleared the resulting regex capture mismatch: ```text regexp_capture_empty_group regexp_capture_groups ``` -The full sweep moved from the previous checkpoint: +That earlier full sweep moved from: ```text 239 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 241 compared @@ -134,7 +178,18 @@ to: 241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared ``` -Implementation changes in `src/semgrep/scan.ss`: +Latest implementation changes: + +- `src/semgrep/lang.ss` adds Dockerfile to the non-parser supported language + set so language inference can select it for `Dockerfile`/`.dockerfile` + targets. +- `src/semgrep/scan.ss` treats Dockerfile as a generic text-matching language. +- Generic `$X` captures now accept numeric word atoms as well as identifier-like + atoms. +- `tests/oracle/upstream-sweep.sh` now includes `.dockerfile` targets in the + same-basename rule/target corpus. + +Earlier implementation changes in `src/semgrep/scan.ss`: - Added a targeted regex-engine compatibility rewrite for optional quoted captures of the form `(.+)?` followed by a literal `"`. Chez regex uses @@ -153,6 +208,7 @@ Other changes: New smoke coverage in `tests/smoke.ss`: ```text +scan Dockerfile patterns with generic text fallback scan generic optional regex capture in quoted text ``` @@ -183,6 +239,9 @@ vardef_assign_true1 vardef_assign_true2 regexp_capture_empty_group regexp_capture_groups +compare-exposed-port +option_implicit_exprstmt +spacegrep_metavarbug ``` Useful upstream files for the latest cleared cases: @@ -208,6 +267,12 @@ Useful upstream files for the latest cleared cases: /Users/user/mine/semgrep/tests/rules/regexp_capture_empty_group.gem /Users/user/mine/semgrep/tests/rules/regexp_capture_groups.yaml /Users/user/mine/semgrep/tests/rules/regexp_capture_groups.gem +/Users/user/mine/semgrep/tests/rules/compare-exposed-port.yaml +/Users/user/mine/semgrep/tests/rules/compare-exposed-port.dockerfile +/Users/user/mine/semgrep/tests/rules/option_implicit_exprstmt.yaml +/Users/user/mine/semgrep/tests/rules/option_implicit_exprstmt.dockerfile +/Users/user/mine/semgrep/tests/rules/spacegrep_metavarbug.yaml +/Users/user/mine/semgrep/tests/rules/spacegrep_metavarbug.dockerfile ``` Important observations: @@ -231,14 +296,20 @@ Important observations: `gem "(.+)?", "(.*)"` against `gem "", "~> 1.29"`. Semgrep leaves `$1` unresolved; Jerboa now rewrites this quoted optional capture shape so it does the same. +- `compare-exposed-port` depends on `$PORT` binding numeric Dockerfile atoms + and then flowing into `metavariable-comparison`. +- `option_implicit_exprstmt` checks that a plain Dockerfile line pattern + matches `RUN b` without matching `RUN a && b`. +- `spacegrep_metavarbug` is a generic rule over a `.dockerfile` target; it + remains a zero-finding parity case. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 243 rule/target pairs and has +The current full sorted upstream sweep compared 246 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 241 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 243 compared +upstream-sweep: 244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared ``` Refresh the frontier with: @@ -250,14 +321,14 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Unsupported Upstream Frontier The upstream `tests/rules` same-basename corpus currently contains 444 rule -configs. The sweep can compare 243 because `target_for_rule` only considers +configs. The sweep can compare 246 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx +py js json txt generic gem ts tsx dockerfile ``` -That leaves 201 upstream configs outside the current oracle surface. These are +That leaves 198 upstream configs outside the current oracle surface. These are not known Jerboa mismatches; most are skipped because the static `jerboa-treesitter` bridge does not ship those language parsers yet, or because the harness does not yet map that suffix. @@ -280,7 +351,6 @@ Skipped target-extension buckets: 4 kt 3 yl 3 xml - 3 dockerfile 2 html 1 tpl 1 swift @@ -323,13 +393,15 @@ rs: macro_arg_taint, metavar_type_not_rust, misc_macro_call, Lower-count skipped buckets are still important for parity because they exercise target language behavior that generic matching cannot cover: Terraform (`tf`), -Julia (`jl`), C, Move, Dart, Ruby, Kotlin, YAML (`yl`), XML, Dockerfile, HTML, +Julia (`jl`), C, Move, Dart, Ruby, Kotlin, YAML (`yl`), XML, HTML, template/PHP (`tpl`), Swift, Mustache, EJS, and C#. Parser coverage constraints: - `src/semgrep/lang.ss` currently recognizes parse-supported ids `json`, `python`, `js`, and `ts`. +- Dockerfile is supported as a text-backed fallback using the generic matcher; + it is not backed by a Dockerfile parser. - TypeScript and TSX use the JavaScript parser fallback; decorated members are handled by a targeted scanner fallback, not a full TypeScript grammar. - `/Users/user/mine/jerboa-treesitter` currently vendors only JSON, Python, and @@ -341,7 +413,7 @@ Parser coverage constraints: ## Recommended Next Target -There is no known Jerboa mismatch in the current 243-case sorted upstream +There is no known Jerboa mismatch in the current 246-case sorted upstream sweep. The two packaged-Semgrep `current_error` cases are `anywhere_global` and `anywhere_metavar`, both rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. --- a/lib/semgrep/cli.sls +++ b/lib/semgrep/cli.sls @@ -19,7 +19,7 @@ "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") (display - "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript targets.\n")) + "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile targets.\n")) (def (parse-args args) (let loop ([xs args] [config #f] --- a/lib/semgrep/lang.sls +++ b/lib/semgrep/lang.sls @@ -12,10 +12,15 @@ (prefix (semgrep schema lang) schema-)) (def parse-supported-language-ids '("json" "python" "js" "ts")) + (def text-supported-language-ids '("dockerfile")) (def (string-member? needle xs) (and (not (null? xs)) (or (string=? needle (car xs)) (string-member? needle (cdr xs))))) + (def supported-language-ids + (append + parse-supported-language-ids + text-supported-language-ids)) (def (runtime-language-id schema-id) (cond [(or (string=? schema-id "python2") @@ -40,7 +45,9 @@ (and schema-id (string-member? schema-id parse-supported-language-ids)))) (def (supported-language? language) - (if (parse-supported-language? language) #t #f)) + (let ([schema-id (schema-language-id language)]) + (and schema-id + (string-member? schema-id supported-language-ids)))) (def (path-excluded-for-language? info path) (let loop ([exts (schema-language-info-excluded-exts info)]) (and (not (null? exts)) @@ -71,7 +78,7 @@ [(null? xs) #f] [(and (string-member? (schema-language-info-id (car xs)) - parse-supported-language-ids) + supported-language-ids) (language-info-matches-path? (car xs) path)) (runtime-language-id (schema-language-info-id (car xs)))] [else (loop (cdr xs))])))) --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -728,7 +728,8 @@ (def (generic-language? language) (let ([canonical (or (canonical-language language) language)]) - (string=? canonical "generic"))) + (or (string=? canonical "generic") + (string=? canonical "dockerfile")))) (def (javascript-like-language? language) (or (string=? language "javascript") (string=? language "js") @@ -858,7 +859,7 @@ (name-loop (+ j 1)) (loop j - (cons "([A-Za-z_][A-Za-z0-9_]*)" parts) + (cons "([A-Za-z0-9_]+)" parts) (cons (generic-capture-name source i (+ i 1) j) captures))))] --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -1,13 +1,13 @@ (("src/semgrep/output/sarif.ss" . "E935456E4B1921FB") ("src/semgrep/rule/parse-rule.ss" . "EC5BDBE8CB185021") ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") - ("src/semgrep/lang.ss" . "7E5441BD00A7F1D4") + ("src/semgrep/lang.ss" . "8076C5561FECD75D") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "20EF23C347E3D8FB") + ("src/semgrep/scan.ss" . "535A03F3724981B1") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") - ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") - ("src/semgrep/cli.ss" . "D56FC2D2EB449BA6")) + ("src/semgrep/cli.ss" . "84899658C448ADBF")) --- a/src/semgrep/cli.ss +++ b/src/semgrep/cli.ss @@ -15,7 +15,7 @@ (def (usage) (display "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") - (display "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript targets.\n")) + (display "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile targets.\n")) (def (parse-args args) (let loop ([xs args] --- a/src/semgrep/lang.ss +++ b/src/semgrep/lang.ss @@ -12,11 +12,17 @@ (def parse-supported-language-ids '("json" "python" "js" "ts")) +(def text-supported-language-ids + '("dockerfile")) + (def (string-member? needle xs) (and (not (null? xs)) (or (string=? needle (car xs)) (string-member? needle (cdr xs))))) +(def supported-language-ids + (append parse-supported-language-ids text-supported-language-ids)) + (def (runtime-language-id schema-id) (cond [(or (string=? schema-id "python2") @@ -46,7 +52,9 @@ (string-member? schema-id parse-supported-language-ids)))) (def (supported-language? language) - (if (parse-supported-language? language) #t #f)) + (let ([schema-id (schema-language-id language)]) + (and schema-id + (string-member? schema-id supported-language-ids)))) (def (path-excluded-for-language? info path) (let loop ([exts (schema-language-info-excluded-exts info)]) @@ -85,7 +93,7 @@ (cond [(null? xs) #f] [(and (string-member? (schema-language-info-id (car xs)) - parse-supported-language-ids) + supported-language-ids) (language-info-matches-path? (car xs) path)) (runtime-language-id (schema-language-info-id (car xs)))] [else (loop (cdr xs))]))) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -801,7 +801,8 @@ (def (generic-language? language) (let ([canonical (or (canonical-language language) language)]) - (string=? canonical "generic"))) + (or (string=? canonical "generic") + (string=? canonical "dockerfile")))) (def (javascript-like-language? language) (or (string=? language "javascript") @@ -936,7 +937,7 @@ (char=? (string-ref source j) #\_))) (name-loop (+ j 1)) (loop j - (cons "([A-Za-z_][A-Za-z0-9_]*)" parts) + (cons "([A-Za-z0-9_]+)" parts) (cons (generic-capture-name source i (+ i 1) j) captures))))] [(and (< (+ i 2) len) --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -53,7 +53,7 @@ run_json() { target_for_rule() { local base="$1" local ext - for ext in py js json txt generic gem ts tsx; do + for ext in py js json txt generic gem ts tsx dockerfile; do if [[ -f "$RULE_DIR/$base.$ext" ]]; then printf '%s\n' "$RULE_DIR/$base.$ext" return 0 --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -746,6 +746,22 @@ "a // ignore me\nb\n")]) (check (length findings) => 1))) +(test-case "scan Dockerfile patterns with generic text fallback" + (let* ([dockerfile-config + "rules:\n - id: demo.dockerfile.port\n languages: [dockerfile]\n message: invalid port\n severity: ERROR\n patterns:\n - pattern: EXPOSE $PORT\n - metavariable-comparison:\n metavariable: $PORT\n comparison: $PORT > 65535\n strip: true\n"] + [source "FROM busybox\nEXPOSE 65535\nEXPOSE 65536\nEXPOSE 0\n"] + [findings + (scan-config-string dockerfile-config + "dockerfile" + "Dockerfile" + source)]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 3) + (check (substring source + (finding-start-offset (car findings)) + (finding-end-offset (car findings))) + => "EXPOSE 65536"))) + (test-case "scan metavariable-pattern with generic language" (let* ([generic-config "rules:\n - id: demo.generic.metavariable-pattern\n languages: [generic]\n message: generic dependency\n severity: WARNING\n patterns:\n - pattern-regex: \"gem \\\"(?P<ONE>.*)\\\", \\\"(.*)\\\"\"\n - metavariable-pattern:\n metavariable: $ONE\n language: generic\n pattern: google-cloud-storage\n"] @@ -2228,6 +2244,8 @@ (check (guess-language-from-path "demo.ts") => "typescript") (check (guess-language-from-path "demo.tsx") => "typescript") (check (guess-language-from-path "demo.json") => "json") + (check (guess-language-from-path "Dockerfile") => "dockerfile") + (check (guess-language-from-path "demo.dockerfile") => "dockerfile") (check (guess-language-from-path "demo.rb") => #f)) (test-case "generated language registry"