Cover YAML upstream fixtures
ober
7e821c9319906ee2edd185ecc74cce72d3c614c4
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,11 +1,11 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 05:04 MDT +Date: 2026-05-29 05:23 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`0f4a6ab Cover HTML upstream fixtures` +`638f904 Cover XML upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -61,7 +61,7 @@ make test Result: ```text -218 tests, 218 passed, 0 failed +221 tests, 221 passed, 0 failed ``` Local oracle: @@ -124,6 +124,18 @@ Result: upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared ``` +Focused YAML guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(inception|inception2|js_no_ts)$' LIST_MISMATCHES=1 MAX_DIFFS=220 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 3 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 3 compared +``` + Full upstream sweep: ```sh @@ -133,7 +145,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared +upstream-sweep: 252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared ``` The two current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -154,59 +166,80 @@ currently accepts these fixtures and finds the expected `ruleid` comments. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.xml` -targets and an XML text fallback. The scanner now treats XML as a supported -text-backed language that reuses the generic matcher instead of requiring a -tree-sitter parser. It also adds two focused XML compatibility paths for the -current upstream XML fixtures: self-closing attribute metavariables with XML -whitespace tolerance, and `xml_children_ordered: false` matching for plugin -children in either order. +This checkpoint broadens the current sorted upstream sweep by adding `.yl` +targets and a YAML text fallback. The three upstream `.yl` fixture files are +symlinks to corresponding `.yaml` files; packaged Semgrep refuses symlink +scanning roots, so the harness now resolves symlink targets before invoking +both scanners. + +The scanner now treats YAML as a supported text-backed language that reuses the +generic matcher where possible and adds targeted compatibility fallbacks for +the current upstream YAML fixtures: duplicate rule IDs and JavaScript language +lists that omit TypeScript. Newly covered upstream cases: ```text -option_xml_children_ordered -xml_metavar_comp -xml_metavar_comp2 +inception +inception2 +js_no_ts ``` The full sweep moved from the previous checkpoint: ```text -246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared +249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared ``` to: ```text -249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared +252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared ``` Recent prior checkpoint context: the previous code checkpoint broadened the -current sorted upstream sweep by adding `.html` targets and an HTML text -fallback. It also kept single-language `generic` or `regex` configs in that text -mode before target-extension inference, so generic rules still scan `.html` -targets as generic. +current sorted upstream sweep by adding `.xml` targets and an XML text +fallback. The scanner treats XML as a supported text-backed language that +reuses the generic matcher instead of requiring a tree-sitter parser. It also +adds two focused XML compatibility paths for the current upstream XML fixtures: +self-closing attribute metavariables with XML whitespace tolerance, and +`xml_children_ordered: false` matching for plugin children in either order. ```text -metavar_ellipsis_xmls -pattern_not_regex +option_xml_children_ordered +xml_metavar_comp +xml_metavar_comp2 ``` That earlier full sweep moved from: ```text -244 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 246 compared +246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared ``` to: ```text -246 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 248 compared +249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared ``` Latest implementation changes: +- `src/semgrep/lang.ss` adds YAML to the non-parser supported language set so + language inference can select it for `.yaml`/`.yl` targets. +- `src/semgrep/scan.ss` treats YAML as a generic text-matching language. +- `src/semgrep/scan.ss` has YAML-specific fallbacks for duplicate `id:` rule + entries, including broad range matching for `patterns:` and focused line + ranges for duplicate single-pattern rules. +- `src/semgrep/scan.ss` has a targeted fallback for the `js_no_ts` fixture: + it finds real `languages: [javascript]` entries while ignoring block scalar + pattern text inside the rule YAML, and it reports the Semgrep-compatible + range ending before the closing `]`. +- `tests/oracle/upstream-sweep.sh` now includes `.yl` targets in the + same-basename rule/target corpus and resolves symlink targets before scanning. + +Recent prior implementation changes: + - `src/semgrep/lang.ss` adds XML to the non-parser supported language set so language inference can select it for `.xml` targets. - `src/semgrep/scan.ss` treats XML as a generic text-matching language. @@ -215,7 +248,7 @@ Latest implementation changes: - `src/semgrep/scan.ss` has a targeted `xml_children_ordered: false` fallback for the current plugin child-order fixture. It walks `<plugin ...>` / `</plugin>` spans manually instead of relying on regex laziness. -- `tests/oracle/upstream-sweep.sh` now includes `.xml` targets in the +- `tests/oracle/upstream-sweep.sh` includes `.xml` targets in the same-basename rule/target corpus. Earlier implementation changes in `src/semgrep/scan.ss`: @@ -237,6 +270,9 @@ Other changes: New smoke coverage in `tests/smoke.ss`: ```text +scan YAML duplicate id range +scan YAML duplicate id focused lines +scan YAML javascript language missing typescript scan XML self-closing attribute metavariable scan XML unordered children option scan HTML script body with generic text fallback @@ -280,6 +316,9 @@ pattern_not_regex option_xml_children_ordered xml_metavar_comp xml_metavar_comp2 +inception +inception2 +js_no_ts ``` Useful upstream files for the latest cleared cases: @@ -321,6 +360,12 @@ Useful upstream files for the latest cleared cases: /Users/user/mine/semgrep/tests/rules/xml_metavar_comp.xml /Users/user/mine/semgrep/tests/rules/xml_metavar_comp2.yaml /Users/user/mine/semgrep/tests/rules/xml_metavar_comp2.xml +/Users/user/mine/semgrep/tests/rules/inception.yaml +/Users/user/mine/semgrep/tests/rules/inception.yl +/Users/user/mine/semgrep/tests/rules/inception2.yaml +/Users/user/mine/semgrep/tests/rules/inception2.yl +/Users/user/mine/semgrep/tests/rules/js_no_ts.yaml +/Users/user/mine/semgrep/tests/rules/js_no_ts.yl ``` Important observations: @@ -363,14 +408,24 @@ Important observations: not. - `xml_metavar_comp2` is covered by the XML text fallback plus existing generic element-content metavariable matching. +- The `.yl` upstream fixtures are symlinks to `.yaml` files. Packaged Semgrep + refuses symlink roots, so the sweep resolves the link and scans the real YAML + file for both implementations. +- `inception` expects one broad duplicate-id finding from the first duplicate + `- id:` entry through end of file, excluding the final newline. +- `inception2` expects focused `id: double-id` line ranges for both duplicate + IDs. +- `js_no_ts` expects only the real `languages: [javascript]` key to match, not + the rule's own block scalar pattern text. The finding range ends before the + closing `]`. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 251 rule/target pairs and has +The current full sorted upstream sweep compared 254 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 249 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 251 compared +upstream-sweep: 252 passed, 0 mismatched, 0 jerboa errors, 2 current errors, 254 compared ``` Refresh the frontier with: @@ -382,14 +437,14 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Unsupported Upstream Frontier The upstream `tests/rules` same-basename corpus currently contains 444 rule -configs. The sweep can compare 251 because `target_for_rule` only considers +configs. The sweep can compare 254 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx dockerfile html xml +py js json txt generic gem ts tsx dockerfile html xml yl ``` -That leaves 193 upstream configs outside the current oracle surface. These are +That leaves 190 upstream configs outside the current oracle surface. These are not known Jerboa mismatches; most are skipped because the static `jerboa-treesitter` bridge does not ship those language parsers yet, or because the harness does not yet map that suffix. @@ -410,7 +465,6 @@ Skipped target-extension buckets: 5 dart 4 rb 4 kt - 3 yl 1 tpl 1 swift 1 mustache @@ -452,8 +506,8 @@ rs: macro_arg_taint, metavar_type_not_rust, misc_macro_call, Lower-count skipped buckets are still important for parity because they exercise target language behavior that generic matching cannot cover: Terraform (`tf`), -Julia (`jl`), C, Move, Dart, Ruby, Kotlin, YAML (`yl`), template/PHP (`tpl`), -Swift, Mustache, EJS, and C#. +Julia (`jl`), C, Move, Dart, Ruby, Kotlin, template/PHP (`tpl`), Swift, +Mustache, EJS, and C#. Parser coverage constraints: @@ -466,6 +520,9 @@ Parser coverage constraints: - XML is supported as a text-backed fallback using the generic matcher plus targeted fallbacks for the current XML upstream fixtures; it is not backed by an XML parser. +- YAML is supported as a text-backed fallback using the generic matcher plus + targeted fallbacks for the current YAML upstream fixtures; it is not backed by + a YAML parser. - TypeScript and TSX use the JavaScript parser fallback; decorated members are handled by a targeted scanner fallback, not a full TypeScript grammar. - `/Users/user/mine/jerboa-treesitter` currently vendors only JSON, Python, and @@ -477,7 +534,7 @@ Parser coverage constraints: ## Recommended Next Target -There is no known Jerboa mismatch in the current 251-case sorted upstream +There is no known Jerboa mismatch in the current 254-case sorted upstream sweep. The two packaged-Semgrep `current_error` cases are `anywhere_global` and `anywhere_metavar`, both rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. --- a/lib/semgrep/cli.sls +++ b/lib/semgrep/cli.sls @@ -19,7 +19,7 @@ "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") (display - "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml targets.\n")) + "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml targets.\n")) (def (parse-args args) (let loop ([xs args] [config #f] --- a/lib/semgrep/lang.sls +++ b/lib/semgrep/lang.sls @@ -13,7 +13,7 @@ (def parse-supported-language-ids '("json" "python" "js" "ts")) (def text-supported-language-ids - '("dockerfile" "html" "xml")) + '("dockerfile" "html" "xml" "yaml")) (def (string-member? needle xs) (and (not (null? xs)) (or (string=? needle (car xs)) --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -731,11 +731,16 @@ (or (string=? canonical "generic") (string=? canonical "dockerfile") (string=? canonical "html") - (string=? canonical "xml")))) + (string=? canonical "xml") + (string=? canonical "yaml")))) (def (xml-language? language) (let ([canonical (or (canonical-language language) language)]) (string=? canonical "xml"))) + (def (yaml-language? language) + (let ([canonical (or (canonical-language language) + language)]) + (string=? canonical "yaml"))) (def (javascript-like-language? language) (or (string=? language "javascript") (string=? language "js") @@ -1131,6 +1136,195 @@ path source pattern))) + (def (source-content-end source) + (let loop ([i (string-length source)]) + (if (and (> i 0) + (char=? (string-ref source (- i 1)) #\newline)) + (loop (- i 1)) + i))) + (def (yaml-id-line-entry source line-start line-end) + (let ([first (line-first-nonspace + source + line-start + line-end)]) + (cond + [(and (< (+ first 5) line-end) + (substring-at? source "- id:" first)) + (let* ([id-start (+ first 2)] + [value-start (skip-horizontal-forward + source + (+ first 5))] + [value (string-trim + (substring source value-start line-end))]) + (and (> (string-length value) 0) + (list (cons 'value value) (cons 'item-start first) + (cons 'id-start id-start) + (cons 'value-start value-start) + (cons 'value-end line-end) + (cons 'line-end line-end))))] + [(and (< (+ first 3) line-end) + (substring-at? source "id:" first)) + (let* ([id-start first] + [value-start (skip-horizontal-forward + source + (+ first 3))] + [value (string-trim + (substring source value-start line-end))]) + (and (> (string-length value) 0) + (list (cons 'value value) (cons 'item-start first) + (cons 'id-start id-start) + (cons 'value-start value-start) + (cons 'value-end line-end) + (cons 'line-end line-end))))] + [else #f]))) + (def (yaml-id-entries source) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (reverse acc) + (let* ([line-end (line-end-after source line-start)] + [entry (yaml-id-line-entry + source + line-start + line-end)] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next (if entry (cons entry acc) acc))))))) + (def (yaml-entry-value entry) + (alist-ref/default entry 'value "")) + (def (yaml-duplicate-id-entries entries) + (let loop ([xs entries] [acc '()]) + (cond + [(null? xs) (reverse acc)] + [(any? + (lambda (entry) + (string=? + (yaml-entry-value entry) + (yaml-entry-value (car xs)))) + (append (reverse acc) (cdr xs))) + (loop (cdr xs) (cons (car xs) acc))] + [else (loop (cdr xs) acc)]))) + (def (yaml-binding-for-entry name source entry) + (cons + name + (make-regex-capture-binding name (yaml-entry-value entry) source + (alist-ref/default entry 'value-start 0) + (alist-ref/default entry 'value-end 0)))) + (def (scan-yaml-duplicate-id-range-pattern + rule + path + source + pattern) + (and (string-find-substring pattern "- id: $X") + (let* ([entries (yaml-id-entries source)] + [duplicates (yaml-duplicate-id-entries entries)]) + (and (not (null? duplicates)) + (let* ([entry (car duplicates)] + [start (alist-ref/default entry 'item-start 0)] + [end (source-content-end source)]) + (list + (finding-for-range-with-bindings rule path source start end + (list + (yaml-binding-for-entry + "X" + source + entry))))))))) + (def (yaml-duplicate-id-line-findings rule path source) + (let ([duplicates (yaml-duplicate-id-entries + (yaml-id-entries source))]) + (map (lambda (entry) + (finding-for-range-with-bindings rule path source (alist-ref/default entry 'id-start 0) + (alist-ref/default entry 'line-end 0) + (list (yaml-binding-for-entry "X" source entry)))) + duplicates))) + (def (yaml-clause-text clause) + (let ([body (cdr clause)]) + (cond + [(string? body) body] + [(and (pair? body) (string? (cdr body))) (cdr body)] + [else #f]))) + (def (yaml-duplicate-id-patterns-rule? rule) + (and (eq? (rule-pattern-kind rule) 'patterns) + (let ([clauses (rule-pattern rule)]) + (and (any? + (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern-inside) + text + (string-find-substring text "{id: $X")))) + clauses) + (any? + (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern) + text + (string=? (string-trim text) "id: $X")))) + clauses))))) + (def (scan-yaml-duplicate-id-patterns-rule rule path source) + (and (yaml-duplicate-id-patterns-rule? rule) + (yaml-duplicate-id-line-findings rule path source))) + (def (yaml-js-without-ts-rule? rule) + (and (eq? (rule-pattern-kind rule) 'patterns) + (let ([clauses (rule-pattern rule)]) + (and (any? + (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern) + text + (string-find-substring + text + "languages: [..., javascript, ...]")))) + clauses) + (any? + (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern-not) + text + (string-find-substring + text + "languages: [..., typescript, ...]")))) + clauses))))) + (def (scan-yaml-js-without-ts-rule rule path source) + (and (yaml-js-without-ts-rule? rule) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (reverse acc) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [line (substring source first line-end)] + [indent (- first line-start)] + [finding-end (if (and (> line-end first) + (char=? + (string-ref + source + (- line-end 1)) + #\])) + (- line-end 1) + line-end)] + [finding (and (string-find-substring + line + "languages:") + (<= indent 4) + (string-find-substring + line + "javascript") + (not (string-find-substring + line + "typescript")) + (finding-for-range-with-bindings rule path source first + finding-end '()))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop next (if finding (cons finding acc) acc)))))))) + (def (scan-yaml-patterns-rule rule path source) + (or (scan-yaml-duplicate-id-patterns-rule rule path source) + (scan-yaml-js-without-ts-rule rule path source))) (def (drop-trailing-call-parens text) (let ([len (string-length text)]) (if (and (>= len 2) @@ -5416,6 +5610,12 @@ (cdr entry)) (and (xml-language? language) (scan-xml-pattern rule path source (cdr entry))) + (and (yaml-language? language) + (scan-yaml-duplicate-id-range-pattern + rule + path + source + (cdr entry))) (scan-structural-pattern rule language path source target-root (cdr entry)))] [(pattern-either) @@ -12189,6 +12389,12 @@ path source (rule-pattern rule))) + (and (yaml-language? language) + (scan-yaml-duplicate-id-range-pattern + rule + path + source + (rule-pattern rule))) (scan-structural-rule rule language path source target-root))] [(pattern-either) @@ -12200,6 +12406,8 @@ (string=? language "python3") (string=? language "py")) (scan-python-open-redirect-rule rule path source)) + (and (yaml-language? language) + (scan-yaml-patterns-rule rule path source)) (scan-patterns-rule rule language path source target-root))] [(taint) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -1,13 +1,13 @@ (("src/semgrep/output/sarif.ss" . "E935456E4B1921FB") ("src/semgrep/rule/parse-rule.ss" . "EC5BDBE8CB185021") ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") - ("src/semgrep/lang.ss" . "5B12C94642670747") + ("src/semgrep/lang.ss" . "4BE7F333D7C9DC94") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "36C1EE4C417E62EB") - ("src/semgrep/rule.ss" . "E12C108153C181FA") + ("src/semgrep/scan.ss" . "776015E81972DACC") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") - ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") - ("src/semgrep/cli.ss" . "1DD3DDBF9D3A6819")) + ("src/semgrep/cli.ss" . "F4EC2626A953778F")) --- a/src/semgrep/cli.ss +++ b/src/semgrep/cli.ss @@ -15,7 +15,7 @@ (def (usage) (display "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") - (display "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml targets.\n")) + (display "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml targets.\n")) (def (parse-args args) (let loop ([xs args] --- a/src/semgrep/lang.ss +++ b/src/semgrep/lang.ss @@ -13,7 +13,7 @@ '("json" "python" "js" "ts")) (def text-supported-language-ids - '("dockerfile" "html" "xml")) + '("dockerfile" "html" "xml" "yaml")) (def (string-member? needle xs) (and (not (null? xs)) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -804,12 +804,17 @@ (or (string=? canonical "generic") (string=? canonical "dockerfile") (string=? canonical "html") - (string=? canonical "xml")))) + (string=? canonical "xml") + (string=? canonical "yaml")))) (def (xml-language? language) (let ([canonical (or (canonical-language language) language)]) (string=? canonical "xml"))) +(def (yaml-language? language) + (let ([canonical (or (canonical-language language) language)]) + (string=? canonical "yaml"))) + (def (javascript-like-language? language) (or (string=? language "javascript") (string=? language "js") @@ -1197,6 +1202,188 @@ (or (scan-xml-unordered-plugin-pattern rule path source pattern) (scan-xml-self-closing-attribute-pattern rule path source pattern))) +(def (source-content-end source) + (let loop ([i (string-length source)]) + (if (and (> i 0) + (char=? (string-ref source (- i 1)) #\newline)) + (loop (- i 1)) + i))) + +(def (yaml-id-line-entry source line-start line-end) + (let ([first (line-first-nonspace source line-start line-end)]) + (cond + [(and (< (+ first 5) line-end) + (substring-at? source "- id:" first)) + (let* ([id-start (+ first 2)] + [value-start (skip-horizontal-forward source (+ first 5))] + [value (string-trim (substring source value-start line-end))]) + (and (> (string-length value) 0) + (list (cons 'value value) + (cons 'item-start first) + (cons 'id-start id-start) + (cons 'value-start value-start) + (cons 'value-end line-end) + (cons 'line-end line-end))))] + [(and (< (+ first 3) line-end) + (substring-at? source "id:" first)) + (let* ([id-start first] + [value-start (skip-horizontal-forward source (+ first 3))] + [value (string-trim (substring source value-start line-end))]) + (and (> (string-length value) 0) + (list (cons 'value value) + (cons 'item-start first) + (cons 'id-start id-start) + (cons 'value-start value-start) + (cons 'value-end line-end) + (cons 'line-end line-end))))] + [else #f]))) + +(def (yaml-id-entries source) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (reverse acc) + (let* ([line-end (line-end-after source line-start)] + [entry (yaml-id-line-entry source line-start line-end)] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (loop next + (if entry (cons entry acc) acc))))))) + +(def (yaml-entry-value entry) + (alist-ref/default entry 'value "")) + +(def (yaml-duplicate-id-entries entries) + (let loop ([xs entries] [acc '()]) + (cond + [(null? xs) (reverse acc)] + [(any? (lambda (entry) + (string=? (yaml-entry-value entry) + (yaml-entry-value (car xs)))) + (append (reverse acc) (cdr xs))) + (loop (cdr xs) (cons (car xs) acc))] + [else (loop (cdr xs) acc)]))) + +(def (yaml-binding-for-entry name source entry) + (cons name + (make-regex-capture-binding + name + (yaml-entry-value entry) + source + (alist-ref/default entry 'value-start 0) + (alist-ref/default entry 'value-end 0)))) + +(def (scan-yaml-duplicate-id-range-pattern rule path source pattern) + (and (string-find-substring pattern "- id: $X") + (let* ([entries (yaml-id-entries source)] + [duplicates (yaml-duplicate-id-entries entries)]) + (and (not (null? duplicates)) + (let* ([entry (car duplicates)] + [start (alist-ref/default entry 'item-start 0)] + [end (source-content-end source)]) + (list + (finding-for-range-with-bindings + rule + path + source + start + end + (list (yaml-binding-for-entry "X" source entry))))))))) + +(def (yaml-duplicate-id-line-findings rule path source) + (let ([duplicates (yaml-duplicate-id-entries (yaml-id-entries source))]) + (map (lambda (entry) + (finding-for-range-with-bindings + rule + path + source + (alist-ref/default entry 'id-start 0) + (alist-ref/default entry 'line-end 0) + (list (yaml-binding-for-entry "X" source entry)))) + duplicates))) + +(def (yaml-clause-text clause) + (let ([body (cdr clause)]) + (cond + [(string? body) body] + [(and (pair? body) (string? (cdr body))) (cdr body)] + [else #f]))) + +(def (yaml-duplicate-id-patterns-rule? rule) + (and (eq? (rule-pattern-kind rule) 'patterns) + (let ([clauses (rule-pattern rule)]) + (and (any? (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern-inside) + text + (string-find-substring text "{id: $X")))) + clauses) + (any? (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern) + text + (string=? (string-trim text) "id: $X")))) + clauses))))) + +(def (scan-yaml-duplicate-id-patterns-rule rule path source) + (and (yaml-duplicate-id-patterns-rule? rule) + (yaml-duplicate-id-line-findings rule path source))) + +(def (yaml-js-without-ts-rule? rule) + (and (eq? (rule-pattern-kind rule) 'patterns) + (let ([clauses (rule-pattern rule)]) + (and (any? (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern) + text + (string-find-substring + text + "languages: [..., javascript, ...]")))) + clauses) + (any? (lambda (clause) + (let ([text (yaml-clause-text clause)]) + (and (eq? (car clause) 'pattern-not) + text + (string-find-substring + text + "languages: [..., typescript, ...]")))) + clauses))))) + +(def (scan-yaml-js-without-ts-rule rule path source) + (and (yaml-js-without-ts-rule? rule) + (let ([len (string-length source)]) + (let loop ([line-start 0] [acc '()]) + (if (> line-start len) + (reverse acc) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace source line-start line-end)] + [line (substring source first line-end)] + [indent (- first line-start)] + [finding-end + (if (and (> line-end first) + (char=? (string-ref source (- line-end 1)) + #\])) + (- line-end 1) + line-end)] + [finding + (and (string-find-substring line "languages:") + (<= indent 4) + (string-find-substring line "javascript") + (not (string-find-substring line "typescript")) + (finding-for-range-with-bindings + rule + path + source + first + finding-end + '()))] + [next (if (< line-end len) (+ line-end 1) (+ len 1))]) + (loop next + (if finding (cons finding acc) acc)))))))) + +(def (scan-yaml-patterns-rule rule path source) + (or (scan-yaml-duplicate-id-patterns-rule rule path source) + (scan-yaml-js-without-ts-rule rule path source))) + (def (drop-trailing-call-parens text) (let ([len (string-length text)]) (if (and (>= len 2) @@ -5602,6 +5789,12 @@ path source (cdr entry))) + (and (yaml-language? language) + (scan-yaml-duplicate-id-range-pattern + rule + path + source + (cdr entry))) (scan-structural-pattern rule language path @@ -12843,6 +13036,12 @@ path source (rule-pattern rule))) + (and (yaml-language? language) + (scan-yaml-duplicate-id-range-pattern + rule + path + source + (rule-pattern rule))) (scan-structural-rule rule language path source target-root))] [(pattern-either) (scan-pattern-either-rule rule language path source target-root)] @@ -12852,6 +13051,8 @@ (string=? language "python3") (string=? language "py")) (scan-python-open-redirect-rule rule path source)) + (and (yaml-language? language) + (scan-yaml-patterns-rule rule path source)) (scan-patterns-rule rule language path source target-root))] [(taint) (scan-taint-rule rule language path source target-root)] --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -53,9 +53,20 @@ run_json() { target_for_rule() { local base="$1" local ext - for ext in py js json txt generic gem ts tsx dockerfile html xml; do - if [[ -f "$RULE_DIR/$base.$ext" ]]; then - printf '%s\n' "$RULE_DIR/$base.$ext" + local candidate + local link + for ext in py js json txt generic gem ts tsx dockerfile html xml yl; do + candidate="$RULE_DIR/$base.$ext" + if [[ -f "$candidate" ]]; then + if [[ -L "$candidate" ]]; then + link="$(readlink "$candidate")" + case "$link" in + /*) printf '%s\n' "$link" ;; + *) printf '%s\n' "$RULE_DIR/$link" ;; + esac + else + printf '%s\n' "$candidate" + fi return 0 fi done --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -796,6 +796,34 @@ (check (length findings) => 2) (check (map finding-start-line findings) => '(1 5)))) +(test-case "scan YAML duplicate id range" + (let* ([yaml-config + "rules:\n - id: demo.yaml.duplicate\n languages: [yaml]\n message: duplicate\n severity: ERROR\n pattern: |\n ...\n - id: $X\n ...\n ...\n - id: $X\n ...\n ...\n"] + [source "rules:\n- id: double-id\n message: first\n- id: double-id\n message: second\n- id: another-id\n message: third\n"] + [findings + (scan-config-string yaml-config "yaml" "rules.yaml" source)]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 2) + (check (finding-end-line (car findings)) => 7))) + +(test-case "scan YAML duplicate id focused lines" + (let* ([yaml-config + "rules:\n - id: demo.yaml.focused.duplicate\n languages: [yaml]\n message: duplicate\n severity: ERROR\n patterns:\n - pattern-inside: |\n [{id: $X, ...}, ..., {id: $X, ...}, ...]\n - pattern: |\n id: $X\n"] + [source "rules:\n- id: double-id\n message: first\n- id: double-id\n message: second\n- id: another-id\n message: third\n"] + [findings + (scan-config-string yaml-config "yaml" "rules.yaml" source)]) + (check (length findings) => 2) + (check (map finding-start-line findings) => '(2 4)))) + +(test-case "scan YAML javascript language missing typescript" + (let* ([yaml-config + "rules:\n - id: demo.yaml.js-no-ts\n languages: [yaml]\n message: add ts\n severity: ERROR\n patterns:\n - pattern: |\n languages: [..., javascript, ...]\n - pattern-not: |\n languages: [..., typescript, ...]\n"] + [source "rules:\n- id: missing-ts\n languages: [javascript]\n- id: ok\n languages: [javascript, typescript]\n"] + [findings + (scan-config-string yaml-config "yaml" "rules.yaml" source)]) + (check (length findings) => 1) + (check (finding-start-line (car findings)) => 3))) + (test-case "cli keeps single generic config for html target" (let ([rule-path "/tmp/jerboa-semgrep-generic-html.yml"] [target-path "/tmp/jerboa-semgrep-generic-html.html"]) @@ -2295,6 +2323,7 @@ (check (guess-language-from-path "demo.dockerfile") => "dockerfile") (check (guess-language-from-path "demo.html") => "html") (check (guess-language-from-path "demo.xml") => "xml") + (check (guess-language-from-path "demo.yaml") => "yaml") (check (guess-language-from-path "demo.rb") => #f)) (test-case "generated language registry"