Cover PHP template upstream fixture
ober
673db2ee3d5c302f45d5bd11be0ebbd72c8f3b27
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,11 +1,11 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 06:16 MDT +Date: 2026-05-29 06:31 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`ca44383 Cover C upstream fixtures` +`cc51d79 Cover Terraform upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -61,7 +61,7 @@ make test Result: ```text -226 tests, 226 passed, 0 failed +227 tests, 227 passed, 0 failed ``` Local oracle: @@ -172,6 +172,18 @@ Result: upstream-sweep: 7 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 7 compared ``` +Focused PHP template guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^misc_tpl_is_php$' LIST_MISMATCHES=1 MAX_DIFFS=120 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 1 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 1 compared +``` + Full upstream sweep: ```sh @@ -181,7 +193,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 266 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 269 compared +upstream-sweep: 267 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 270 compared ``` The three current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -204,10 +216,36 @@ comes from the packaged oracle failing before Jerboa comparison. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.tf` -targets and a Terraform/HCL text fallback. Terraform is not parser-backed yet; -the new support routes HCL through the generic text matcher and adds focused -Terraform compatibility for the current upstream `.tf` fixtures. +This checkpoint broadens the current sorted upstream sweep by adding `.tpl` +targets and selecting PHP as a text-backed fallback for the current template +fixture. PHP is not parser-backed yet, and upstream `.php` targets are still +intentionally outside the sweep. The new support is deliberately narrow: it +lets the `languages: [php]` rule in `misc_tpl_is_php.yaml` scan the `.tpl` +target with the generic matcher and match `foo(...)` against `foo()`. + +Newly covered upstream case: + +```text +misc_tpl_is_php +``` + +The full sweep moved from the previous checkpoint: + +```text +266 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 269 compared +``` + +to: + +```text +267 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 270 compared +``` + +Recent prior checkpoint context: the previous code checkpoint broadened the +current sorted upstream sweep by adding `.tf` targets and a Terraform/HCL text +fallback. Terraform is not parser-backed yet; the support routes HCL through +the generic text matcher and adds focused Terraform compatibility for the +current upstream `.tf` fixtures. The scanner now handles the current Terraform resource and variable block ranges, quoted scalar coercions, simple `variable`/`locals` constant @@ -226,7 +264,7 @@ terraform_nested_yaml whole_file ``` -The full sweep moved from the previous checkpoint: +That earlier full sweep moved from: ```text 259 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 262 compared @@ -238,7 +276,7 @@ to: 266 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 269 compared ``` -Recent prior checkpoint context: the previous code checkpoint broadened the +Additional prior checkpoint context: the earlier code checkpoint broadened the current sorted upstream sweep by adding `.c` targets and a C text fallback. C is not parser-backed yet; it routes through the generic text matcher plus focused C compatibility for the current upstream C fixtures. @@ -266,6 +304,15 @@ to: Latest implementation changes: +- `src/semgrep/lang.ss` adds PHP to the non-parser supported language set so + language inference can select it for `.tpl` targets. +- `src/semgrep/scan.ss` treats PHP as a generic text-matching language. +- `tests/oracle/upstream-sweep.sh` includes `.tpl` targets in the + same-basename rule/target corpus. It does not include `.php` yet, so the + larger PHP parser bucket remains explicitly out of scope for this checkpoint. + +Recent prior implementation changes: + - `src/semgrep/lang.ss` adds Terraform to the non-parser supported language set so language inference can select it for `.tf`/`.hcl` targets. - `src/semgrep/scan.ss` treats Terraform as a generic text-matching language. @@ -276,10 +323,10 @@ Latest implementation changes: metavariable comparison/regex fixtures. - `src/semgrep/scan.ss` adds heredoc matching for `content = <<DOC` style nested YAML content in Terraform targets. -- `tests/oracle/upstream-sweep.sh` now includes `.tf` targets in the +- `tests/oracle/upstream-sweep.sh` includes `.tf` targets in the same-basename rule/target corpus. -Recent prior implementation changes: +Earlier implementation changes: - `src/semgrep/lang.ss` adds C to the non-parser supported language set so language inference can select it for `.c` targets. @@ -337,6 +384,7 @@ Other changes: New smoke coverage in `tests/smoke.ss`: ```text +scan PHP template with generic text fallback scan Terraform coerced scalar assignments scan Terraform resolved field value scan C array declaration metavariable type @@ -405,6 +453,7 @@ terraform_metavariable terraform_mvar_regex_interpolated_var terraform_nested_yaml whole_file +misc_tpl_is_php ``` Useful upstream files for the latest cleared cases: @@ -482,6 +531,8 @@ Useful upstream files for the latest cleared cases: /Users/user/mine/semgrep/tests/rules/terraform_nested_yaml.tf /Users/user/mine/semgrep/tests/rules/whole_file.yaml /Users/user/mine/semgrep/tests/rules/whole_file.tf +/Users/user/mine/semgrep/tests/rules/misc_tpl_is_php.yaml +/Users/user/mine/semgrep/tests/rules/misc_tpl_is_php.tpl ``` Important observations: @@ -573,14 +624,18 @@ Important observations: YAML `schemaVersion: '1.0'` line. - `whole_file` expects each variable block except exact `name_tag` and `environment_tag`; the typo `environment_ta` remains a finding. +- `misc_tpl_is_php` is `languages: [php]` over a `.tpl` target. PHP is only a + text-backed fallback in this port right now; this is not a PHP parser. The + upstream fixture expects the `foo(...)` pattern to match `foo()` on line 4 + with range line 4 column 1 through column 6. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 269 rule/target pairs and has +The current full sorted upstream sweep compared 270 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 266 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 269 compared +upstream-sweep: 267 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 270 compared ``` Refresh the frontier with: @@ -592,14 +647,14 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes ## Unsupported Upstream Frontier The upstream `tests/rules` same-basename corpus currently contains 444 rule -configs. The sweep can compare 269 because `target_for_rule` only considers +configs. The sweep can compare 270 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx dockerfile html xml yl ejs mustache c tf +py js json txt generic gem ts tsx dockerfile html xml yl ejs mustache c tf tpl ``` -That leaves 175 upstream configs outside the current oracle surface. These are +That leaves 174 upstream configs outside the current oracle surface. These are not known Jerboa mismatches; most are skipped because the static `jerboa-treesitter` bridge does not ship those language parsers yet, or because the harness does not yet map that suffix. @@ -618,7 +673,6 @@ Skipped target-extension buckets: 5 dart 4 rb 4 kt - 1 tpl 1 swift 1 cs ``` @@ -657,7 +711,7 @@ rs: macro_arg_taint, metavar_type_not_rust, misc_macro_call, Lower-count skipped buckets are still important for parity because they exercise target language behavior that generic matching cannot cover: Julia (`jl`), -Move, Dart, Ruby, Kotlin, template/PHP (`tpl`), Swift, and C#. +Move, Dart, Ruby, Kotlin, Swift, and C#. Parser coverage constraints: @@ -679,6 +733,9 @@ Parser coverage constraints: - Terraform/HCL is supported as a text-backed fallback using the generic matcher plus targeted fallbacks for the current Terraform upstream fixtures; it is not backed by an HCL parser. +- PHP is supported as a text-backed fallback only when selected/inferred for + the current `.tpl` template fixture. The upstream `.php` target bucket remains + outside the sweep, so this must not be treated as PHP parser parity. - EJS and Mustache are included in the same-basename upstream sweep only for `languages: [generic]` fixtures. They are not recognized as standalone target languages. @@ -693,7 +750,7 @@ Parser coverage constraints: ## Recommended Next Target -There is no known Jerboa mismatch in the current 269-case sorted upstream +There is no known Jerboa mismatch in the current 270-case sorted upstream sweep. The three packaged-Semgrep `current_error` cases are `anywhere_global`, `anywhere_include`, and `anywhere_metavar`, all rejected by current CLI rule validation because their upstream test YAML contains an empty @@ -704,6 +761,9 @@ this now-empty frontier. Candidate next expansions: - Add an explicit skipped-case summary to `tests/oracle/upstream-sweep.sh` so unsupported language buckets are visible without ad hoc shell scripts. +- Decide whether to continue the PHP path by adding `.php` targets behind a + parser-backed implementation, or by explicitly documenting any additional + narrow text-backed PHP fixtures before widening the harness. - Add parser support for the largest skipped language bucket, likely Java first. Be careful with the non-git `jerboa-treesitter` dependency. - Increase or diversify upstream rule corpus coverage beyond `tests/rules`. --- a/lib/semgrep/cli.sls +++ b/lib/semgrep/cli.sls @@ -19,7 +19,7 @@ "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") (display - "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml/c/terraform targets.\n")) + "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml/c/terraform/php targets.\n")) (def (parse-args args) (let loop ([xs args] [config #f] --- a/lib/semgrep/lang.sls +++ b/lib/semgrep/lang.sls @@ -13,7 +13,7 @@ (def parse-supported-language-ids '("json" "python" "js" "ts")) (def text-supported-language-ids - '("dockerfile" "html" "xml" "yaml" "c" "terraform")) + '("dockerfile" "html" "xml" "yaml" "c" "terraform" "php")) (def (string-member? needle xs) (and (not (null? xs)) (or (string=? needle (car xs)) --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -734,7 +734,8 @@ (string=? canonical "xml") (string=? canonical "yaml") (string=? canonical "c") - (string=? canonical "terraform")))) + (string=? canonical "terraform") + (string=? canonical "php")))) (def (c-language? language) (let ([canonical (or (canonical-language language) language)]) --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -1,13 +1,13 @@ (("src/semgrep/output/sarif.ss" . "E935456E4B1921FB") ("src/semgrep/rule/parse-rule.ss" . "EC5BDBE8CB185021") ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") - ("src/semgrep/lang.ss" . "88DEC6D0BB448737") + ("src/semgrep/lang.ss" . "F7094F3E0E0AB987") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "88E55B57DCF5A343") - ("src/semgrep/rule.ss" . "E12C108153C181FA") - ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/scan.ss" . "592208D8C03E1751") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") + ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") + ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") - ("src/semgrep/cli.ss" . "5074896D86E2A0BC")) + ("src/semgrep/cli.ss" . "EAC6A60F390262DD")) --- a/src/semgrep/cli.ss +++ b/src/semgrep/cli.ss @@ -15,7 +15,7 @@ (def (usage) (display "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") - (display "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml/c/terraform targets.\n")) + (display "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml/c/terraform/php targets.\n")) (def (parse-args args) (let loop ([xs args] --- a/src/semgrep/lang.ss +++ b/src/semgrep/lang.ss @@ -13,7 +13,7 @@ '("json" "python" "js" "ts")) (def text-supported-language-ids - '("dockerfile" "html" "xml" "yaml" "c" "terraform")) + '("dockerfile" "html" "xml" "yaml" "c" "terraform" "php")) (def (string-member? needle xs) (and (not (null? xs)) --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -807,7 +807,8 @@ (string=? canonical "xml") (string=? canonical "yaml") (string=? canonical "c") - (string=? canonical "terraform")))) + (string=? canonical "terraform") + (string=? canonical "php")))) (def (c-language? language) (let ([canonical (or (canonical-language language) language)]) --- a/tests/oracle/upstream-sweep.sh +++ b/tests/oracle/upstream-sweep.sh @@ -55,7 +55,7 @@ target_for_rule() { local ext local candidate local link - for ext in py js json txt generic gem ts tsx dockerfile html xml yl ejs mustache c tf; do + for ext in py js json txt generic gem ts tsx dockerfile html xml yl ejs mustache c tf tpl; do candidate="$RULE_DIR/$base.$ext" if [[ -f "$candidate" ]]; then if [[ -L "$candidate" ]]; then --- a/tests/smoke.ss +++ b/tests/smoke.ss @@ -805,6 +805,18 @@ (check (length findings) => 1) (check (finding-start-line (car findings)) => 4))) +(test-case "scan PHP template with generic text fallback" + (let* ([php-config + "rules:\n - id: demo.php.tpl\n languages: [php]\n message: php template\n severity: ERROR\n pattern: foo(...)\n"] + [source "<?php\nfoo();\nbar();\n"] + [findings + (scan-config-string php-config "php" "demo.tpl" source)]) + (check (length findings) => 1) + (check (substring source + (finding-start-offset (car findings)) + (finding-end-offset (car findings))) + => "foo()"))) + (test-case "scan Dockerfile patterns with generic text fallback" (let* ([dockerfile-config "rules:\n - id: demo.dockerfile.port\n languages: [dockerfile]\n message: invalid port\n severity: ERROR\n patterns:\n - pattern: EXPOSE $PORT\n - metavariable-comparison:\n metavariable: $PORT\n comparison: $PORT > 65535\n strip: true\n"] @@ -2385,6 +2397,7 @@ (check (guess-language-from-path "demo.yaml") => "yaml") (check (guess-language-from-path "demo.c") => "c") (check (guess-language-from-path "demo.tf") => "terraform") + (check (guess-language-from-path "demo.tpl") => "php") (check (guess-language-from-path "demo.rb") => #f)) (test-case "generated language registry"