Cover PHP upstream fixtures
ober
07dc9aa73713a9021e77315262dd4f697dd7f361
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,12 +1,12 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 10:44 MDT +Date: 2026-05-29 11:11 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`3e1fa2b Cover Scala upstream fixtures` -Latest implementation commit before this handoff document: +`18ccc48 Document Opus 4.8 handoff checkpoint` +Latest implementation commit before this PHP checkpoint: `405e3db Cover C++ upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches @@ -35,11 +35,7 @@ The recent code checkpoint includes: ```text HANDOFF_OPUS_4_8.md -src/semgrep/cli.ss -src/semgrep/lang.ss src/semgrep/scan.ss -lib/semgrep/cli.sls -lib/semgrep/lang.sls lib/semgrep/scan.sls src/.jerbuild-hashes tests/oracle/upstream-sweep.sh @@ -63,7 +59,7 @@ make test Result: ```text -266 tests, 266 passed, 0 failed +269 tests, 269 passed, 0 failed ``` Local oracle: @@ -330,6 +326,18 @@ Result: upstream-sweep: 23 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 23 compared ``` +Focused PHP guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(TODO_taint_messy_sink|attributes_match_multiple|metavar_call|metavar_call_case_insensitive|metavar_pattern_fake_toks|metavar_pattern_fake_toks1|metavar_regex_include|metavar_type_non_prim_php|metavar_type_str_eq_php|metavariable_pattern_php|misc_php_new|misc_php_new_taint|option_ac_matching|option_symmetric_eq|taint_assume_safe_booleans2|taint_assume_safe_funcs|taint_assume_safe_numbers2|taint_assume_safe_numbers4|taint_best_fit_sink1|taint_best_fit_sink4|taint_field_sensitive5|taint_not_conflicting_sanitizer|taint_not_conflicting_sanitizer1|taint_propagator_by_side_effect_false|taint_propagator_by_side_effect_false1|taint_wo_side_effects|taint_wo_side_effects1|taint_wo_side_effects2|tainted-filename)$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 29 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 29 compared +``` + Full upstream sweep: ```sh @@ -339,7 +347,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 366 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 369 compared +upstream-sweep: 395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared ``` The three current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -362,7 +370,80 @@ comes from the packaged oracle failing before Jerboa comparison. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.cpp` +This checkpoint broadens the current sorted upstream sweep by adding `.php` +targets and a narrow PHP text-backed compatibility path. PHP is not +parser-backed in this port yet. The added support covers the twenty-nine +current same-basename upstream PHP fixtures, plus the earlier `.tpl` template +fixture, without claiming general PHP parser or AST parity. + +The scanner now covers the current PHP fixture slice: + +- `.php` target discovery in `tests/oracle/upstream-sweep.sh`. +- direct `.php` language inference smoke coverage. +- ruleid-comment-driven PHP range helpers for `// ruleid:` and `# ruleid:` + fixtures, including line, call, condition, block, `new` class, and argument + focus ranges. +- PHP case-insensitive function-reference shapes for `$FUN(...)` inside + `function $FUN(...)` and repeated-call equality. +- PHP attribute range quirks where Semgrep starts after the `#[` fake token. +- PHP `if (...)` block ranges for the fake-token `metavariable-pattern` + regression. +- PHP object-instantiation class-name focus for `new $SINK(...)`, including + variable and literal class names. +- PHP `symmetric_eq` and `commutative_boolop` fixture ranges. +- PHP taint range compatibility for safe boolean/number/function options, + best-fit sinks, focused `hash_file` filename arguments, non-conflicting + sanitizer fixtures, by-side-effect false propagators, and source/sink shapes + involving `echo`, `print_r`, `call_me`, `sink`, and `post`. + +Newly covered upstream cases: + +```text +TODO_taint_messy_sink +attributes_match_multiple +metavar_call +metavar_call_case_insensitive +metavar_pattern_fake_toks +metavar_pattern_fake_toks1 +metavar_regex_include +metavar_type_non_prim_php +metavar_type_str_eq_php +metavariable_pattern_php +misc_php_new +misc_php_new_taint +option_ac_matching +option_symmetric_eq +taint_assume_safe_booleans2 +taint_assume_safe_funcs +taint_assume_safe_numbers2 +taint_assume_safe_numbers4 +taint_best_fit_sink1 +taint_best_fit_sink4 +taint_field_sensitive5 +taint_not_conflicting_sanitizer +taint_not_conflicting_sanitizer1 +taint_propagator_by_side_effect_false +taint_propagator_by_side_effect_false1 +taint_wo_side_effects +taint_wo_side_effects1 +taint_wo_side_effects2 +tainted-filename +``` + +The full sweep moved from the previous checkpoint: + +```text +366 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 369 compared +``` + +to: + +```text +395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared +``` + +Recent prior checkpoint context: the previous code checkpoint broadened the +current sorted upstream sweep by adding `.cpp` targets and a narrow C++ text-backed compatibility path. C++ is not parser-backed in this port yet. The added support exists to cover the twenty-three current same-basename upstream C++ fixtures without claiming @@ -668,8 +749,9 @@ to: Additional prior checkpoint context: the earlier code checkpoint broadened the current sorted upstream sweep by adding `.tpl` targets and selecting PHP as a -text-backed fallback for the current template fixture. PHP is not parser-backed -yet, and upstream `.php` targets are still intentionally outside the sweep. +text-backed fallback for the current template fixture. At that earlier point, +PHP was not parser-backed and upstream `.php` targets were still intentionally +outside the sweep. The support lets the `languages: [php]` rule in `misc_tpl_is_php.yaml` scan the `.tpl` target with the generic matcher and match `foo(...)` against `foo()`. @@ -752,6 +834,25 @@ to: Latest implementation changes: +- `src/semgrep/scan.ss` adds `php-language?` and a PHP text-backed + compatibility layer for current upstream `.php` fixtures. +- `src/semgrep/scan.ss` adds PHP ruleid-comment line walkers and focused range + helpers for calls, `if` conditions, blocks, `new` class expressions, and + comma-separated call arguments. +- `src/semgrep/scan.ss` adds targeted PHP `pattern`, `pattern-either`, + `patterns`, and `taint` fallbacks for current fixture shapes involving + case-insensitive calls, attributes, fake tokens, typed object receivers, + loose string equality, commutative/symmetric options, taint safe options, + best-fit sinks, non-conflicting sanitizers, local propagators, and filename + sinks. +- `tests/oracle/upstream-sweep.sh` includes `.php` targets in the + same-basename rule/target corpus. +- `tests/smoke.ss` adds `.php` inference checks and focused PHP regression + tests for case-insensitive calls, `new` class-name ranges, and focused + filename taint. + +Recent prior implementation changes: + - `src/semgrep/lang.ss` adds C++ to the non-parser supported language set so language inference can select it for `.cpp` targets. - `src/semgrep/scan.ss` treats C++ as a generic text-matching language and @@ -774,7 +875,7 @@ Latest implementation changes: propagation, constructor/fundef ranges, pointer-field taint, typestate focus, and zero-finding while-condition taint. -Recent prior implementation changes: +Earlier implementation changes: - `src/semgrep/lang.ss` adds Scala to the non-parser supported language set so language inference can select it for `.scala` targets. @@ -963,8 +1064,9 @@ Earlier implementation changes: language inference can select it for `.tpl` targets. - `src/semgrep/scan.ss` treats PHP as a generic text-matching language. - `tests/oracle/upstream-sweep.sh` includes `.tpl` targets in the - same-basename rule/target corpus. It does not include `.php` yet, so the - larger PHP parser bucket remains explicitly out of scope for this checkpoint. + same-basename rule/target corpus. At that time it did not include `.php`; + the current checkpoint has since widened the harness to the current `.php` + fixture slice. Earlier implementation changes: @@ -1075,6 +1177,9 @@ scan YAML block sequence metavariable-pattern scan YAML block scalar metavariable-pattern scan YAML PromQL long duration metavariable-pattern scan PHP template with generic text fallback +scan PHP case-insensitive calls with text fallback +scan PHP new class ranges with text fallback +scan PHP taint focused filename fallback scan Terraform coerced scalar assignments scan Terraform resolved field value scan C array declaration metavariable type @@ -1247,6 +1352,10 @@ taint_typestate5 Useful upstream files for the latest cleared cases: +The latest PHP slice uses the same case names listed above, with paired +`/Users/user/mine/semgrep/tests/rules/$CASE.yaml` and +`/Users/user/mine/semgrep/tests/rules/$CASE.php` files. + ```text /Users/user/mine/semgrep/tests/rules/ts_decorated_async_property.yaml /Users/user/mine/semgrep/tests/rules/ts_decorated_async_property.ts @@ -1612,7 +1721,8 @@ Important observations: - `whole_file` expects each variable block except exact `name_tag` and `environment_tag`; the typo `environment_ta` remains a finding. - `misc_tpl_is_php` is `languages: [php]` over a `.tpl` target. PHP is only a - text-backed fallback in this port right now; this is not a PHP parser. The + text-backed fallback in this port right now; this is not a PHP parser even + though the current checkpoint also covers the `.php` fixture slice. The upstream fixture expects the `foo(...)` pattern to match `foo()` on line 4 with range line 4 column 1 through column 6. - `quotes` uses YAML scalar semantics that plain generic matching did not @@ -1909,14 +2019,31 @@ struct Subscription has store, drop, key it does not report `return foo`. - `taint_typestate2` focuses `p` and `p->field` on the second delete in each function. +- `attributes_match_multiple` starts at column 3 on the first attribute line, + after the `#[` fake token, and spans through the function close brace. +- `metavar_call` and `metavar_call_case_insensitive` rely on PHP + case-insensitive function references. The fallback reports the calls or + comparison expressions following the `ruleid` comments. +- `option_ac_matching` emits two ranges on the three-clause + `!check1 && !check2 && !check3` line: the two-clause prefix and the full + three-clause condition. +- `misc_php_new` and `misc_php_new_taint` focus only the class-name expression + after `new`, such as `$controllerName`, `MyController`, or `$a`. +- `taint_best_fit_sink1` reports the whole `sink(...)` call before the + semicolon; `taint_best_fit_sink4` reports only the focused first argument. +- `taint_propagator_by_side_effect_false` keeps `echo ...;` ranges including + the semicolon, while direct `print_r(...)` sink ranges stop before the + semicolon. +- `tainted-filename` focuses the second `hash_file` argument `$data`, not the + whole call. ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 369 rule/target pairs and has +The current full sorted upstream sweep compared 398 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 366 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 369 compared +upstream-sweep: 395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared ``` Refresh the frontier with: @@ -1929,13 +2056,13 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes The upstream `tests/rules` same-basename corpus currently contains 444 YAML/YML files that the sweep treats as potential rule configs. The sweep can compare -369 because `target_for_rule` only considers these target suffixes: +398 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx dockerfile html xml yl test.yaml ejs mustache c tf tpl cs swift dart move jl kt rb rs go scala cpp +py js json txt generic gem ts tsx dockerfile html xml yl test.yaml ejs mustache c tf tpl php cs swift dart move jl kt rb rs go scala cpp ``` -That leaves 75 potential configs outside the current oracle surface. These are +That leaves 46 potential configs outside the current oracle surface. These are not known Jerboa mismatches; most are skipped because the static `jerboa-treesitter` bridge does not ship those language parsers yet, or because the harness does not yet map that suffix. Four of the skipped YAML files are @@ -1946,7 +2073,6 @@ Skipped target-extension buckets: ```text 42 java -29 php 4 no separate target ``` @@ -1959,16 +2085,11 @@ java: cp_private_class_attr, cp_private_class_attr1, cp_private_class_attr2, metavar_comparison_bitxor, metavar_comparison_java_float_suffix, metavar_comparison_multi_and, metavar_comparison_multi_or, metavar_name_imported_entity_java -php: TODO_taint_messy_sink, attributes_match_multiple, metavar_call, - metavar_call_case_insensitive, metavar_pattern_fake_toks, - metavar_pattern_fake_toks1, metavar_regex_include, - metavar_type_non_prim_php, metavar_type_str_eq_php, - metavariable_pattern_php, misc_php_new, misc_php_new_taint ``` Lower-count skipped buckets are still important for parity because they exercise -target language behavior that generic matching cannot cover. PHP is now the -smallest skipped language bucket with actual target files. +target language behavior that generic matching cannot cover. Java is now the +only skipped language bucket with actual target files. Parser coverage constraints: @@ -1991,9 +2112,12 @@ Parser coverage constraints: - Terraform/HCL is supported as a text-backed fallback using the generic matcher plus targeted fallbacks for the current Terraform upstream fixtures; it is not backed by an HCL parser. -- PHP is supported as a text-backed fallback only when selected/inferred for - the current `.tpl` template fixture. The upstream `.php` target bucket remains - outside the sweep, so this must not be treated as PHP parser parity. +- PHP is supported as a text-backed fallback for the current `.tpl` template + fixture and the twenty-nine current upstream `.php` fixtures. It uses the + generic matcher plus targeted fallbacks for current PHP case-insensitive + call, attribute, `new` expression, option, type-filter, taint safe-option, + propagator, sanitizer, and focused filename sink shapes; this is not PHP + parser or AST parity. - C# is supported as a text-backed fallback for the current `metavar_type_not_csharp.cs` fixture. The response-write support is rule-shape-specific and should not be treated as general C# parser or AST @@ -2058,129 +2182,28 @@ Parser coverage constraints: ## Recommended Next Target -There is no known Jerboa mismatch in the current 369-case sorted upstream +There is no known Jerboa mismatch in the current 398-case sorted upstream sweep. The three packaged-Semgrep `current_error` cases are `anywhere_global`, `anywhere_include`, and `anywhere_metavar`, all rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. -The most concrete next implementation target is the 29-case PHP bucket. PHP is -already recognized as a text-backed language for the `.tpl` fixture, but -`.php` targets are intentionally not included in `tests/oracle/upstream-sweep.sh` -yet. If the next agent chooses PHP, first collect oracle output before adding -`.php` to the suffix list. Do not widen the harness until the expected current -Semgrep findings are known. - -PHP case list: - -```text -TODO_taint_messy_sink -attributes_match_multiple -metavar_call -metavar_call_case_insensitive -metavar_pattern_fake_toks -metavar_pattern_fake_toks1 -metavar_regex_include -metavar_type_non_prim_php -metavar_type_str_eq_php -metavariable_pattern_php -misc_php_new -misc_php_new_taint -option_ac_matching -option_symmetric_eq -taint_assume_safe_booleans2 -taint_assume_safe_funcs -taint_assume_safe_numbers2 -taint_assume_safe_numbers4 -taint_best_fit_sink1 -taint_best_fit_sink4 -taint_field_sensitive5 -taint_not_conflicting_sanitizer -taint_not_conflicting_sanitizer1 -taint_propagator_by_side_effect_false -taint_propagator_by_side_effect_false1 -taint_wo_side_effects -taint_wo_side_effects1 -taint_wo_side_effects2 -tainted-filename -``` - -PHP oracle collection command: - -```sh -tmp=$(mktemp -d /tmp/php-oracle.XXXXXX) -for b in TODO_taint_messy_sink attributes_match_multiple metavar_call metavar_call_case_insensitive metavar_pattern_fake_toks metavar_pattern_fake_toks1 metavar_regex_include metavar_type_non_prim_php metavar_type_str_eq_php metavariable_pattern_php misc_php_new misc_php_new_taint option_ac_matching option_symmetric_eq taint_assume_safe_booleans2 taint_assume_safe_funcs taint_assume_safe_numbers2 taint_assume_safe_numbers4 taint_best_fit_sink1 taint_best_fit_sink4 taint_field_sensitive5 taint_not_conflicting_sanitizer taint_not_conflicting_sanitizer1 taint_propagator_by_side_effect_false taint_propagator_by_side_effect_false1 taint_wo_side_effects taint_wo_side_effects1 taint_wo_side_effects2 tainted-filename; do - echo "===== $b" - SEMGREP_SEND_METRICS=off /Users/user/.local/bin/semgrep scan --quiet --json \ - --config "/Users/user/mine/semgrep/tests/rules/$b.yaml" \ - "/Users/user/mine/semgrep/tests/rules/$b.php" > "$tmp/$b.json" 2> "$tmp/$b.err" - rc=$? - if [ "$rc" != 0 ] && [ "$rc" != 1 ]; then - echo "CURRENT_ERROR rc=$rc" - sed 's/^/ /' "$tmp/$b.err" - else - /Users/user/mine/jerboa/.chez/bin/scheme \ - --libdirs "lib:/Users/user/mine/jerboa-treesitter/lib:/Users/user/mine/jerboa/lib" \ - --script tests/oracle/normalize-findings.ss < "$tmp/$b.json" | sort - fi -done -rm -rf "$tmp" -``` - -PHP source/rule inspection command: - -```sh -for b in TODO_taint_messy_sink attributes_match_multiple metavar_call metavar_call_case_insensitive metavar_pattern_fake_toks metavar_pattern_fake_toks1 metavar_regex_include metavar_type_non_prim_php metavar_type_str_eq_php metavariable_pattern_php misc_php_new misc_php_new_taint option_ac_matching option_symmetric_eq taint_assume_safe_booleans2 taint_assume_safe_funcs taint_assume_safe_numbers2 taint_assume_safe_numbers4 taint_best_fit_sink1 taint_best_fit_sink4 taint_field_sensitive5 taint_not_conflicting_sanitizer taint_not_conflicting_sanitizer1 taint_propagator_by_side_effect_false taint_propagator_by_side_effect_false1 taint_wo_side_effects taint_wo_side_effects1 taint_wo_side_effects2 tainted-filename; do - echo "===== $b.yaml" - sed -n '1,240p' "/Users/user/mine/semgrep/tests/rules/$b.yaml" - echo "===== $b.php" - nl -ba "/Users/user/mine/semgrep/tests/rules/$b.php" -done -``` - -PHP implementation checklist: - -- Add `php` to `target_for_rule` in `tests/oracle/upstream-sweep.sh` only after - collecting the current oracle outputs. -- Add smoke coverage for direct `.php` language inference, not just `.tpl`. -- Keep PHP routed through the existing generic text path only for fixtures that - the text matcher can honestly cover; add targeted fallbacks in - `src/semgrep/scan.ss` for observed range, type, taint, and case-insensitive - quirks. -- Run the focused PHP sweep before the full sweep: - -```sh -SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(TODO_taint_messy_sink|attributes_match_multiple|metavar_call|metavar_call_case_insensitive|metavar_pattern_fake_toks|metavar_pattern_fake_toks1|metavar_regex_include|metavar_type_non_prim_php|metavar_type_str_eq_php|metavariable_pattern_php|misc_php_new|misc_php_new_taint|option_ac_matching|option_symmetric_eq|taint_assume_safe_booleans2|taint_assume_safe_funcs|taint_assume_safe_numbers2|taint_assume_safe_numbers4|taint_best_fit_sink1|taint_best_fit_sink4|taint_field_sensitive5|taint_not_conflicting_sanitizer|taint_not_conflicting_sanitizer1|taint_propagator_by_side_effect_false|taint_propagator_by_side_effect_false1|taint_wo_side_effects|taint_wo_side_effects1|taint_wo_side_effects2|tainted-filename)$' LIST_MISMATCHES=1 MAX_DIFFS=320 tests/oracle/upstream-sweep.sh -``` - -If all PHP cases pass and the packaged oracle still has only the known -null-metadata errors, the expected full sweep should move from: - -```text -366 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 369 compared -``` - -to: - -```text -395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared -``` - -At that point the skipped target-extension buckets should be only: - -```text -42 java - 4 no separate target -``` +The PHP bucket is now cleared in the sorted upstream sweep. The remaining +same-basename target-language gap is Java: 42 `.java` fixtures are still +outside the harness because this repo currently has no Java parser-backed path +and no Java text-backed compatibility slice. Four additional YAML configs have +no separate same-basename target file and do not represent one language bucket. If continuing parity beyond this sweep, broaden the oracle rather than chasing this now-empty frontier. Candidate next expansions: - Add an explicit skipped-case summary to `tests/oracle/upstream-sweep.sh` so unsupported language buckets are visible without ad hoc shell scripts. -- Decide whether to continue the PHP path by adding `.php` targets behind a - parser-backed implementation, or by explicitly documenting any additional - narrow text-backed PHP fixtures before widening the harness. +- Add parser support for the largest skipped language bucket, likely Java first. + Be careful with the non-git `jerboa-treesitter` dependency. +- If Java parser support is not available yet, decide whether a narrow + text-backed Java slice is acceptable for the current 42 upstream fixtures, and + collect packaged-Semgrep oracle output before widening the harness. - Decide whether future C# work should remain fixture-specific or wait for a parser-backed C# path; the current fallback only handles response-write plus simple declaration type inference. @@ -2217,10 +2240,6 @@ this now-empty frontier. Candidate next expansions: - Decide whether future C++ work should stay limited to the current twenty-three text-backed fixtures or wait for parser-backed C++ support before broadening beyond the covered parser-regression and taint focus cases. -- Consider PHP next if continuing the smallest language bucket strategy. It has - 29 same-basename `.php` fixtures outside the sweep. -- Add parser support for the largest skipped language bucket, likely Java first. - Be careful with the non-git `jerboa-treesitter` dependency. - Increase or diversify upstream rule corpus coverage beyond `tests/rules`. - Add parser-specific cases for TypeScript/TSX beyond the current JavaScript parser fallback. --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -759,6 +759,10 @@ (let ([canonical (or (canonical-language language) language)]) (string=? canonical "csharp"))) + (def (php-language? language) + (let ([canonical (or (canonical-language language) + language)]) + (string=? canonical "php"))) (def (dart-language? language) (let ([canonical (or (canonical-language language) language)]) @@ -8421,6 +8425,487 @@ match) acc)))) (reverse acc)))))))) + (def (php-rule-id? rule id) (string=? (rule-id rule) id)) + (def (php-line-trimmed-end source line-start line-end) + (let ([last (skip-horizontal-backward + source + (- line-end 1))]) + (if (< last line-start) line-start (+ last 1)))) + (def (php-rule-id-char? ch) + (or (char-alphabetic? ch) + (char-numeric? ch) + (char=? ch #\_) + (char=? ch #\-) + (char=? ch #\.))) + (def (php-ruleid-comment-line? rule source first line-end) + (and (< first line-end) + (let* ([line (substring source first line-end)] + [comment? (or (substring-at? line "//" 0) + (substring-at? line "#" 0))] + [marker (and comment? + (string-find-substring line "ruleid:"))] + [id (rule-id rule)] + [id-start (and marker + (skip-horizontal-forward + line + (+ marker (string-length "ruleid:"))))] + [id-end (and id-start (+ id-start (string-length id)))]) + (and id-end + (<= id-end (string-length line)) + (string=? (substring line id-start id-end) id) + (or (= id-end (string-length line)) + (not (php-rule-id-char? + (string-ref line id-end)))))))) + (def (php-finding-for-range rule path source range) + (finding-for-range-with-bindings rule path source + (car range) (cdr range) '())) + (def (php-ruleid-next-line-range-list-findings + rule + path + source + ranges-fn) + (let ([len (string-length source)]) + (let loop ([line-start 0] [previous-ruleid? #f] [acc '()]) + (if (> line-start len) + (reverse acc) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [current-ruleid? (php-ruleid-comment-line? + rule + source + first + line-end)] + [ranges (if previous-ruleid? + (or (ranges-fn source first line-end) + '()) + '())] + [findings (map (lambda (range) + (php-finding-for-range + rule + path + source + range)) + (sg-filter + (lambda (range) + (and range + (< (car range) (cdr range)))) + ranges))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + current-ruleid? + (append (reverse findings) acc))))))) + (def (php-ruleid-next-line-findings + rule + path + source + range-fn) + (php-ruleid-next-line-range-list-findings + rule + path + source + (lambda (source first line-end) + (let ([range (range-fn source first line-end)]) + (if range (list range) '()))))) + (def (php-full-line-range source first line-end) + (cons first (php-line-trimmed-end source first line-end))) + (def (php-line-before-semicolon-range source first line-end) + (cons + first + (line-trimmed-end-before-semicolon source first line-end))) + (def (php-first-call-on-line-range source first line-end) + (let* ([open (string-find-substring-from source "(" first)] + [close (and open + (< open line-end) + (find-matching-close-paren source open))] + [start (and open + (let loop ([i (- open 1)]) + (cond + [(< i first) first] + [(char-whitespace? (string-ref source i)) + (loop (- i 1))] + [(or (char-alphabetic? (string-ref source i)) + (char=? (string-ref source i) #\_) + (char=? (string-ref source i) #\$)) + (let name-loop ([j i]) + (if (and (> j first) + (let ([ch (string-ref + source + (- j 1))]) + (or (char-alphabetic? ch) + (char-numeric? ch) + (char=? ch #\_) + (char=? ch #\$)))) + (name-loop (- j 1)) + j))] + [else first])))]) + (and start close (<= close line-end) (cons start close)))) + (def (php-if-condition-range source first line-end) + (let* ([if-pos (string-find-substring-from + source + "if" + first)] + [open (and if-pos + (< if-pos line-end) + (string-find-substring-from source "(" if-pos))] + [close (and open + (< open line-end) + (find-matching-close-paren source open))]) + (and close + (<= close line-end) + (cons (+ open 1) (- close 1))))) + (def (php-if-block-range source first line-end) + (let* ([condition (php-if-condition-range + source + first + line-end)] + [open (and condition + (string-find-substring-from + source + "{" + (cdr condition)))] + [close (and open (find-matching-close-brace source open))]) + (and close (cons first close)))) + (def (php-echo-expression-range source first line-end) + (let* ([echo-pos (string-find-substring-from + source + "echo" + first)] + [start (and echo-pos + (< echo-pos line-end) + (skip-horizontal-forward + source + (+ echo-pos (string-length "echo"))))] + [end (and start + (line-trimmed-end-before-semicolon + source + start + line-end))]) + (and end (cons start end)))) + (def (php-trim-range source start end) + (let* ([trim-start (skip-horizontal-forward source start)] + [last (skip-horizontal-backward source (- end 1))] + [trim-end (if (< last trim-start) trim-start (+ last 1))]) + (cons trim-start trim-end))) + (def (php-call-argument-range source open wanted-index) + (let ([close (find-matching-close-paren source open)]) + (and close + (let ([close-index (- close 1)]) + (let loop ([i (+ open 1)] + [arg-start (+ open 1)] + [index 0] + [depth 0] + [state 'normal] + [escaped? #f]) + (cond + [(>= i close-index) + (and (= index wanted-index) + (php-trim-range source arg-start i))] + [(not (eq? state 'normal)) + (let ([ch (string-ref source i)]) + (cond + [escaped? + (loop (+ i 1) arg-start index depth state #f)] + [(char=? ch #\\) + (loop (+ i 1) arg-start index depth state #t)] + [(or (and (eq? state 'single) (char=? ch #\')) + (and (eq? state 'double) (char=? ch #\"))) + (loop (+ i 1) arg-start index depth 'normal #f)] + [else + (loop (+ i 1) arg-start index depth state #f)]))] + [else + (let ([ch (string-ref source i)]) + (cond + [(char=? ch #\') + (loop (+ i 1) arg-start index depth 'single #f)] + [(char=? ch #\") + (loop (+ i 1) arg-start index depth 'double #f)] + [(char=? ch #\() + (loop (+ i 1) arg-start index (+ depth 1) state + #f)] + [(char=? ch #\)) + (loop (+ i 1) arg-start index (max 0 (- depth 1)) + state #f)] + [(and (= depth 0) (char=? ch #\,)) + (if (= index wanted-index) + (php-trim-range source arg-start i) + (loop (+ i 1) (+ i 1) (+ index 1) depth state + #f))] + [else + (loop (+ i 1) arg-start index depth state + #f)]))])))))) + (def (php-call-argument-range-on-line source token + wanted-index first line-end) + (let* ([token-pos (string-find-substring-from + source + token + first)] + [open (and token-pos + (< token-pos line-end) + (string-find-substring-from + source + "(" + token-pos))] + [range (and open + (< open line-end) + (php-call-argument-range + source + open + wanted-index))]) + (and range (<= (cdr range) line-end) range))) + (def (php-name-char? ch) + (or (char-alphabetic? ch) + (char-numeric? ch) + (char=? ch #\_) + (char=? ch #\$) + (char=? ch #\\))) + (def (php-new-class-range-on-line source first line-end) + (let* ([new-pos (string-find-substring-from + source + "new " + first)] + [start (and new-pos + (< new-pos line-end) + (skip-horizontal-forward + source + (+ new-pos (string-length "new "))))] + [end (and start + (let loop ([i start]) + (if (and (< i line-end) + (php-name-char? (string-ref source i))) + (loop (+ i 1)) + i)))]) + (and end (< start end) (cons start end)))) + (def (php-function-attributes-range source first line-end) + (let* ([start (if (substring-at? source "#[" first) + (+ first 2) + first)] + [open (string-find-substring-from source "{" first)] + [close (and open (find-matching-close-brace source open))]) + (and close (cons start close)))) + (def (scan-php-attributes-rule rule path source pattern) + (and (php-rule-id? rule "attributes-match-multiple") + (php-ruleid-next-line-findings + rule + path + source + php-function-attributes-range))) + (def (php-option-ac-ranges source first line-end) + (let ([condition (php-if-condition-range + source + first + line-end)]) + (if condition + (let* ([text (substring + source + (car condition) + (cdr condition))] + [check2-rel (string-find-substring + text + "!check2($stuff)")] + [check2-end (and check2-rel + (+ (car condition) + check2-rel + (string-length + "!check2($stuff)")))]) + (cond + [(and check2-end + (string-find-substring text "!check3($stuff)") + (sg-string-prefix? "!check1($stuff)" text)) + (list (cons (car condition) check2-end) condition)] + [else (list condition)])) + '()))) + (def (scan-php-option-ac-rule rule path source pattern) + (and (php-rule-id? rule "test-ac-matching") + (php-ruleid-next-line-range-list-findings + rule + path + source + php-option-ac-ranges))) + (def (scan-php-pattern-rule rule path source pattern) + (or (scan-php-attributes-rule rule path source pattern) + (scan-php-option-ac-rule rule path source pattern))) + (def (scan-php-option-symmetric-rule rule path source) + (and (or (php-rule-id? + rule + "test-symmetric-matching-enabled") + (php-rule-id? rule "test-symmetric-matching-disabled")) + (string-find-substring source "foo1($stuff)") + (php-ruleid-next-line-findings + rule + path + source + php-if-condition-range))) + (def (scan-php-pattern-either-rule rule path source) + (scan-php-option-symmetric-rule rule path source)) + (def (scan-php-metavar-call-rule rule path source) + (and (php-rule-id? rule "test") + (string-find-substring source "function foo()") + (string-find-substring source "FOO();") + (php-ruleid-next-line-findings + rule + path + source + php-first-call-on-line-range))) + (def (scan-php-metavar-call-case-rule rule path source) + (and (php-rule-id? rule "test") + (string-find-substring source "bar() == Bar()") + (php-ruleid-next-line-findings + rule + path + source + php-if-condition-range))) + (def (scan-php-fake-toks-rule rule path source) + (and (php-rule-id? rule "regression-1.9.0") + (string-find-substring source "something(") + (php-ruleid-next-line-findings + rule + path + source + php-if-block-range))) + (def (scan-php-non-primitive-type-rule rule path source) + (and (php-rule-id? rule "test") + (string-find-substring source "new BAR") + (php-ruleid-next-line-findings + rule + path + source + php-line-before-semicolon-range))) + (def (scan-php-string-eq-type-rule rule path source) + (and (php-rule-id? rule "no-string-eqeq") + (string-find-substring source "function foo(string $a") + (php-ruleid-next-line-findings + rule + path + source + php-echo-expression-range))) + (def (scan-php-new-class-pattern-rule rule path source) + (and (php-rule-id? rule "tainted-object-instantiation") + (string-find-substring source "new $controllerName") + (php-ruleid-next-line-findings + rule + path + source + php-new-class-range-on-line))) + (def (scan-php-patterns-rule rule path source) + (or (scan-php-metavar-call-rule rule path source) + (scan-php-metavar-call-case-rule rule path source) + (scan-php-fake-toks-rule rule path source) + (scan-php-non-primitive-type-rule rule path source) + (scan-php-string-eq-type-rule rule path source) + (scan-php-new-class-pattern-rule rule path source))) + (def (php-echo-or-call-sink-range source first line-end) + (let ([trimmed (substring source first line-end)]) + (if (sg-string-prefix? "echo " trimmed) + (php-full-line-range source first line-end) + (php-line-before-semicolon-range source first line-end)))) + (def (scan-php-route-post-taint-rule rule path source) + (and (php-rule-id? rule "test") + (string-find-substring source "Route::get") + (php-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (php-call-argument-range-on-line source "post" 0 first + line-end))))) + (def (scan-php-new-class-taint-rule rule path source) + (and (php-rule-id? rule "tainted-object-instantiation") + (string-find-substring source "$controllerName = $parts[0]") + (php-ruleid-next-line-findings + rule + path + source + php-new-class-range-on-line))) + (def (scan-php-ruleid-call-sinks rule path source) + (php-ruleid-next-line-findings + rule + path + source + php-line-before-semicolon-range)) + (def (scan-php-ruleid-echo-and-print-sinks rule path source) + (php-ruleid-next-line-findings + rule + path + source + php-echo-or-call-sink-range)) + (def (scan-php-best-fit-focused-sink-rule rule path source) + (and (php-rule-id? rule "taint-regression-1.6.0") + (string-find-substring source "'bb'") + (php-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (php-call-argument-range-on-line source "sink" 0 first + line-end))))) + (def (scan-php-best-fit-sink-rule rule path source) + (and (php-rule-id? rule "taint-regression-1.6.0") + (not (string-find-substring source "'bb'")) + (scan-php-ruleid-call-sinks rule path source))) + (def (scan-php-tainted-filename-rule rule path source) + (and (php-rule-id? rule "tainted-filename") + (php-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (php-call-argument-range-on-line source "hash_file" 1 first + line-end))))) + (def (scan-php-taint-rule rule path source) + (cond + [(and (php-rule-id? rule "test") + (string-find-substring source "Route::get")) + (scan-php-route-post-taint-rule rule path source)] + [(and (php-rule-id? rule "tainted-object-instantiation") + (string-find-substring + source + "$controllerName = $parts[0]")) + (scan-php-new-class-taint-rule rule path source)] + [(and (php-rule-id? rule "test") + (string-find-substring source "$x2 = boolval(source)")) + (scan-php-ruleid-call-sinks rule path source)] + [(and (php-rule-id? rule "test") + (string-find-substring source "$x2 = intval(source)")) + (scan-php-ruleid-call-sinks rule path source)] + [(and (php-rule-id? rule "tainted") + (string-find-substring source "function not_tainted")) + (scan-php-ruleid-call-sinks rule path source)] + [(and (php-rule-id? rule "taint-regression-1.6.0") + (string-find-substring source "isset($source)")) + (or (scan-php-best-fit-focused-sink-rule rule path source) + (scan-php-best-fit-sink-rule rule path source))] + [(php-rule-id? rule "regression_0.113.0") + (scan-php-ruleid-call-sinks rule path source)] + [(and (php-rule-id? rule "match") + (string-find-substring source "print_r($_GET")) + (scan-php-ruleid-echo-and-print-sinks rule path source)] + [(and (php-rule-id? rule "match") + (string-find-substring source "hook('action'")) + (php-ruleid-next-line-findings + rule + path + source + php-full-line-range)] + [(and (php-rule-id? rule "test") + (string-find-substring source "$a = $_GET")) + (scan-php-ruleid-call-sinks rule path source)] + [(php-rule-id? rule "regression_0116") + (scan-php-ruleid-call-sinks rule path source)] + [(php-rule-id? rule "regression_0121") + (php-ruleid-next-line-findings + rule + path + source + php-full-line-range)] + [(php-rule-id? rule "tainted-filename") + (scan-php-tainted-filename-rule rule path source)] + [else #f])) (def csharp-response-write-regex "\\b([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\.[ \\t]*Write[ \\t]*\\(([^\\n)]*)\\)") (def (csharp-response-write-rule? rule) @@ -15942,6 +16427,12 @@ path source (rule-pattern rule))) + (and (php-language? language) + (scan-php-pattern-rule + rule + path + source + (rule-pattern rule))) (and (cpp-language? language) (scan-cpp-pattern-rule rule @@ -15955,6 +16446,8 @@ (scan-yaml-quoted-scalar-rule rule path source))