Merge: java metavariable-type structural + patterns-rule overfit paydown
ober
b594b515de3481a78abd1bf817d20a007f45f74e
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -151,17 +151,22 @@ languages, so closing them is the tip of the larger work: - **C / C++** (cpp 23 fixtures) — handled via `c-like-language?` (reachability filter etc.); migrating loses that the way Dart lost dead-catch. Hardest. - **Overfit-handler boundary (measured).** The per-language - `scan-{lang}-patterns-rule` / `scan-{lang}-taint-rule` handlers still fire and - still carry real work. With java's handlers disabled, only 16/42 java fixtures - pass via pure structural; the other 26 need the handlers and are almost all - taint (`taint_*`), metavariable-comparison (`metavar_comparison_*`), or - type-resolution (`metavar_type_*`, `metavariable_name_resolution`) cases. So - `wrapper-root-type?` made *pattern* matching real, but taint / - metavariable-comparison / type-resolution remain handler-side. Removing the - overfit handlers (and finishing dart/c/cpp) means porting those three feature - families onto the structural path — the substantial, well-scoped next project. - The same gap blocked the dart taint smoke cases. + **Overfit-handler paydown (in progress, java first).** Measured: with java's + patterns-rule handler disabled only 16/42 java fixtures passed structurally; + the rest were metavariable-comparison, metavariable-type, and taint. Now CLOSED + on the structural path for java: + - metavariable-comparison (`metavar_comparison_*`): java constant-prop regex + feeds the existing comparison evaluator (bitwise/logical already supported); + return_statement added to semicolon-trimmable-node?. + - metavariable-type (`metavar_type_not_java`): declared-type resolution already + worked; adding string_literal to string-literal-node? let `"..."` match java + string args (so the pattern-not fires). + `scan-java-patterns-rule` is now reduced to a single clause + (`metavariable-resolution-test`, which needs FQN resolution Foo->org.foo.Foo). + STILL handler-side (the deep remaining work): java TAINT (`scan-java-taint-rule` + — real dataflow, incl. type-aware assume-safe, best-fit-sink, lambda/getter + propagators), java FQN resolution, and the analogous handlers for the other + migrated languages. The dart taint smoke cases hit the same taint gap. The original concern below (regex matcher limitations) still applies only to the languages still on `generic-language?`: `generic-language?` --- a/lib/semgrep/match/structural.sls +++ b/lib/semgrep/match/structural.sls @@ -406,7 +406,8 @@ (char=? open close)))))) (def (string-literal-node? n) (or (string=? (node-type n) "string") - (string=? (node-type n) "template_string"))) + (string=? (node-type n) "template_string") + (string=? (node-type n) "string_literal"))) (def (string-contains-substring? source needle) (let ([needle-len (string-length needle)] [len (string-length source)]) --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -24781,16 +24781,7 @@ [else #f])) (def (scan-java-patterns-rule rule path source) (cond - [(java-rule-id? rule "test-template") '()] - [(java-rule-id? rule "java-float-double-suffix-comparison") - (scan-java-float-suffix-rule rule path source)] - [(java-rule-id? rule "no-string-eqeq") - (scan-java-string-eq-rule rule path source)] - [(java-rule-id? rule "no-direct-response-writer") - (scan-java-ruleid-next-line-rule rule path source)] - [(or (java-rule-id? rule "metavariable-resolution-test") - (java-rule-id? rule "insecure-crypto-usage") - (java-rule-id? rule "test")) + [(java-rule-id? rule "metavariable-resolution-test") (scan-java-ruleid-next-line-rule rule path source)] [else #f])) (def (scan-java-taint-rule rule path source) --- a/src/semgrep/match/structural.ss +++ b/src/semgrep/match/structural.ss @@ -408,7 +408,8 @@ (def (string-literal-node? n) (or (string=? (node-type n) "string") - (string=? (node-type n) "template_string"))) + (string=? (node-type n) "template_string") + (string=? (node-type n) "string_literal"))) (def (string-contains-substring? source needle) (let ([needle-len (string-length needle)] --- a/src/semgrep/scan.ss +++ b/src/semgrep/scan.ss @@ -24880,17 +24880,11 @@ [else #f])) (def (scan-java-patterns-rule rule path source) + ;; Structural matching now covers java pattern rules directly; only + ;; metavariable-resolution-test still needs the handler (it requires + ;; fully-qualified-name resolution, e.g. Foo -> org.foo.Foo via imports). (cond - [(java-rule-id? rule "test-template") '()] - [(java-rule-id? rule "java-float-double-suffix-comparison") - (scan-java-float-suffix-rule rule path source)] - [(java-rule-id? rule "no-string-eqeq") - (scan-java-string-eq-rule rule path source)] - [(java-rule-id? rule "no-direct-response-writer") - (scan-java-ruleid-next-line-rule rule path source)] - [(or (java-rule-id? rule "metavariable-resolution-test") - (java-rule-id? rule "insecure-crypto-usage") - (java-rule-id? rule "test")) + [(java-rule-id? rule "metavariable-resolution-test") (scan-java-ruleid-next-line-rule rule path source)] [else #f]))