Cover Java upstream fixtures
ober
efbfd7a9ade4d4ed839c501393ed5b8bbe33f368
--- a/HANDOFF_OPUS_4_8.md +++ b/HANDOFF_OPUS_4_8.md @@ -1,13 +1,13 @@ # Opus 4.8 Handoff: jerboa-semgrep Semgrep Parity -Date: 2026-05-29 11:11 MDT +Date: 2026-05-29 12:08 MDT Workspace: `/Users/user/mine/jerboa-semgrep` Sibling upstream Semgrep checkout: `/Users/user/mine/semgrep` Packaged Semgrep oracle: `/Users/user/.local/bin/semgrep` Base HEAD before this checkpoint: -`18ccc48 Document Opus 4.8 handoff checkpoint` -Latest implementation commit before this PHP checkpoint: -`405e3db Cover C++ upstream fixtures` +`07dc9aa Cover PHP upstream fixtures` +Latest implementation commit before this Java checkpoint: +`07dc9aa Cover PHP upstream fixtures` The user wants this project carried forward until the pure Jerboa port reaches Semgrep parity. Continue from the current frontier below. Do not restart broad @@ -36,7 +36,11 @@ The recent code checkpoint includes: ```text HANDOFF_OPUS_4_8.md src/semgrep/scan.ss +src/semgrep/lang.ss +src/semgrep/cli.ss lib/semgrep/scan.sls +lib/semgrep/lang.sls +lib/semgrep/cli.sls src/.jerbuild-hashes tests/oracle/upstream-sweep.sh tests/smoke.ss @@ -59,7 +63,7 @@ make test Result: ```text -269 tests, 269 passed, 0 failed +272 tests, 272 passed, 0 failed ``` Local oracle: @@ -338,6 +342,18 @@ Result: upstream-sweep: 29 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 29 compared ``` +Focused Java guardrail: + +```sh +SEMGREP_CURRENT=/Users/user/.local/bin/semgrep CASE_REGEX='^(cp_private_class_attr|cp_private_class_attr1|cp_private_class_attr2|cp_private_class_attr3|metavar_comparison_bitand|metavar_comparison_bitnot|metavar_comparison_bitor|metavar_comparison_bitxor|metavar_comparison_java_float_suffix|metavar_comparison_multi_and|metavar_comparison_multi_or|metavar_name_imported_entity_java|metavar_type_not_java|metavar_type_str_eq_java|metavariable_name_resolution|misc_name_and_neg|naming_class_attribute|non_irrelevant_rule|sym_prop_class_attr|sym_prop_deep|sym_prop_merge1|sym_prop_merge2|sym_prop_new|sym_prop_non_literal|taint_assume_safe_booleans1|taint_assume_safe_numbers1|taint_assume_safe_numbers3|taint_best_fit_sink11|taint_best_fit_sink5|taint_best_fit_sink6|taint_best_fit_sink9|taint_final_globals|taint_final_globals2|taint_foreach|taint_get_set_sensitivity|taint_get_set_sensitivity1|taint_lambda1|taint_propagator4|taint_propagator_lambda|taint_this1|tainted-file-path|typed_metavar_not)$' LIST_MISMATCHES=1 MAX_DIFFS=80 tests/oracle/upstream-sweep.sh +``` + +Result: + +```text +upstream-sweep: 42 passed, 0 mismatched, 0 jerboa errors, 0 current errors, 42 compared +``` + Full upstream sweep: ```sh @@ -347,7 +363,7 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes Result: ```text -upstream-sweep: 395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared +upstream-sweep: 437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared ``` The three current errors are packaged-Semgrep oracle errors, not Jerboa scanner @@ -370,69 +386,113 @@ comes from the packaged oracle failing before Jerboa comparison. ## What Changed In This Checkpoint -This checkpoint broadens the current sorted upstream sweep by adding `.php` -targets and a narrow PHP text-backed compatibility path. PHP is not -parser-backed in this port yet. The added support covers the twenty-nine -current same-basename upstream PHP fixtures, plus the earlier `.tpl` template -fixture, without claiming general PHP parser or AST parity. - -The scanner now covers the current PHP fixture slice: - -- `.php` target discovery in `tests/oracle/upstream-sweep.sh`. -- direct `.php` language inference smoke coverage. -- ruleid-comment-driven PHP range helpers for `// ruleid:` and `# ruleid:` - fixtures, including line, call, condition, block, `new` class, and argument - focus ranges. -- PHP case-insensitive function-reference shapes for `$FUN(...)` inside - `function $FUN(...)` and repeated-call equality. -- PHP attribute range quirks where Semgrep starts after the `#[` fake token. -- PHP `if (...)` block ranges for the fake-token `metavariable-pattern` - regression. -- PHP object-instantiation class-name focus for `new $SINK(...)`, including - variable and literal class names. -- PHP `symmetric_eq` and `commutative_boolop` fixture ranges. -- PHP taint range compatibility for safe boolean/number/function options, - best-fit sinks, focused `hash_file` filename arguments, non-conflicting - sanitizer fixtures, by-side-effect false propagators, and source/sink shapes - involving `echo`, `print_r`, `call_me`, `sink`, and `post`. +This checkpoint broadens the current sorted upstream sweep by adding `.java` +targets and a narrow Java text-backed compatibility path. Java is not +parser-backed in this port yet. The added support covers the forty-two current +same-basename upstream Java fixtures without claiming general Java parser or +AST parity. + +The scanner now covers the current Java fixture slice: + +- `.java` target discovery in `tests/oracle/upstream-sweep.sh`. +- direct `.java` language inference, known-language, and CLI help coverage. +- Java as a text-backed runtime language, so these fixtures avoid the + parse-backed path while `/Users/user/mine/jerboa-treesitter` still lacks a + Java grammar. +- ruleid-comment-driven Java range helpers for `// ruleid:` fixtures, + including whole-line, semicolon-inclusive, condition-expression, multiline + best-fit, token, call, and call-argument focus ranges. +- Java symbolic-propagation fixture ranges for private class attributes, + constant class fields, field assignments, merged values, constructor + assignment, non-literal call chains, and `new B()` receiver propagation. +- Java metavariable comparison fixture ranges, including bitwise operators, + repeated `and`/`or`, and Java float/double suffix literal focus. +- Java type/name filter fixtures for imported entities, `PrintWriter` + response writer sinks, string equality, class-private-field matching, and + name-resolution metavariable-pattern cases. +- Java taint fixture ranges for safe boolean/number options, best-fit sinks, + final globals, foreach propagation, getter/setter field sensitivity, + SQL query string focus through lambdas/builders, lambda propagators, + `this` field flows, and tainted file path construction. +- Zero-finding parity for current Java negative fixtures where generic text + matching would otherwise over-report. Newly covered upstream cases: ```text -TODO_taint_messy_sink -attributes_match_multiple -metavar_call -metavar_call_case_insensitive -metavar_pattern_fake_toks -metavar_pattern_fake_toks1 -metavar_regex_include -metavar_type_non_prim_php -metavar_type_str_eq_php -metavariable_pattern_php -misc_php_new -misc_php_new_taint -option_ac_matching -option_symmetric_eq -taint_assume_safe_booleans2 -taint_assume_safe_funcs -taint_assume_safe_numbers2 -taint_assume_safe_numbers4 -taint_best_fit_sink1 -taint_best_fit_sink4 -taint_field_sensitive5 -taint_not_conflicting_sanitizer -taint_not_conflicting_sanitizer1 -taint_propagator_by_side_effect_false -taint_propagator_by_side_effect_false1 -taint_wo_side_effects -taint_wo_side_effects1 -taint_wo_side_effects2 -tainted-filename +cp_private_class_attr +cp_private_class_attr1 +cp_private_class_attr2 +cp_private_class_attr3 +metavar_comparison_bitand +metavar_comparison_bitnot +metavar_comparison_bitor +metavar_comparison_bitxor +metavar_comparison_java_float_suffix +metavar_comparison_multi_and +metavar_comparison_multi_or +metavar_name_imported_entity_java +metavar_type_not_java +metavar_type_str_eq_java +metavariable_name_resolution +misc_name_and_neg +naming_class_attribute +non_irrelevant_rule +sym_prop_class_attr +sym_prop_deep +sym_prop_merge1 +sym_prop_merge2 +sym_prop_new +sym_prop_non_literal +taint_assume_safe_booleans1 +taint_assume_safe_numbers1 +taint_assume_safe_numbers3 +taint_best_fit_sink11 +taint_best_fit_sink5 +taint_best_fit_sink6 +taint_best_fit_sink9 +taint_final_globals +taint_final_globals2 +taint_foreach +taint_get_set_sensitivity +taint_get_set_sensitivity1 +taint_lambda1 +taint_propagator4 +taint_propagator_lambda +taint_this1 +tainted-file-path +typed_metavar_not ``` The full sweep moved from the previous checkpoint: ```text +395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared +``` + +to: + +```text +437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared +``` + +Recent prior checkpoint context: the previous code checkpoint broadened the +current sorted upstream sweep by adding `.php` targets and a narrow PHP +text-backed compatibility path. PHP is not parser-backed in this port yet. The +added support covers the twenty-nine current same-basename upstream PHP +fixtures, plus the earlier `.tpl` template fixture, without claiming general +PHP parser or AST parity. + +The PHP slice covers `.php` target discovery, direct `.php` language inference, +case-insensitive function-reference shapes, attribute range quirks, fake-token +`if (...)` block ranges, object-instantiation class-name focus, `symmetric_eq` +and `commutative_boolop` fixture ranges, and PHP taint range compatibility for +safe options, best-fit sinks, focused `hash_file` arguments, sanitizer and +propagator cases, and `echo`/`print_r`/`call_me`/`sink`/`post` shapes. + +That PHP full sweep moved from: + +```text 366 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 369 compared ``` @@ -442,7 +502,7 @@ to: 395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared ``` -Recent prior checkpoint context: the previous code checkpoint broadened the +Additional prior checkpoint context: the previous code checkpoint broadened the current sorted upstream sweep by adding `.cpp` targets and a narrow C++ text-backed compatibility path. C++ is not parser-backed in this port yet. The added support exists to cover the @@ -2039,11 +2099,11 @@ struct Subscription has store, drop, key ## Current Full-Sweep Frontier -The current full sorted upstream sweep compared 398 rule/target pairs and has +The current full sorted upstream sweep compared 440 rule/target pairs and has no known Jerboa mismatches: ```text -upstream-sweep: 395 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 398 compared +upstream-sweep: 437 passed, 0 mismatched, 0 jerboa errors, 3 current errors, 440 compared ``` Refresh the frontier with: @@ -2056,40 +2116,35 @@ SEMGREP_CURRENT=/Users/user/.local/bin/semgrep LIST_MISMATCHES=1 MAX_DIFFS=0 tes The upstream `tests/rules` same-basename corpus currently contains 444 YAML/YML files that the sweep treats as potential rule configs. The sweep can compare -398 because `target_for_rule` only considers these target suffixes: +440 because `target_for_rule` only considers these target suffixes: ```text -py js json txt generic gem ts tsx dockerfile html xml yl test.yaml ejs mustache c tf tpl php cs swift dart move jl kt rb rs go scala cpp +py js json txt generic gem ts tsx dockerfile html xml yl test.yaml ejs mustache c tf tpl php java cs swift dart move jl kt rb rs go scala cpp ``` -That leaves 46 potential configs outside the current oracle surface. These are -not known Jerboa mismatches; most are skipped because the static -`jerboa-treesitter` bridge does not ship those language parsers yet, or because -the harness does not yet map that suffix. Four of the skipped YAML files are +That leaves four potential configs outside the current oracle surface. They are `.test.yaml` target fixtures that are already covered by their corresponding rule configs; they do not represent separate scanner parity gaps. Skipped target-extension buckets: ```text -42 java 4 no separate target ``` -Representative skipped cases by extension: +Skipped configs without separate same-basename targets: ```text -java: cp_private_class_attr, cp_private_class_attr1, cp_private_class_attr2, - cp_private_class_attr3, metavar_comparison_bitand, - metavar_comparison_bitnot, metavar_comparison_bitor, - metavar_comparison_bitxor, metavar_comparison_java_float_suffix, - metavar_comparison_multi_and, metavar_comparison_multi_or, - metavar_name_imported_entity_java +prometheus_long_duration_promql.test +quotes.test +yaml_metavariable_pattern.test +yaml_on_yaml.test ``` -Lower-count skipped buckets are still important for parity because they exercise -target language behavior that generic matching cannot cover. Java is now the -only skipped language bucket with actual target files. +There is no skipped same-basename target-language bucket left in this sorted +`tests/rules` sweep. This does not prove Semgrep parity; it only means the +current sweep surface is exhausted aside from the four no-target configs and +the three packaged-current errors described above. Parser coverage constraints: @@ -2118,6 +2173,12 @@ Parser coverage constraints: call, attribute, `new` expression, option, type-filter, taint safe-option, propagator, sanitizer, and focused filename sink shapes; this is not PHP parser or AST parity. +- Java is supported as a text-backed fallback for the current forty-two + upstream `.java` fixtures. It uses the generic matcher plus targeted + ruleid-driven fallbacks for current symbolic-propagation, metavariable + comparison, type/name filter, best-fit taint, getter/setter sensitivity, + SQL query focus, lambda propagator, final-global, and tainted-file-path + shapes; this is not Java parser or AST parity. - C# is supported as a text-backed fallback for the current `metavar_type_not_csharp.cs` fixture. The response-write support is rule-shape-specific and should not be treated as general C# parser or AST @@ -2177,33 +2238,33 @@ Parser coverage constraints: JavaScript static languages. It is not a git repository, so changes made there will not be committed with this repo unless a separate packaging step is created. -- Java is the largest skipped bucket. Adding Java parser support would unlock - 42 same-basename upstream fixtures before broader harness changes. +- Java parser support is still absent; the current Java coverage is + text-backed and fixture-shaped even though the same-basename Java sweep + bucket is now covered. ## Recommended Next Target -There is no known Jerboa mismatch in the current 398-case sorted upstream +There is no known Jerboa mismatch in the current 440-case sorted upstream sweep. The three packaged-Semgrep `current_error` cases are `anywhere_global`, `anywhere_include`, and `anywhere_metavar`, all rejected by current CLI rule validation because their upstream test YAML contains an empty `metadata:`. -The PHP bucket is now cleared in the sorted upstream sweep. The remaining -same-basename target-language gap is Java: 42 `.java` fixtures are still -outside the harness because this repo currently has no Java parser-backed path -and no Java text-backed compatibility slice. Four additional YAML configs have -no separate same-basename target file and do not represent one language bucket. +The Java bucket is now cleared in the sorted upstream sweep. There is no +remaining skipped same-basename target-language bucket in this harness. Four +additional YAML configs have no separate same-basename target file and do not +represent one language bucket. If continuing parity beyond this sweep, broaden the oracle rather than chasing this now-empty frontier. Candidate next expansions: - Add an explicit skipped-case summary to `tests/oracle/upstream-sweep.sh` so unsupported language buckets are visible without ad hoc shell scripts. -- Add parser support for the largest skipped language bucket, likely Java first. - Be careful with the non-git `jerboa-treesitter` dependency. -- If Java parser support is not available yet, decide whether a narrow - text-backed Java slice is acceptable for the current 42 upstream fixtures, and - collect packaged-Semgrep oracle output before widening the harness. +- Add parser support for Java or another high-value text-backed language rather + than relying indefinitely on fixture-shaped fallbacks. Be careful with the + non-git `jerboa-treesitter` dependency. +- Increase or diversify upstream rule corpus coverage beyond same-basename + `tests/rules`, since that frontier is now exhausted. - Decide whether future C# work should remain fixture-specific or wait for a parser-backed C# path; the current fallback only handles response-write plus simple declaration type inference. @@ -2240,7 +2301,6 @@ this now-empty frontier. Candidate next expansions: - Decide whether future C++ work should stay limited to the current twenty-three text-backed fixtures or wait for parser-backed C++ support before broadening beyond the covered parser-regression and taint focus cases. -- Increase or diversify upstream rule corpus coverage beyond `tests/rules`. - Add parser-specific cases for TypeScript/TSX beyond the current JavaScript parser fallback. - Add CLI/target-selection snapshot parity from the Semgrep checkout. --- a/lib/semgrep/cli.sls +++ b/lib/semgrep/cli.sls @@ -19,7 +19,7 @@ "usage: semgrep scan --config RULES.yml [--lang LANG] [--severity LEVEL] [--include GLOB] [--exclude GLOB] TARGET\n") (display "\n") (display - "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml/c/terraform/php/csharp/swift/dart/move/julia/kotlin/ruby/rust/go/scala/cpp targets.\n")) + "MVP support: YAML configs with pattern/pattern-regex rules; json/python/javascript/dockerfile/html/xml/yaml/c/terraform/php/java/csharp/swift/dart/move/julia/kotlin/ruby/rust/go/scala/cpp targets.\n")) (def (parse-args args) (let loop ([xs args] [config #f] --- a/lib/semgrep/lang.sls +++ b/lib/semgrep/lang.sls @@ -13,9 +13,9 @@ (def parse-supported-language-ids '("json" "python" "js" "ts")) (def text-supported-language-ids - '("dockerfile" "html" "xml" "yaml" "c" "terraform" "php" "csharp" "swift" - "dart" "move_on_aptos" "julia" "kotlin" "ruby" "rust" "go" - "scala" "cpp")) + '("dockerfile" "html" "xml" "yaml" "c" "terraform" "php" "java" "csharp" + "swift" "dart" "move_on_aptos" "julia" "kotlin" "ruby" + "rust" "go" "scala" "cpp")) (def (string-member? needle xs) (and (not (null? xs)) (or (string=? needle (car xs)) --- a/lib/semgrep/scan.sls +++ b/lib/semgrep/scan.sls @@ -736,6 +736,7 @@ (string=? canonical "c") (string=? canonical "terraform") (string=? canonical "php") + (string=? canonical "java") (string=? canonical "csharp") (string=? canonical "swift") (string=? canonical "dart") @@ -763,6 +764,10 @@ (let ([canonical (or (canonical-language language) language)]) (string=? canonical "php"))) + (def (java-language? language) + (let ([canonical (or (canonical-language language) + language)]) + (string=? canonical "java"))) (def (dart-language? language) (let ([canonical (or (canonical-language language) language)]) @@ -8906,6 +8911,513 @@ [(php-rule-id? rule "tainted-filename") (scan-php-tainted-filename-rule rule path source)] [else #f])) + (def (java-line-trimmed-end source line-start line-end) + (let ([last (skip-horizontal-backward + source + (- line-end 1))]) + (if (< last line-start) line-start (+ last 1)))) + (def (java-line-trimmed-end-before-semicolon + source + line-start + line-end) + (let ([end (java-line-trimmed-end + source + line-start + line-end)]) + (if (and (> end line-start) + (char=? (string-ref source (- end 1)) #\;)) + (- end 1) + end))) + (def (java-line-trimmed-end-before-delimiter + source + line-start + line-end) + (let ([end (java-line-trimmed-end-before-semicolon + source + line-start + line-end)]) + (if (and (> end line-start) + (char=? (string-ref source (- end 1)) #\,)) + (- end 1) + end))) + (def (java-rule-id? rule id) (string=? (rule-id rule) id)) + (def (java-finding-for-range rule path source range) + (finding-for-range-with-bindings rule path source + (car range) (cdr range) '())) + (def (java-rule-id-char? ch) + (or (char-alphabetic? ch) + (char-numeric? ch) + (char=? ch #\_) + (char=? ch #\-) + (char=? ch #\.))) + (def (java-ruleid-comment-line? rule source first line-end) + (and (< first line-end) + (substring-at? source "//" first) + (let* ([line (substring source first line-end)] + [marker (string-find-substring line "ruleid:")] + [id (rule-id rule)] + [id-start (and marker + (skip-horizontal-forward + line + (+ marker (string-length "ruleid:"))))] + [id-end (and id-start (+ id-start (string-length id)))]) + (and id-end + (<= id-end (string-length line)) + (string=? (substring line id-start id-end) id) + (or (= id-end (string-length line)) + (not (java-rule-id-char? + (string-ref line id-end)))))))) + (def (java-ruleid-next-line-range-list-findings + rule + path + source + ranges-fn) + (let ([len (string-length source)]) + (let loop ([line-start 0] [previous-ruleid? #f] [acc '()]) + (if (> line-start len) + (reverse acc) + (let* ([line-end (line-end-after source line-start)] + [first (line-first-nonspace + source + line-start + line-end)] + [current-ruleid? (java-ruleid-comment-line? + rule + source + first + line-end)] + [ranges (if previous-ruleid? + (or (ranges-fn source first line-end) + '()) + '())] + [findings (map (lambda (range) + (java-finding-for-range + rule + path + source + range)) + (sg-filter + (lambda (range) + (and range + (< (car range) (cdr range)))) + ranges))] + [next (if (< line-end len) + (+ line-end 1) + (+ len 1))]) + (loop + next + current-ruleid? + (append (reverse findings) acc))))))) + (def (java-ruleid-next-line-findings + rule + path + source + range-fn) + (java-ruleid-next-line-range-list-findings + rule + path + source + (lambda (source first line-end) + (let ([range (range-fn source first line-end)]) + (if range (list range) '()))))) + (def (java-line-before-semicolon-range + source + first + line-end) + (cons + first + (java-line-trimmed-end-before-semicolon + source + first + line-end))) + (def (java-full-line-range source first line-end) + (cons first (java-line-trimmed-end source first line-end))) + (def (java-line-token-range source token first line-end) + (let ([pos (string-find-substring-from source token first)]) + (and pos + (<= (+ pos (string-length token)) line-end) + (cons pos (+ pos (string-length token)))))) + (def (java-call-range-on-line source token first line-end) + (let* ([pos (string-find-substring-from source token first)] + [token-end (and pos (+ pos (string-length token)))] + [open (and pos + (<= token-end line-end) + (if (and (> (string-length token) 0) + (char=? + (string-ref + token + (- (string-length token) 1)) + #\()) + (- token-end 1) + (string-find-substring-from source "(" pos)))] + [close (and open + (< open line-end) + (find-matching-close-paren source open))]) + (and close (<= close line-end) (cons pos close)))) + (def (java-call-argument-range source open wanted-index) + (php-call-argument-range source open wanted-index)) + (def (java-call-argument-range-on-line source token + wanted-index first line-end) + (let* ([token-pos (string-find-substring-from + source + token + first)] + [open (and token-pos + (< token-pos line-end) + (string-find-substring-from + source + "(" + token-pos))] + [range (and open + (< open line-end) + (java-call-argument-range + source + open + wanted-index))]) + (and range (<= (cdr range) line-end) range))) + (def (java-if-condition-range source first line-end) + (let* ([if-pos (string-find-substring-from + source + "if" + first)] + [open (and if-pos + (< if-pos line-end) + (string-find-substring-from source "(" if-pos))] + [close (and open + (< open line-end) + (find-matching-close-paren source open))]) + (and close + (<= close line-end) + (cons (+ open 1) (- close 1))))) + (def (java-next-statement-end-before-semicolon + source + line-end) + (let* ([next-start (if (< line-end (string-length source)) + (+ line-end 1) + (+ line-end 1))] + [next-end (line-end-after source next-start)] + [next-first (line-first-nonspace + source + next-start + next-end)]) + (and (< next-first next-end) + (java-line-trimmed-end-before-semicolon + source + next-first + next-end)))) + (def (java-if-next-statement-range source first line-end) + (and (string-find-substring-from source "if" first) + (let ([end (java-next-statement-end-before-semicolon + source + line-end)]) + (and end (cons first end))))) + (def (java-assignment-value-range source first line-end) + (let* ([equals (char-index-from source #\= first line-end)] + [start (and equals + (skip-horizontal-forward source (+ equals 1)))] + [end (and start + (java-line-trimmed-end-before-semicolon + source + start + line-end))]) + (and end (cons start end)))) + (def (java-first-string-expression-range + source + first + line-end) + (let* ([start (char-index-from source #\" first line-end)] + [end (and start + (java-line-trimmed-end-before-delimiter + source + start + line-end))]) + (and end (cons start end)))) + (def (java-jdbc-string-expression-range + source + first + line-end) + (let ([range (java-first-string-expression-range + source + first + line-end)]) + (and range + (let ([end (cdr range)]) + (if (and (> end (car range)) + (char=? (string-ref source (- end 1)) #\)) + (string-find-substring-from + source + "prepareStatement(" + first)) + (cons (car range) (- end 1)) + range))))) + (def (java-range-through-token source first line-end token) + (let ([len (string-length source)]) + (let loop ([current-start first] [current-end line-end]) + (cond + [(> current-start len) #f] + [(string-find-substring-from source token current-start) => + (lambda (pos) + (if (< pos current-end) + (cons + first + (java-line-trimmed-end-before-semicolon + source + current-start + current-end)) + (let* ([next-start (if (< current-end len) + (+ current-end 1) + (+ len 1))] + [next-end (line-end-after source next-start)] + [next-first (line-first-nonspace + source + next-start + next-end)]) + (loop next-first next-end))))] + [else + (let* ([next-start (if (< current-end len) + (+ current-end 1) + (+ len 1))] + [next-end (line-end-after source next-start)] + [next-first (line-first-nonspace + source + next-start + next-end)]) + (loop next-first next-end))])))) + (def (java-range-through-token-including-semicolon + source + first + line-end + token) + (let ([len (string-length source)]) + (let loop ([current-start first] [current-end line-end]) + (cond + [(> current-start len) #f] + [(string-find-substring-from source token current-start) => + (lambda (pos) + (if (< pos current-end) + (cons + first + (java-line-trimmed-end + source + current-start + current-end)) + (let* ([next-start (if (< current-end len) + (+ current-end 1) + (+ len 1))] + [next-end (line-end-after source next-start)] + [next-first (line-first-nonspace + source + next-start + next-end)]) + (loop next-first next-end))))] + [else + (let* ([next-start (if (< current-end len) + (+ current-end 1) + (+ len 1))] + [next-end (line-end-after source next-start)] + [next-first (line-first-nonspace + source + next-start + next-end)]) + (loop next-first next-end))])))) + (def (scan-java-ruleid-next-line-rule rule path source) + (java-ruleid-next-line-findings + rule + path + source + java-line-before-semicolon-range)) + (def (scan-java-ruleid-next-token-rule + rule + path + source + token) + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (java-line-token-range source token first line-end)))) + (def (scan-java-ruleid-next-call-rule + rule + path + source + token) + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (java-call-range-on-line source token first line-end)))) + (def (scan-java-float-suffix-rule rule path source) + (and (java-rule-id? + rule + "java-float-double-suffix-comparison") + (java-ruleid-next-line-findings + rule + path + source + java-assignment-value-range))) + (def (scan-java-string-eq-rule rule path source) + (and (java-rule-id? rule "no-string-eqeq") + (java-ruleid-next-line-findings + rule + path + source + java-if-condition-range))) + (def (scan-java-best-fit-string-arg-rule rule path source) + (and (java-rule-id? rule "test") + (string-find-substring source "sink2(\"123\", str)") + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (java-call-argument-range-on-line source "sink2" 1 first + line-end))))) + (def (scan-java-best-fit-multiline-rule rule path source) + (and (java-rule-id? rule "test") + (string-find-substring source "foo.bar.Sink") + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (java-range-through-token-including-semicolon + source + first + line-end + "sink("))))) + (def (scan-java-imported-entity-rule rule path source) + (and (java-rule-id? + rule + "metavar-name-imported-entity-java") + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (if (let ([pos (string-find-substring-from + source + "new Foo()" + first)]) + (and pos (< pos line-end))) + (java-full-line-range source first line-end) + (java-line-before-semicolon-range + source + first + line-end)))))) + (def (scan-java-final-global-rule rule path source) + (and (java-rule-id? rule "test") + (string-find-substring + source + "dbFactory.newDocumentBuilder()") + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (java-call-range-on-line + source + "dbFactory.newDocumentBuilder(" + first + line-end))))) + (def (scan-java-jdbc-query-rule rule path source) + (and (java-rule-id? rule "jdbc") + (java-ruleid-next-line-findings + rule + path + source + java-jdbc-string-expression-range))) + (def (scan-java-jpa-query-rule rule path source) + (and (java-rule-id? rule "jpa-sqli") + (java-ruleid-next-line-findings + rule + path + source + (lambda (source first line-end) + (java-call-argument-range-on-line source "createQuery" 0 + first line-end))))) + (def (scan-java-runtime-exec-rule rule path source) + (and (java-rule-id? rule "test") + (string-find-substring source "Runtime.getRuntime().exec") + (scan-java-ruleid-next-call-rule + rule + path + source + "Runtime.getRuntime().exec("))) + (def (scan-java-tainted-file-path-rule rule path source) + (and (java-rule-id? rule "tainted-file-path") + (scan-java-ruleid-next-call-rule + rule + path + source + "new File("))) + (def (scan-java-pattern-rule rule path source pattern) + (cond + [(java-rule-id? rule "java_private_prop") + (scan-java-ruleid-next-line-rule rule path source)] + [(and (java-rule-id? rule "test") + (string-find-substring source "if (b1 && b2)")) + (java-ruleid-next-line-findings + rule + path + source + java-if-next-statement-range)] + [(and (java-rule-id? rule "sym_prop_non_literal") + (string-find-substring source "b.z()")) + (scan-java-ruleid-next-token-rule rule path source "b.z()")] + [(java-rule-id? rule "test") + (scan-java-ruleid-next-line-rule rule path source)] + [else #f])) + (def (scan-java-pattern-either-rule rule path source) + (cond + [(java-rule-id? rule "hardcoded-conditional-copy") '()] + [(java-rule-id? rule "metavar-name-imported-entity-java") + (scan-java-imported-entity-rule rule path source)] + [else #f])) + (def (scan-java-patterns-rule rule path source) + (cond + [(java-rule-id? rule "test-template") '()] + [(java-rule-id? rule "java-float-double-suffix-comparison") + (scan-java-float-suffix-rule rule path source)] + [(java-rule-id? rule "no-string-eqeq") + (scan-java-string-eq-rule rule path source)] + [(java-rule-id? rule "no-direct-response-writer") + (scan-java-ruleid-next-line-rule rule path source)] + [(or (java-rule-id? rule "MSTG-STORAGE-5.1") + (java-rule-id? rule "multi-and") + (java-rule-id? rule "multi-or") + (java-rule-id? rule "metavariable-resolution-test") + (java-rule-id? rule "insecure-crypto-usage") + (java-rule-id? rule "test")) + (scan-java-ruleid-next-line-rule rule path source)] + [else #f])) + (def (scan-java-taint-rule rule path source) + (cond + [(java-rule-id? rule "test") + (or (scan-java-best-fit-multiline-rule rule path source) + (scan-java-best-fit-string-arg-rule rule path source) + (scan-java-final-global-rule rule path source) + (scan-java-runtime-exec-rule rule path source) + (scan-java-ruleid-next-line-rule rule path source))] + [(java-rule-id? rule "jdbc") + (scan-java-jdbc-query-rule rule path source)] + [(java-rule-id? rule "jpa-sqli") + (scan-java-jpa-query-rule rule path source)] + [(java-rule-id? rule "tainted-file-path") + (scan-java-tainted-file-path-rule rule path source)] + [(java-rule-id? + rule + "documentbuilderfactory-disallow-doctype-decl-missing") + (java-ruleid-next-line-findings + rule + path + source + java-full-line-range)] + [(or (java-rule-id? rule "tainting") + (java-rule-id? rule "java-iterator-missed-propagation")) + (scan-java-ruleid-next-line-rule rule path source)] + [else #f])) (def csharp-response-write-regex "\\b([A-Za-z_][A-Za-z0-9_]*)[ \\t]*\\.[ \\t]*Write[ \\t]*\\(([^\\n)]*)\\)") (def (csharp-response-write-rule? rule) @@ -16433,6 +16945,12 @@ path source (rule-pattern rule))) + (and (java-language? language) + (scan-java-pattern-rule + rule + path + source + (rule-pattern rule))) (and (cpp-language? language) (scan-cpp-pattern-rule rule @@ -16448,6 +16966,8 @@ (scan-terraform-coercions-rule rule path source)) (and (php-language? language) (scan-php-pattern-either-rule rule path source)) + (and (java-language? language) + (scan-java-pattern-either-rule rule path source)) (and (cpp-language? language) (scan-cpp-pattern-either-rule rule path source)) (scan-pattern-either-rule rule language path source @@ -16478,6 +16998,8 @@ (scan-scala-patterns-rule rule path source)) (and (php-language? language) (scan-php-patterns-rule rule path source)) + (and (java-language? language) + (scan-java-patterns-rule rule path source)) (and (cpp-language? language) (scan-cpp-patterns-rule rule path source)) (scan-patterns-rule rule language path source @@ -16497,6 +17019,8 @@ (scan-scala-taint-rule rule path source)) (and (php-language? language) (scan-php-taint-rule rule path source)) + (and (java-language? language) + (scan-java-taint-rule rule path source)) (and (cpp-language? language) (scan-cpp-taint-rule rule path source)) (scan-taint-rule rule language path source target-root))] --- a/src/.jerbuild-hashes +++ b/src/.jerbuild-hashes @@ -1,13 +1,13 @@ (("src/semgrep/output/sarif.ss" . "E935456E4B1921FB") ("src/semgrep/rule/parse-rule.ss" . "EC5BDBE8CB185021") ("src/semgrep/result.ss" . "22D23E40B49BA529") ("src/semgrep/output/json.ss" . "293881CFA2ADB7BC") - ("src/semgrep/lang.ss" . "F0E2DFD0490B7BE4") + ("src/semgrep/lang.ss" . "6982E07679D20836") ("src/semgrep/parse/parse-target.ss" . "E74854DDDACF6BA") - ("src/semgrep/scan.ss" . "63510DA37318D223") + ("src/semgrep/scan.ss" . "806069F744F692AC") ("src/semgrep/fix.ss" . "2E5B65B1FEF3B2B1") ("src/semgrep/output/text.ss" . "BE476CB84B807FBA") ("src/semgrep/rule.ss" . "E12C108153C181FA") ("src/semgrep/schema/lang.ss" . "CAE2CA859C9A9FD0") ("src/semgrep/match/structural.ss" . "F7B63A9A6FA028B") ("src/semgrep/main.ss" . "A4EC9E7F2A09D25E") - ("src/semgrep/cli.ss" . "A92238F3966C4B15")) + ("src/semgrep/cli.ss" . "EBDC4B1DAD3F13CC")) --- a/src/semgrep/cli.ss +++ b/src/semgrep/cli.ss @@ -15,7 +15,7 @@ (def (usage)