Port detect_frequency_spikes' per-host hourly-spike core into (jsecmon analytics)
ober
763b4adf88c93effcbb195aefde4eaa8b756274b
--- a/README.md +++ b/README.md @@ -103,6 +103,7 @@ then crypto orchestration, then I/O / async / FFI (monitors, server, storage). | `analytics` grouping + `group_incidents` | `jsecmon/analytics.ss` | ✅ **untyped layer** — per-host accumulation/sort/top-N driving the risk-score kernel, plus incident dedup/collapse; secmon analytics vectors pass (`make analytics-check`) | | `storage::detect_sequence_pair` (kill-chain core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure pairing primitive behind `detect_priv_escalation_chain`/`lateral_after_shell`/`persistence_after_access`/`log_cover`: given two event streams as `(host . ts-ms)` lists (the SQL `ORDER BY host,timestamp_ms` fetch is deferred I/O), pair each A with the **first** same-host B strictly later and within `window-ms` — at most one per A (Rust's inner `break`) — returning `((host …) (a-ts …) (b-ts …) (gap-seconds …))` for the caller to wrap as an Anomaly (`format_ts` is calendar-deferred). `gap-seconds` is integer ms/1000 (Rust i64 `/`). `make analytics-check` adds window-edge (≤ inclusive), strictly-later, cross-host, first-B-only, multi-A, and empty-stream cases. | | `storage::detect_severity_clusters` (cluster core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure sliding-window clustering: given critical/high events as `(host ts-ms event-type)` pre-sorted by host then ts (SQL fetch deferred), slide from each i, greedily take the same-host run with `ts ≤ ts_i + window-ms`, and when it holds `≥ min-count` (5) events emit a cluster then skip past it (Rust `i = j`), else advance one. Emits `((host …) (window-start …) (window-end …) (count …) (event-types …))` for the caller to format (`format_ts` deferred). `make analytics-check` adds exactly-5, only-4, past-edge, run-of-6-then-skip, host-boundary, two-clusters-after-skip, and empty cases. | +| `storage::detect_frequency_spikes` (spike core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure per-(host,event-type) hourly-spike test: given `(host event-type hour count)` rows (the `hourly_counts` GROUP BY aggregate is deferred I/O), sum count and tally hours per key, then emit any row whose key average `> 0` and whose `count` strictly exceeds `3×` that average. Returns `((host …) (event-type …) (hour …) (count …) (average …) (ratio …))` in input row order for the caller to wrap (`parse_hour_to_ms` is calendar-deferred); `average`/`ratio` are f64 like Rust's `total/hours` and `count/avg`. `make analytics-check` adds 3×-spike, exact-3×-excluded (strict `>`), flat, avg-0-guard, key-independence, two-group-order, and empty cases. | | `storage::detect_lolbin_cmdline` + `detect_dga_domain` | `jsecmon/detect.ss` | ✅ **untyped layer** — the kernel-driven detection rules: score every process_start cmdline (lolbin) / dns_query (dga) into anomalies above threshold. `make detect-check` runs the full events→detect→analytics pipeline; all three scoring kernels fire. Per-pattern label lists + label-level DGA dedup pending (need kernels that return the match breakdown). | | `triage` classifiers | `typed/triage.ss` | ✅ pure predicates (transient-unit?, phantom-rootkit-race?); vectors pass | | `triage` engine (rules + dispatch) | `jsecmon/triage.ss` | ✅ **untyped layer** — all 18 false-positive rules + first-match engine, in secmon's exact RULES order, dispatch in ordinary Jerboa delegating byte/string classification to the typed kernels; 40 triage vectors pass (`make triage-check`), incl. the security-relevant negatives (non-sshd reading host keys, systemd impersonated from /tmp, unknown daemon reading passwd). | --- a/examples/analytics_check.ss +++ b/examples/analytics_check.ss @@ -148,6 +148,54 @@ (cl "h1" 1000000 (+ 1000000 W) 5 '("f" "g" "h" "i" "j")))) (check "empty -> ()" (detect-severity-clusters '() W 5) '()) +;; ── detect-frequency-spikes (storage detect_frequency_spikes core) ──────────── +;; hourly rows are (host event-type hour count), the GROUP BY aggregate. A row +;; spikes when its (host,event-type) average > 0 and count strictly exceeds 3× +;; that average. Vectors use dyadic-exact averages so equal? on the f64s holds. +(displayln "detect-frequency-spikes:") +(def (row host type hour count) (list host type hour count)) +(def (spike host type hour count avg ratio) + (list (cons 'host host) (cons 'event-type type) (cons 'hour hour) + (cons 'count count) (cons 'average avg) (cons 'ratio ratio))) + +;; h1/a counts 2,2,2,26 -> avg 8.0; only 26 > 24 spikes, ratio 26/8 = 3.25 +(check "3x spike emitted, flat rows skipped" + (detect-frequency-spikes + (list (row "h1" "a" "h0" 2) (row "h1" "a" "h1" 2) + (row "h1" "a" "h2" 2) (row "h1" "a" "h3" 26))) + (list (spike "h1" "a" "h3" 26 8.0 3.25))) +;; counts 2,2,2,18 -> avg 6.0; 18 == 3×avg exactly, strict > excludes it +(check "exactly 3x not a spike (strict >)" + (detect-frequency-spikes + (list (row "h1" "a" "h0" 2) (row "h1" "a" "h1" 2) + (row "h1" "a" "h2" 2) (row "h1" "a" "h3" 18))) + '()) +(check "flat counts -> no spike" + (detect-frequency-spikes + (list (row "h1" "a" "h0" 5) (row "h1" "a" "h1" 5) + (row "h1" "a" "h2" 5) (row "h1" "a" "h3" 5))) + '()) +;; a lone zero-count row -> avg 0.0, guarded out (avg > 0.0 is false) +(check "avg 0 guarded -> no spike" + (detect-frequency-spikes (list (row "h9" "z" "h0" 0))) '()) +;; (host,event-type) keys are independent: h1/b's flat 5s don't dilute h1/a +(check "distinct (host,event-type) keys are independent" + (detect-frequency-spikes + (list (row "h1" "a" "h0" 2) (row "h1" "b" "h0" 5) + (row "h1" "a" "h1" 2) (row "h1" "b" "h1" 5) + (row "h1" "a" "h2" 2) (row "h1" "a" "h3" 26))) + (list (spike "h1" "a" "h3" 26 8.0 3.25))) +;; two groups each spike; results come back in input row order +(check "two groups spike, input order preserved" + (detect-frequency-spikes + (list (row "h1" "a" "h0" 2) (row "h1" "a" "h1" 2) + (row "h1" "a" "h2" 2) (row "h1" "a" "h3" 26) + (row "h2" "c" "h0" 0) (row "h2" "c" "h1" 0) + (row "h2" "c" "h2" 0) (row "h2" "c" "h3" 28))) + (list (spike "h1" "a" "h3" 26 8.0 3.25) + (spike "h2" "c" "h3" 28 7.0 4.0))) +(check "empty -> ()" (detect-frequency-spikes '()) '()) + (newline) (if (= fails 0) (displayln "OK: untyped analytics matches secmon's vectors.") --- a/jsecmon/analytics.ss +++ b/jsecmon/analytics.ss @@ -21,6 +21,7 @@ host-risk-dga host-risk-rootkit-or-tamper host-risk-persistence host-risk-first-seen-ms host-risk-last-seen-ms group-incidents detect-sequence-pairs detect-severity-clusters + detect-frequency-spikes incident? incident-rule incident-host incident-key incident-severity incident-attack incident-first-ms incident-last-ms incident-occurrences incident-sample) @@ -214,6 +215,39 @@ acc)) (loop (+ i 1) acc)))))))))) + ;; The pure core of secmon's storage detect_frequency_spikes. `hourly` is a + ;; list of (host event-type hour count) rows — the SQL `hourly_counts` + ;; GROUP BY host,event_type,hour aggregate is the deferred I/O. First sum + ;; count and tally hours per (host,event-type); then for each row whose + ;; (host,event-type) average is > 0 and whose count strictly exceeds 3× that + ;; average, emit a spike. Rows are scanned in input order, so spikes come back + ;; in the same order Rust pushes them. -> list of ((host . s) (event-type . s) + ;; (hour . s) (count . n) (average . f) (ratio . f)); the caller builds the + ;; Anomaly and parse_hour_to_ms's the hour (calendar-deferred). average/ratio + ;; are f64 like Rust's `total/hours` and `count/avg`. + (def (detect-frequency-spikes hourly) + (let ((sums (make-hash-table))) + (for-each + (lambda (row) + (let* ((key (str (car row) "\x1f;" (cadr row))) + (cur (or (hash-get sums key) (cons 0 0)))) + (hash-put! sums key + (cons (+ (car cur) (cadddr row)) (+ (cdr cur) 1))))) + hourly) + (filter-map + (lambda (row) + (let* ((host (car row)) (type (cadr row)) + (hour (caddr row)) (count (cadddr row)) + (acc (hash-get sums (str host "\x1f;" type))) + (avg (/ (exact->inexact (car acc)) (cdr acc)))) + (if (and (> avg 0.0) (> (exact->inexact count) (* avg 3.0))) + (list (cons 'host host) (cons 'event-type type) + (cons 'hour hour) (cons 'count count) + (cons 'average avg) + (cons 'ratio (/ (exact->inexact count) avg))) + #f))) + hourly))) + (def (group-incidents detections) (let ((groups (make-hash-table))) (for-each