Port detect_severity_clusters' sliding-window core into (jsecmon analytics)
ober
7ce36c7f8184d4d5e52aaeaa58ac655f721d961b
--- a/README.md +++ b/README.md @@ -102,6 +102,7 @@ then crypto orchestration, then I/O / async / FFI (monitors, server, storage). | `analytics::compute_host_risks` | `typed/analytics.ss` | ✅ risk-score kernel (clamped weighted sum); vectors pass | | `analytics` grouping + `group_incidents` | `jsecmon/analytics.ss` | ✅ **untyped layer** — per-host accumulation/sort/top-N driving the risk-score kernel, plus incident dedup/collapse; secmon analytics vectors pass (`make analytics-check`) | | `storage::detect_sequence_pair` (kill-chain core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure pairing primitive behind `detect_priv_escalation_chain`/`lateral_after_shell`/`persistence_after_access`/`log_cover`: given two event streams as `(host . ts-ms)` lists (the SQL `ORDER BY host,timestamp_ms` fetch is deferred I/O), pair each A with the **first** same-host B strictly later and within `window-ms` — at most one per A (Rust's inner `break`) — returning `((host …) (a-ts …) (b-ts …) (gap-seconds …))` for the caller to wrap as an Anomaly (`format_ts` is calendar-deferred). `gap-seconds` is integer ms/1000 (Rust i64 `/`). `make analytics-check` adds window-edge (≤ inclusive), strictly-later, cross-host, first-B-only, multi-A, and empty-stream cases. | +| `storage::detect_severity_clusters` (cluster core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure sliding-window clustering: given critical/high events as `(host ts-ms event-type)` pre-sorted by host then ts (SQL fetch deferred), slide from each i, greedily take the same-host run with `ts ≤ ts_i + window-ms`, and when it holds `≥ min-count` (5) events emit a cluster then skip past it (Rust `i = j`), else advance one. Emits `((host …) (window-start …) (window-end …) (count …) (event-types …))` for the caller to format (`format_ts` deferred). `make analytics-check` adds exactly-5, only-4, past-edge, run-of-6-then-skip, host-boundary, two-clusters-after-skip, and empty cases. | | `storage::detect_lolbin_cmdline` + `detect_dga_domain` | `jsecmon/detect.ss` | ✅ **untyped layer** — the kernel-driven detection rules: score every process_start cmdline (lolbin) / dns_query (dga) into anomalies above threshold. `make detect-check` runs the full events→detect→analytics pipeline; all three scoring kernels fire. Per-pattern label lists + label-level DGA dedup pending (need kernels that return the match breakdown). | | `triage` classifiers | `typed/triage.ss` | ✅ pure predicates (transient-unit?, phantom-rootkit-race?); vectors pass | | `triage` engine (rules + dispatch) | `jsecmon/triage.ss` | ✅ **untyped layer** — all 18 false-positive rules + first-match engine, in secmon's exact RULES order, dispatch in ordinary Jerboa delegating byte/string classification to the typed kernels; 40 triage vectors pass (`make triage-check`), incl. the security-relevant negatives (non-sshd reading host keys, systemd impersonated from /tmp, unknown daemon reading passwd). | --- a/examples/analytics_check.ss +++ b/examples/analytics_check.ss @@ -104,6 +104,50 @@ (check "empty A -> ()" (detect-sequence-pairs '() '(("h1" . 1)) 60000) '()) (check "empty B -> ()" (detect-sequence-pairs '(("h1" . 1)) '() 60000) '()) +;; ── detect-severity-clusters (storage detect_severity_clusters core) ────────── +;; events are (host ts-ms event-type), pre-sorted by host then ts. W = 5min window. +(displayln "detect-severity-clusters:") +(def W 300000) +(def (ev host ts type) (list host ts type)) +(def (cl host ws we cnt types) + (list (cons 'host host) (cons 'window-start ws) (cons 'window-end we) + (cons 'count cnt) (cons 'event-types types))) + +(check "exactly 5 in window -> one cluster" + (detect-severity-clusters + (list (ev "h1" 0 "a") (ev "h1" 1000 "b") (ev "h1" 2000 "c") + (ev "h1" 3000 "d") (ev "h1" 4000 "e")) W 5) + (list (cl "h1" 0 W 5 '("a" "b" "c" "d" "e")))) +(check "only 4 -> no cluster" + (detect-severity-clusters + (list (ev "h1" 0 "a") (ev "h1" 1000 "b") (ev "h1" 2000 "c") (ev "h1" 3000 "d")) W 5) + '()) +(check "5th past window edge -> no cluster" + (detect-severity-clusters + (list (ev "h1" 0 "a") (ev "h1" 1000 "b") (ev "h1" 2000 "c") + (ev "h1" 3000 "d") (ev "h1" 400000 "e")) W 5) + '()) +(check "whole run of 6 collected then skipped" + (detect-severity-clusters + (list (ev "h1" 0 "a") (ev "h1" 1000 "b") (ev "h1" 2000 "c") + (ev "h1" 3000 "d") (ev "h1" 4000 "e") (ev "h1" 5000 "f")) W 5) + (list (cl "h1" 0 W 6 '("a" "b" "c" "d" "e" "f")))) +(check "host boundary ends the run" + (detect-severity-clusters + (list (ev "h1" 0 "a") (ev "h1" 1000 "b") (ev "h1" 2000 "c") + (ev "h1" 3000 "d") (ev "h1" 4000 "e") + (ev "h2" 5000 "x") (ev "h2" 6000 "y")) W 5) + (list (cl "h1" 0 W 5 '("a" "b" "c" "d" "e")))) +(check "two clusters after i=j skip" + (detect-severity-clusters + (list (ev "h1" 0 "a") (ev "h1" 1000 "b") (ev "h1" 2000 "c") + (ev "h1" 3000 "d") (ev "h1" 4000 "e") + (ev "h1" 1000000 "f") (ev "h1" 1001000 "g") (ev "h1" 1002000 "h") + (ev "h1" 1003000 "i") (ev "h1" 1004000 "j")) W 5) + (list (cl "h1" 0 W 5 '("a" "b" "c" "d" "e")) + (cl "h1" 1000000 (+ 1000000 W) 5 '("f" "g" "h" "i" "j")))) +(check "empty -> ()" (detect-severity-clusters '() W 5) '()) + (newline) (if (= fails 0) (displayln "OK: untyped analytics matches secmon's vectors.") --- a/jsecmon/analytics.ss +++ b/jsecmon/analytics.ss @@ -20,7 +20,7 @@ host-risk-distinct-rules host-risk-chains host-risk-suspicious-cmdline host-risk-dga host-risk-rootkit-or-tamper host-risk-persistence host-risk-first-seen-ms host-risk-last-seen-ms - group-incidents detect-sequence-pairs + group-incidents detect-sequence-pairs detect-severity-clusters incident? incident-rule incident-host incident-key incident-severity incident-attack incident-first-ms incident-last-ms incident-occurrences incident-sample) @@ -181,6 +181,39 @@ (loop (cdr bs))))))))) events-a)) + ;; The pure core of secmon's storage detect_severity_clusters. `events` is a + ;; list of (host ts-ms event-type), pre-sorted by host then ts (the SQL fetch of + ;; critical/high events is the deferred I/O). Slide from each i: greedily take + ;; the run of SAME-host events whose ts <= ts_i + window-ms; if that run holds + ;; >= min-count events, emit a cluster and skip past it (faithful to Rust's + ;; `i = j`), otherwise advance by one (`i += 1`). -> list of ((host . s) + ;; (window-start . n) (window-end . n) (count . n) (event-types . (s …))); the + ;; caller formats window-start/end (format_ts is calendar-deferred). + (def (detect-severity-clusters events window-ms min-count) + (let ((vec (list->vector events)) + (n (length events))) + (let loop ((i 0) (acc '())) + (if (>= i n) + (reverse acc) + (let* ((ev (vector-ref vec i)) + (host (car ev)) + (ws (cadr ev)) + (we (+ ws window-ms))) + (let scan ((j i) (types '())) + (if (and (< j n) + (string=? (car (vector-ref vec j)) host) + (<= (cadr (vector-ref vec j)) we)) + (scan (+ j 1) (cons (caddr (vector-ref vec j)) types)) + (let ((cnt (- j i))) + (if (>= cnt min-count) + (loop j (cons (list (cons 'host host) + (cons 'window-start ws) + (cons 'window-end we) + (cons 'count cnt) + (cons 'event-types (reverse types))) + acc)) + (loop (+ i 1) acc)))))))))) + (def (group-incidents detections) (let ((groups (make-hash-table))) (for-each