Port detect_sequence_pair's kill-chain core into (jsecmon analytics)
ober
d26b217093b355b278e880ef773ac04710ca5ba8
--- a/README.md +++ b/README.md @@ -101,6 +101,7 @@ then crypto orchestration, then I/O / async / FFI (monitors, server, storage). | `lolbin::score` + `severity` | `typed/lolbin.ss` | ✅ full 25-pattern table + severity buckets; vectors pass (JSON-cmdline parse stays in untyped wrapper; diagnostic match list with caller) | | `analytics::compute_host_risks` | `typed/analytics.ss` | ✅ risk-score kernel (clamped weighted sum); vectors pass | | `analytics` grouping + `group_incidents` | `jsecmon/analytics.ss` | ✅ **untyped layer** — per-host accumulation/sort/top-N driving the risk-score kernel, plus incident dedup/collapse; secmon analytics vectors pass (`make analytics-check`) | +| `storage::detect_sequence_pair` (kill-chain core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure pairing primitive behind `detect_priv_escalation_chain`/`lateral_after_shell`/`persistence_after_access`/`log_cover`: given two event streams as `(host . ts-ms)` lists (the SQL `ORDER BY host,timestamp_ms` fetch is deferred I/O), pair each A with the **first** same-host B strictly later and within `window-ms` — at most one per A (Rust's inner `break`) — returning `((host …) (a-ts …) (b-ts …) (gap-seconds …))` for the caller to wrap as an Anomaly (`format_ts` is calendar-deferred). `gap-seconds` is integer ms/1000 (Rust i64 `/`). `make analytics-check` adds window-edge (≤ inclusive), strictly-later, cross-host, first-B-only, multi-A, and empty-stream cases. | | `storage::detect_lolbin_cmdline` + `detect_dga_domain` | `jsecmon/detect.ss` | ✅ **untyped layer** — the kernel-driven detection rules: score every process_start cmdline (lolbin) / dns_query (dga) into anomalies above threshold. `make detect-check` runs the full events→detect→analytics pipeline; all three scoring kernels fire. Per-pattern label lists + label-level DGA dedup pending (need kernels that return the match breakdown). | | `triage` classifiers | `typed/triage.ss` | ✅ pure predicates (transient-unit?, phantom-rootkit-race?); vectors pass | | `triage` engine (rules + dispatch) | `jsecmon/triage.ss` | ✅ **untyped layer** — all 18 false-positive rules + first-match engine, in secmon's exact RULES order, dispatch in ordinary Jerboa delegating byte/string classification to the typed kernels; 40 triage vectors pass (`make triage-check`), incl. the security-relevant negatives (non-sshd reading host keys, systemd impersonated from /tmp, unknown daemon reading passwd). | --- a/examples/analytics_check.ss +++ b/examples/analytics_check.ss @@ -62,6 +62,48 @@ (check "alice first_ms" (incident-first-ms alice) 1000) (check "alice last_ms" (incident-last-ms alice) 2000) +;; ── detect-sequence-pairs (storage detect_sequence_pair kill-chain core) ────── +;; A/B streams are (host . ts-ms) pairs, pre-sorted by host then ts (as the SQL +;; ORDER BY hands them back). window = 60_000 ms below unless noted. +(displayln "detect-sequence-pairs:") +(def (pair host ats bts gap) + (list (cons 'host host) (cons 'a-ts ats) (cons 'b-ts bts) (cons 'gap-seconds gap))) + +;; one A, one B on same host 5s later, within a 60s window -> one pairing +(check "basic pair within window" + (detect-sequence-pairs '(("h1" . 1000)) '(("h1" . 6000)) 60000) + (list (pair "h1" 1000 6000 5))) +;; B exactly at the window edge (a.ts + window) is included (<=) +(check "B at window edge included" + (detect-sequence-pairs '(("h1" . 1000)) '(("h1" . 61000)) 60000) + (list (pair "h1" 1000 61000 60))) +;; B one ms past the edge -> no pairing +(check "B past window excluded" + (detect-sequence-pairs '(("h1" . 1000)) '(("h1" . 61001)) 60000) '()) +;; B at the same ts as A is not "strictly later" -> excluded +(check "B not strictly later excluded" + (detect-sequence-pairs '(("h1" . 1000)) '(("h1" . 1000)) 60000) '()) +;; different host -> no pairing +(check "cross-host excluded" + (detect-sequence-pairs '(("h1" . 1000)) '(("h2" . 2000)) 60000) '()) +;; only the FIRST qualifying B is taken (Rust's break) — at most one per A +(check "first B only per A" + (detect-sequence-pairs '(("h1" . 1000)) '(("h1" . 2000) ("h1" . 3000)) 60000) + (list (pair "h1" 1000 2000 1))) +;; two As each pair with their own first later B on the same host +(check "two As pair independently" + (detect-sequence-pairs '(("h1" . 1000) ("h2" . 5000)) + '(("h1" . 1500) ("h2" . 5500)) 60000) + (list (pair "h1" 1000 1500 0) (pair "h2" 5000 5500 0))) ;; 500ms -> 0s (trunc) +;; an A with no qualifying B contributes nothing +(check "unpaired A dropped" + (detect-sequence-pairs '(("h1" . 1000) ("h1" . 100000)) + '(("h1" . 2000)) 60000) + (list (pair "h1" 1000 2000 1))) +;; empty streams +(check "empty A -> ()" (detect-sequence-pairs '() '(("h1" . 1)) 60000) '()) +(check "empty B -> ()" (detect-sequence-pairs '(("h1" . 1)) '() 60000) '()) + (newline) (if (= fails 0) (displayln "OK: untyped analytics matches secmon's vectors.") --- a/jsecmon/analytics.ss +++ b/jsecmon/analytics.ss @@ -20,7 +20,7 @@ host-risk-distinct-rules host-risk-chains host-risk-suspicious-cmdline host-risk-dga host-risk-rootkit-or-tamper host-risk-persistence host-risk-first-seen-ms host-risk-last-seen-ms - group-incidents + group-incidents detect-sequence-pairs incident? incident-rule incident-host incident-key incident-severity incident-attack incident-first-ms incident-last-ms incident-occurrences incident-sample) @@ -155,6 +155,32 @@ (else (pick details '("process_name" "username" "remote_host" "label" "query_name"))))) + ;; The pure core of secmon's storage detect_sequence_pair — the kill-chain + ;; pairing primitive behind detect_priv_escalation_chain / lateral_after_shell / + ;; persistence_after_access / log_cover. Given two event streams as lists of + ;; (host . ts-ms) pairs (the SQL fetch, ORDER BY host,timestamp_ms, is the + ;; deferred I/O), pair each A with the FIRST B on the same host strictly later + ;; than A and within window-ms — at most one pairing per A (Rust's inner `break`). + ;; Returns a list of ((host . s) (a-ts . n) (b-ts . n) (gap-seconds . n)); the + ;; caller builds the Anomaly record and formats the timestamps (format_ts is + ;; calendar-deferred). gap-seconds is integer ms/1000, matching Rust's i64 `/`. + (def (detect-sequence-pairs events-a events-b window-ms) + (append-map + (lambda (a) + (let ((ah (car a)) (ats (cdr a))) + (let loop ((bs events-b)) + (cond + ((null? bs) '()) + (else + (let ((bh (caar bs)) (bts (cdar bs))) + (if (and (string=? bh ah) (> bts ats) (<= bts (+ ats window-ms))) + (list (list (cons 'host ah) + (cons 'a-ts ats) + (cons 'b-ts bts) + (cons 'gap-seconds (quotient (- bts ats) 1000)))) + (loop (cdr bs))))))))) + events-a)) + (def (group-incidents detections) (let ((groups (make-hash-table))) (for-each