Port detect_kill_chain core + detect_off_hours predicate into (jsecmon analytics)
ober
5ae09008ba630d97411621a928516e5c40eebca4
--- a/README.md +++ b/README.md @@ -104,6 +104,8 @@ then crypto orchestration, then I/O / async / FFI (monitors, server, storage). | `storage::detect_sequence_pair` (kill-chain core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure pairing primitive behind `detect_priv_escalation_chain`/`lateral_after_shell`/`persistence_after_access`/`log_cover`: given two event streams as `(host . ts-ms)` lists (the SQL `ORDER BY host,timestamp_ms` fetch is deferred I/O), pair each A with the **first** same-host B strictly later and within `window-ms` — at most one per A (Rust's inner `break`) — returning `((host …) (a-ts …) (b-ts …) (gap-seconds …))` for the caller to wrap as an Anomaly (`format_ts` is calendar-deferred). `gap-seconds` is integer ms/1000 (Rust i64 `/`). `make analytics-check` adds window-edge (≤ inclusive), strictly-later, cross-host, first-B-only, multi-A, and empty-stream cases. | | `storage::detect_severity_clusters` (cluster core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure sliding-window clustering: given critical/high events as `(host ts-ms event-type)` pre-sorted by host then ts (SQL fetch deferred), slide from each i, greedily take the same-host run with `ts ≤ ts_i + window-ms`, and when it holds `≥ min-count` (5) events emit a cluster then skip past it (Rust `i = j`), else advance one. Emits `((host …) (window-start …) (window-end …) (count …) (event-types …))` for the caller to format (`format_ts` deferred). `make analytics-check` adds exactly-5, only-4, past-edge, run-of-6-then-skip, host-boundary, two-clusters-after-skip, and empty cases. | | `storage::detect_frequency_spikes` (spike core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure per-(host,event-type) hourly-spike test: given `(host event-type hour count)` rows (the `hourly_counts` GROUP BY aggregate is deferred I/O), sum count and tally hours per key, then emit any row whose key average `> 0` and whose `count` strictly exceeds `3×` that average. Returns `((host …) (event-type …) (hour …) (count …) (average …) (ratio …))` in input row order for the caller to wrap (`parse_hour_to_ms` is calendar-deferred); `average`/`ratio` are f64 like Rust's `total/hours` and `count/avg`. `make analytics-check` adds 3×-spike, exact-3×-excluded (strict `>`), flat, avg-0-guard, key-independence, two-group-order, and empty cases. | +| `storage::detect_kill_chain` (chain core) | `jsecmon/analytics.ss` | ✅ **untyped layer** — the pure multi-phase kill-chain detector: given `(host ts-ms event-type)` rows pre-sorted by host then ts (SQL fetch deferred), slide from each i over the same-host run with `ts ≤ ts_i + window-ms`, map each type to an ATT&CK-ish phase via `event-type->attack-phase` (also exported; unmapped types skipped), and when the **distinct** phases reach `min-phases` (3) emit a chain then skip past it (Rust `i = j`), else advance one. Emits `((host …) (window-start …) (window-end …) (phases …) (event-types …))`; Rust collects phases from an unordered `HashSet`, so `phases` is canonicalized to first-seen order (treat as a set) while `event-types` keeps phase-mapped types in order. `make analytics-check` adds the classifier table, three-phases, two-distinct-only, unmapped-skip, host-boundary, window-edge, past-edge, two-chains-after-skip, and empty cases. | +| `storage::detect_off_hours` (predicate) | `jsecmon/analytics.ss` | ✅ **untyped layer** — `off-hours?`, the decision rule factored out of the SQL `WHERE`: a critical/high event is off-hours on a weekend or outside 08:00–18:00 UTC (`weekday` = strftime `%w` 0=Sun…6=Sat, `hour` = `%H` 0–23). The timestamp→(weekday,hour) decomposition is calendar-deferred. `make analytics-check` adds weekend, midday, and the 08:00/17:00/18:00 boundaries. | | `storage::detect_lolbin_cmdline` + `detect_dga_domain` | `jsecmon/detect.ss` | ✅ **untyped layer** — the kernel-driven detection rules: score every process_start cmdline (lolbin) / dns_query (dga) into anomalies above threshold. `make detect-check` runs the full events→detect→analytics pipeline; all three scoring kernels fire. Per-pattern label lists + label-level DGA dedup pending (need kernels that return the match breakdown). | | `triage` classifiers | `typed/triage.ss` | ✅ pure predicates (transient-unit?, phantom-rootkit-race?); vectors pass | | `triage` engine (rules + dispatch) | `jsecmon/triage.ss` | ✅ **untyped layer** — all 18 false-positive rules + first-match engine, in secmon's exact RULES order, dispatch in ordinary Jerboa delegating byte/string classification to the typed kernels; 40 triage vectors pass (`make triage-check`), incl. the security-relevant negatives (non-sshd reading host keys, systemd impersonated from /tmp, unknown daemon reading passwd). | --- a/examples/analytics_check.ss +++ b/examples/analytics_check.ss @@ -196,6 +196,88 @@ (spike "h2" "c" "h3" 28 7.0 4.0))) (check "empty -> ()" (detect-frequency-spikes '()) '()) +;; ── detect-kill-chains (storage detect_kill_chain core) ─────────────────────── +;; events are (host ts-ms event-type), pre-sorted by host then ts. H = 1h window, +;; min 3 DISTINCT attack phases. phases is first-seen order (Rust HashSet is +;; unordered → treat as a set); event-types keeps phase-mapped types in order. +(displayln "detect-kill-chains:") +(def H 3600000) +(def (kc host ws we phases types) + (list (cons 'host host) (cons 'window-start ws) (cons 'window-end we) + (cons 'phases phases) (cons 'event-types types))) + +(check "phase classifier table" + (map event-type->attack-phase + '("dns_query" "reverse_shell" "persistence_event" "lateral_movement" + "privilege_change" "log_tampering" "sensitive_file_access" "nope")) + '("recon" "initial_access" "persistence" "lateral_movement" + "privilege_escalation" "defense_evasion" "exfiltration" #f)) +;; three distinct phases within the hour -> one chain +(check "three phases -> chain" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "reverse_shell") + (ev "h1" 2000 "persistence_event")) H 3) + (list (kc "h1" 0 H '("recon" "initial_access" "persistence") + '("dns_query" "reverse_shell" "persistence_event")))) +;; two distinct phases (recon twice) -> none +(check "two distinct phases -> none" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "network_connection") + (ev "h1" 2000 "reverse_shell")) H 3) + '()) +;; an unmapped type is scanned over but adds no phase; the other 3 still chain +(check "unmapped type skipped, still chains" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "unknown_thing") + (ev "h1" 2000 "reverse_shell") (ev "h1" 3000 "log_tampering")) H 3) + (list (kc "h1" 0 H '("recon" "initial_access" "defense_evasion") + '("dns_query" "reverse_shell" "log_tampering")))) +;; host boundary stops the run before the 3rd phase joins +(check "host boundary ends run -> none" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "reverse_shell") + (ev "h2" 2000 "persistence_event")) H 3) + '()) +;; 3rd phase exactly at the window edge (ws+H) is included (<=) +(check "phase at window edge included" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "reverse_shell") + (ev "h1" 3600000 "persistence_event")) H 3) + (list (kc "h1" 0 H '("recon" "initial_access" "persistence") + '("dns_query" "reverse_shell" "persistence_event")))) +;; 3rd phase one ms past the first event's window -> no window holds 3 phases +(check "third past window -> none" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "reverse_shell") + (ev "h1" 3600001 "persistence_event")) H 3) + '()) +;; emit then i=j skip; a second, far-later chain emits independently +(check "two chains after i=j skip" + (detect-kill-chains + (list (ev "h1" 0 "dns_query") (ev "h1" 1000 "reverse_shell") + (ev "h1" 2000 "persistence_event") + (ev "h1" 10000000 "dns_query") (ev "h1" 10001000 "lateral_movement") + (ev "h1" 10002000 "log_tampering")) H 3) + (list (kc "h1" 0 H '("recon" "initial_access" "persistence") + '("dns_query" "reverse_shell" "persistence_event")) + (kc "h1" 10000000 (+ 10000000 H) + '("recon" "lateral_movement" "defense_evasion") + '("dns_query" "lateral_movement" "log_tampering")))) +(check "empty -> ()" (detect-kill-chains '() H 3) '()) + +;; ── off-hours? (storage detect_off_hours WHERE-clause predicate) ────────────── +;; weekday 0=Sun … 6=Sat (strftime %w), hour 0-23 UTC. Off-hours = weekend, or +;; before 08:00, or at/after 18:00. +(displayln "off-hours?:") +(check "Sunday is off-hours" (off-hours? 0 12) #t) +(check "Saturday is off-hours" (off-hours? 6 12) #t) +(check "Wed midday is on-hours" (off-hours? 3 12) #f) +(check "Wed 07:00 is off-hours" (off-hours? 3 7) #t) +(check "Wed 08:00 is on-hours" (off-hours? 3 8) #f) +(check "Wed 17:00 is on-hours" (off-hours? 3 17) #f) +(check "Wed 18:00 is off-hours" (off-hours? 3 18) #t) +(check "Mon 23:00 is off-hours" (off-hours? 1 23) #t) + (newline) (if (= fails 0) (displayln "OK: untyped analytics matches secmon's vectors.") --- a/jsecmon/analytics.ss +++ b/jsecmon/analytics.ss @@ -21,7 +21,8 @@ host-risk-dga host-risk-rootkit-or-tamper host-risk-persistence host-risk-first-seen-ms host-risk-last-seen-ms group-incidents detect-sequence-pairs detect-severity-clusters - detect-frequency-spikes + detect-frequency-spikes detect-kill-chains event-type->attack-phase + off-hours? incident? incident-rule incident-host incident-key incident-severity incident-attack incident-first-ms incident-last-ms incident-occurrences incident-sample) @@ -248,6 +249,65 @@ #f))) hourly))) + ;; secmon detect_kill_chain's nested attack_phase: event type -> ATT&CK-ish + ;; phase string, or #f for types that aren't part of a chain. Verbatim table. + (def (event-type->attack-phase ty) + (cond ((in? ty "dns_query" "network_connection") "recon") + ((in? ty "suspicious_exec" "reverse_shell" "webshell") "initial_access") + ((in? ty "persistence_event" "scheduled_task_change") "persistence") + ((string=? ty "lateral_movement") "lateral_movement") + ((in? ty "privilege_escalation" "privilege_change") "privilege_escalation") + ((string=? ty "log_tampering") "defense_evasion") + ((in? ty "sensitive_file_access" "suspicious_connection") "exfiltration") + (else #f))) + + ;; The pure core of secmon's storage detect_kill_chain. `events` is a list of + ;; (host ts-ms event-type) pre-sorted by host then ts (SQL fetch deferred). + ;; Slide from each i: scan the same-host run with ts <= ts_i + window-ms, + ;; mapping each type to an attack phase (skipping unmapped ones), and when the + ;; DISTINCT phases reach min-phases (3) emit a chain and skip past it (Rust + ;; `i = j`), else advance one. Emits ((host …) (window-start …) (window-end …) + ;; (phases (s …)) (event-types (s …))). Rust collects phases from a HashSet + ;; (unordered) — we canonicalize to first-seen order, so treat `phases` as a + ;; set; `event-types` keeps every phase-mapped type in order (dups allowed, + ;; like Rust's Vec). The caller formats the timestamps (format_ts deferred). + (def (detect-kill-chains events window-ms min-phases) + (let ((vec (list->vector events)) + (n (length events))) + (let loop ((i 0) (acc '())) + (if (>= i n) + (reverse acc) + (let* ((ev (vector-ref vec i)) + (host (car ev)) + (ws (cadr ev)) + (we (+ ws window-ms))) + (let scan ((j i) (phases '()) (types '())) + (if (and (< j n) + (string=? (car (vector-ref vec j)) host) + (<= (cadr (vector-ref vec j)) we)) + (let ((ph (event-type->attack-phase (caddr (vector-ref vec j))))) + (if ph + (scan (+ j 1) + (if (member ph phases) phases (cons ph phases)) + (cons (caddr (vector-ref vec j)) types)) + (scan (+ j 1) phases types))) + (if (>= (length phases) min-phases) + (loop j (cons (list (cons 'host host) + (cons 'window-start ws) + (cons 'window-end we) + (cons 'phases (reverse phases)) + (cons 'event-types (reverse types))) + acc)) + (loop (+ i 1) acc))))))))) + + ;; secmon detect_off_hours' decision rule, factored out of the SQL: a critical/ + ;; high event is "off-hours" when it lands on a weekend or outside 08:00–18:00. + ;; weekday is SQLite strftime('%w') in UTC (0=Sunday … 6=Saturday); hour is + ;; strftime('%H') in UTC (0–23). The timestamp→(weekday,hour) decomposition is + ;; calendar-deferred — this is just the predicate the WHERE clause encodes. + (def (off-hours? weekday hour) + (or (= weekday 0) (= weekday 6) (< hour 8) (>= hour 18))) + (def (group-incidents detections) (let ((groups (make-hash-table))) (for-each