correlate Rule 13: port detect_impossible_travel (geo-correlation)
ober
fe3fb3a8f815fbec02afec6c9aacc177425e68eb
--- a/README.md +++ b/README.md @@ -128,7 +128,7 @@ then crypto orchestration, then I/O / async / FFI (monitors, server, storage). | `geoip` (CSV GeoIP/ASN, IPv4+IPv6, binary-search range lookup, is_private) | `jsecmon/geoip.ss` | ✅ **untyped layer** — full port of secmon's `src/geoip.rs`: parse `start,end,country,asn,name` CSV rows (v4 + one-`::`-expanding v6), sort-by-start + binary-search lookup, RFC1918/loopback/link-local/multicast/ULA/CGNAT → synthetic `PRIVATE`. Pure parsing + integer math + file read, so untyped. `make geoip-check` runs secmon's geoip vectors. | | `storage` impossible_travel | `jsecmon/threats.ss` | ✅ **untyped layer** — geoip-gated (reads `SECMON_GEOIP_CSV`): pair a user's consecutive successful `auth_event`s, fire `high` when the two source IPs resolve to different countries within `SECMON_TRAVEL_GAP_MIN` (default 30). Private IPs are dropped before pairing. `make geoip-check` proves the fire + the gap/same-country/private/cross-user negatives. | | `buffer::ring` (StoredEvent ring buffer) | `jsecmon/buffer.ss` | ✅ **untyped layer** — port of secmon's `src/buffer/ring.rs`: the agent's bounded in-memory event ring. FIFO list + monotonic seq numbering, priority eviction (`event_severity_u8` table, drop lowest-severity oldest-first, oldest-critical last), seq/time-range polling, FIFO delivery-ack (`clear_before`), and the little-endian header codec (`seq u64 ∥ ts i64 ∥ sev u8 ∥ payload`). Pure mechanics, so untyped — the one security step, ECIES payload encryption, is FFI-deferred: the caller hands `buffer-store!` opaque ciphertext bytes. `make buffer-check` reproduces secmon's three ring tests (store/seq, priority eviction, FIFO-oldest) + codec round-trip. | -| `storage` correlation rules (`detect_*`) | `jsecmon/correlate.ss` | ✅ **untyped layer** — secmon's anomaly detectors are "run a SQL query, then map result rows → Anomaly". The SQL fetch (with its `json_extract` / `GROUP BY … HAVING`) is the deferred storage I/O; the pure, portable part is the correlation algorithm over the fetched rows + the Anomaly description, reimplemented here over pre-shaped row lists (which is exactly what `test_detect_*` exercise). **Batch 1** = the GROUP-BY rules: `detect_brute_force` (≥5 failed-auth per user / 10-min bucket), `detect_credential_stuffing` (≥5 **distinct** users per remote / 10-min), `detect_dns_tunnel` (≥50 queries per process / 5-min), `detect_suspicious_cron` (per-row: `scheduled_task_change` by a non-root user, missing user → `""` ≠ root → kept). `timestamp_ms / N` is integer division → `quotient`, bucket anomaly ts is `bucket*N`; COUNT(*) vs COUNT(DISTINCT) become first-seen-order tallies (treat the set, not order, as significant). Each detector returns "anomaly seeds" (an Anomaly minus the calendar `format_ts` fields, which the caller adds, and minus the `attack` tags from `anomaly-rule-attack`). **Batch 2** = the sliding-window / sequence-pair rules over caller-sorted rows: `detect_recon_port_scan` (≥10 distinct remote ports / 5-min, same process) and `detect_data_exfil` (≥20 outbound connections / 5-min) share a `slide` helper reproducing secmon's `while i<len { expand j; if hit { i=j } else { i+=1 } }`; `detect_sequence_pair` (event A then first B on the same host with `a.ts < b.ts ≤ a.ts+window`, one match per A) backs the three chain rules `detect_priv_escalation_chain` (auth-success→priv-esc / 5-min, caller pre-filters A to `success=1`), `detect_persistence_after_access` (reverse_shell **or** webshell→persistence / 1h — two arms appended), `detect_lateral_after_shell` (reverse_shell→lateral / 1h); and `detect_log_cover` (any critical event→log_tampering / 1h, one per critical) carries the event type into the description. `gap_seconds` is `(quotient (- b a) 1000)`. `make correlate-check` reproduces all ten `test_detect_*` (incl. `_below_threshold` and `_outside_window`) + the threshold / bucket-boundary / distinct-dedup / window-edge / cross-host / B-before-A edges (60 cases). | +| `storage` correlation rules (`detect_*`) | `jsecmon/correlate.ss` | ✅ **untyped layer** — secmon's anomaly detectors are "run a SQL query, then map result rows → Anomaly". The SQL fetch (with its `json_extract` / `GROUP BY … HAVING`) is the deferred storage I/O; the pure, portable part is the correlation algorithm over the fetched rows + the Anomaly description, reimplemented here over pre-shaped row lists (which is exactly what `test_detect_*` exercise). **Batch 1** = the GROUP-BY rules: `detect_brute_force` (≥5 failed-auth per user / 10-min bucket), `detect_credential_stuffing` (≥5 **distinct** users per remote / 10-min), `detect_dns_tunnel` (≥50 queries per process / 5-min), `detect_suspicious_cron` (per-row: `scheduled_task_change` by a non-root user, missing user → `""` ≠ root → kept). `timestamp_ms / N` is integer division → `quotient`, bucket anomaly ts is `bucket*N`; COUNT(*) vs COUNT(DISTINCT) become first-seen-order tallies (treat the set, not order, as significant). Each detector returns "anomaly seeds" (an Anomaly minus the calendar `format_ts` fields, which the caller adds, and minus the `attack` tags from `anomaly-rule-attack`). **Batch 2** = the sliding-window / sequence-pair rules over caller-sorted rows: `detect_recon_port_scan` (≥10 distinct remote ports / 5-min, same process) and `detect_data_exfil` (≥20 outbound connections / 5-min) share a `slide` helper reproducing secmon's `while i<len { expand j; if hit { i=j } else { i+=1 } }`; `detect_sequence_pair` (event A then first B on the same host with `a.ts < b.ts ≤ a.ts+window`, one match per A) backs the three chain rules `detect_priv_escalation_chain` (auth-success→priv-esc / 5-min, caller pre-filters A to `success=1`), `detect_persistence_after_access` (reverse_shell **or** webshell→persistence / 1h — two arms appended), `detect_lateral_after_shell` (reverse_shell→lateral / 1h); and `detect_log_cover` (any critical event→log_tampering / 1h, one per critical) carries the event type into the description. `gap_seconds` is `(quotient (- b a) 1000)`. **Batch 3** = the geo-correlation rule `detect_impossible_travel` (Rule 13): over auth points the caller has already geo-resolved (the `GeoDb` lookup is the deferred I/O, like the SQL fetch) and sorted by `(user, ts)`, it walks every adjacent pair (Rust `i += 1` always) and emits a `high` seed when the same user logs in from two **different** countries within `min-gap-ms` (default 30 min = `SECMON_TRAVEL_GAP_MIN`, an optional 2nd arg). The `PRIVATE`/empty-country drop (Rust's `filter_map` before pairing) is pure, so it stays here; `gap_minutes` = `(quotient ms 60000)`; the seed carries raw `first-seen-ms`/`second-seen-ms` for the caller's `format_ts`. `make correlate-check` reproduces all ten `test_detect_*` (incl. `_below_threshold` and `_outside_window`) + the threshold / bucket-boundary / distinct-dedup / window-edge / cross-host / B-before-A edges, plus the impossible-travel country/gap-edge/cross-user/PRIVATE-drop/override edges (76 cases). | | `server::protocol` (length-prefixed framing) | `jsecmon/frame.ss` | ✅ **untyped layer** — the wire framing the collector↔agent transport wraps every `ProtocolMessage` in: `frame_encode` prepends a 4-byte little-endian u32 length to the payload (`to_bytes`'s framing half), `frame_read_length` reads that header as an LE u32 (`read_length`), and `frame_decode` reproduces `from_bytes`'s two guards — `< 4` bytes → `data too short`, `< 4 + declared len` → `incomplete message` — then slices out `data[4 .. 4+len]`. Pure byte mechanics, so untyped; the bincode (de)serialization of the message *body* is the Rust-specific deferred piece (the caller (de)serializes the payload `frame_decode` hands back). LE u32 verified to round-trip `0x12345678` and `0xFFFFFFFF`. `make frame-check` pins the byte layout, the encode∘decode round-trip, both error guards, and the empty/zero-length-payload edge. | | `monitor::events::SuspiciousPatterns` (process-spawn classifier) | `jsecmon/suspicious.ss` | ✅ **untyped layer** — `check_suspicious(process, parent)`: shell-from-service, attack-tool-from-service (name exact-match or exe suffix), reverse-shell command-line patterns, and crypto-miner name/cmdline patterns, in secmon's order, returning the same reason string. Pure string classification like triage. Pins two corners the Rust depends on: a missing parent short-circuits to "clean" before any check, and `str::contains` is a *literal* substring test (so `python -c.*socket` is literal, not a regex). `make suspicious-check` reproduces secmon's two events.rs tests + the other three signals + both corners. | | `monitor::network::NetworkMonitor` (connection classifier) | `jsecmon/netconn.ss` | ✅ **untyped layer** — `check_suspicious(port, addr, process)`: known reverse-shell/C2/l33t port, ephemeral port (49152..65535) that is a round multiple of 1000, and a web-server process (nginx/apache/httpd/php-fpm) connecting to a non-private address, in secmon's order with the same reason string. Pure metadata classification. secmon hides the web-server names with `obfstr!` (same scheme as `typed/obfuscate.ss`); they decode to these plaintext literals at runtime. Pins the faithfulness quirk that the "private" prefix set is literal `{127. 10. 192.168. 172.}`, so `172.` matches all of 172.x, not just RFC1918 172.16/12. `make netconn-check` reproduces secmon's two network.rs tests + the full bad-port list + the high-port and web-server rules with private-address negatives. | --- a/examples/correlate_check.ss +++ b/examples/correlate_check.ss @@ -203,7 +203,64 @@ (check "revshell then lateral -> 1" (length la) 1) (check "rule" (rule (car la)) "lateral_after_shell") +;; ── impossible travel: same user, two countries, within min-gap (no #[test]) ── +;; pts = (host ts user ip country asn), pre-sorted by (user, ts), geo-resolved. +(displayln "detect-impossible-travel:") +(def it + (detect-impossible-travel + (list (list "h1" 1000 "alice" "1.2.3.4" "US" 100) + (list "h2" 601000 "alice" "5.6.7.8" "RU" 200)))) +(check "US then RU 10min -> 1 anomaly" (length it) 1) +(check "rule" (rule (car it)) "impossible_travel") +(check "severity" (sev (car it)) "high") +(check "ts = second login" (ts (car it)) 601000) +(check "host = second host" (cdr (assq 'host (car it))) "h2") +(check "description" + (desc (car it)) + "user 'alice' authenticated from US (1.2.3.4) and RU (5.6.7.8) within 10 min") +(check "gap-minutes detail" (dval (car it) 'gap-minutes) 10) +(check "country-a/b detail" + (list (dval (car it) 'country-a) (dval (car it) 'country-b)) '("US" "RU")) +(check "asn-a/b detail" + (list (dval (car it) 'asn-a) (dval (car it) 'asn-b)) '(100 200)) +;; same country -> no anomaly +(check "same country -> none" + (length (detect-impossible-travel + (list (list "h1" 1000 "alice" "1.1.1.1" "US" 1) + (list "h1" 2000 "alice" "2.2.2.2" "US" 2)))) 0) +;; gap edge: exactly 30min fires (<=), one ms over does not +(check "gap = 30min -> fires" + (length (detect-impossible-travel + (list (list "h1" 0 "alice" "1.1.1.1" "US" 1) + (list "h1" 1800000 "alice" "2.2.2.2" "RU" 2)))) 1) +(check "gap 1ms over 30min -> none" + (length (detect-impossible-travel + (list (list "h1" 0 "alice" "1.1.1.1" "US" 1) + (list "h1" 1800001 "alice" "2.2.2.2" "RU" 2)))) 0) +;; different users adjacent -> not paired +(check "distinct users -> none" + (length (detect-impossible-travel + (list (list "h1" 1000 "alice" "1.1.1.1" "US" 1) + (list "h1" 2000 "bob" "2.2.2.2" "RU" 2)))) 0) +;; PRIVATE / empty country dropped before pairing (US, PRIVATE, RU -> US~RU) +(check "PRIVATE point dropped -> US~RU pairs" + (length (detect-impossible-travel + (list (list "h1" 1000 "alice" "10.0.0.1" "US" 1) + (list "h1" 1500 "alice" "192.168.0.1" "PRIVATE" 0) + (list "h1" 2000 "alice" "5.5.5.5" "RU" 2)))) 1) +(check "empty-country point dropped" + (length (detect-impossible-travel + (list (list "h1" 1000 "alice" "10.0.0.1" "US" 1) + (list "h1" 1500 "alice" "0.0.0.0" "" 0) + (list "h1" 2000 "alice" "5.5.5.5" "RU" 2)))) 1) +;; min-gap-ms override: a 10-min gap under a 5-min cap -> none +(check "min-gap override caps it out" + (length (detect-impossible-travel + (list (list "h1" 0 "alice" "1.1.1.1" "US" 1) + (list "h1" 600000 "alice" "2.2.2.2" "RU" 2)) + 300000)) 0) + (newline) (if (= fails 0) - (displayln "OK: correlate matches secmon's storage/mod.rs detect_* (batch 1+2).") + (displayln "OK: correlate matches secmon's storage/mod.rs detect_* (batch 1+2+3).") (begin (displayln fails " FAILURES") (exit 1))) --- a/jsecmon/correlate.ss +++ b/jsecmon/correlate.ss @@ -32,6 +32,12 @@ ;;; thin wrappers over detect-sequence-pair with the rule's types/window/text ;;; detect-log-cover a=((host ts type)…) b=((host ts)…) crit then tamper/1h ;;; +;;; Batch 3 — the geo-correlation rule, over auth points the caller has already +;;; geo-resolved (the GeoDb lookup is the deferred I/O, like the SQL fetch) and +;;; sorted by (user, ts) the way the SQL ORDER BY does: +;;; detect-impossible-travel pts=((host ts user ip country asn) …) +;;; consecutive same-user logins from different countries ≤ min-gap (def 30min) +;;; ;;; Faithfulness notes: ;;; * SQL `timestamp_ms / N` is integer division on a positive epoch → quotient; ;;; the bucket's anomaly timestamp is bucket*N (start of the 10-/5-min window). @@ -56,7 +62,7 @@ detect-recon-port-scan detect-data-exfil detect-sequence-pair detect-priv-escalation-chain detect-persistence-after-access detect-lateral-after-shell - detect-log-cover) + detect-log-cover detect-impossible-travel) (import (except (chezscheme) make-hash-table hash-table? sort sort! @@ -332,4 +338,53 @@ (cons 'trigger-time-ms ats) (cons 'tamper-time-ms bts) (cons 'gap-seconds (quotient (- bts ats) 1000))))))))) - crit-a))) + crit-a)) + + ;; ── Rule 13: impossible travel — same user, two countries, < min-gap ──────── + ;; pts = ((host ts user ip country asn) …) ALREADY geo-resolved and sorted by + ;; (user, ts). The GeoDb lookup is the caller's deferred I/O; the PRIVATE/empty + ;; country drop (Rust's filter_map before building points) is pure, so it lives + ;; here. Walk every adjacent pair (Rust `i += 1` always); same user + different + ;; country + gap ≤ min-gap-ms emits a seed. gap_minutes = ms/60000 (quotient). + ;; min-gap-ms default 30min = SECMON_TRAVEL_GAP_MIN default; caller overrides. + (def (detect-impossible-travel pts (min-gap-ms 1800000)) + (let ((kept (filter (lambda (p) + (let ((c (list-ref p 4))) + (not (or (string=? c "PRIVATE") (string-empty? c))))) + pts))) + (let loop ((ps kept) (acc '())) + (if (or (null? ps) (null? (cdr ps))) + (reverse acc) + (let* ((a (car ps)) (b (cadr ps)) + (auser (caddr a)) (buser (caddr b)) + (ac (list-ref a 4)) (bc (list-ref b 4)) + (ats (cadr a)) (bts (cadr b))) + (if (and (string=? auser buser) + (not (string=? ac bc)) + (<= (- bts ats) min-gap-ms)) + (loop (cdr ps) + (cons (impossible-travel-seed a b) acc)) + (loop (cdr ps) acc))))))) + + (def (impossible-travel-seed a b) + (let* ((auser (caddr a)) (aip (list-ref a 3)) (ac (list-ref a 4)) (aasn (list-ref a 5)) + (ahost (car a)) (ats (cadr a)) + (bip (list-ref b 3)) (bc (list-ref b 4)) (basn (list-ref b 5)) + (bhost (car b)) (bts (cadr b)) + (gap-min (quotient (- bts ats) 60000))) + (list (cons 'rule "impossible_travel") + (cons 'description + (str "user '" auser "' authenticated from " ac " (" aip ") and " + bc " (" bip ") within " gap-min " min")) + (cons 'severity "high") + (cons 'timestamp-ms bts) + (cons 'host bhost) + (cons 'details + (list (cons 'username auser) + (cons 'country-a ac) (cons 'country-b bc) + (cons 'ip-a aip) (cons 'ip-b bip) + (cons 'asn-a aasn) (cons 'asn-b basn) + (cons 'host-a ahost) (cons 'host-b bhost) + (cons 'gap-minutes gap-min) + (cons 'first-seen-ms ats) + (cons 'second-seen-ms bts)))))))