ioc: expose match_iocs per-type field routing (pure data)
ober
9b5d9280c01b69e824eb40775b417095c81e01d0
--- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ make platform-mounts-check # is_dangerous_path (per-platform exact set) + get_mo make analyze-cli-check # analyze bin: parse_duration_ms + AlertSink::parse + --flag scanners make collector-cli-check # collector bin: --after/--format/--db + host normalize + hosts-file make event-summary-check # storage readers: extract_pid/extract_process_name/build_summary -make ioc-check # storage IOC: detect_ioc_type + is_ipv4 + parse_ioc_text +make ioc-check # storage IOC: detect_ioc_type + is_ipv4 + parse_ioc_text + match field routing make revshell-check # revshell: is_shell/is_c2_port/is_legitimate + classify_connection make cron-check # cron: per-platform CRON/PERIODIC path tables + systemd/periodic route make logtamper-check # logtamper: system-log/history tables + classify-tamper (trunc/mtime) @@ -147,7 +147,7 @@ then crypto orchestration, then I/O / async / FFI (monitors, server, storage). | `bin/analyze` (CLI parse helpers) | `jsecmon/analyze-cli.ss` | ✅ **untyped layer** — the pure argument parsers of the `analyze` binary, returning the prelude Result (ok/err) to mirror Rust's `Result<_, String>` **including the exact error text**: `parse_duration_ms` (`10m`/`2h`/`1d`/bare-seconds → ms; splits leading ASCII digits from the unit; empty → `empty duration`, bad number/leading-non-digit → `invalid duration: {s}`, bad unit like `m5` → `unknown duration unit: …`; the number must fit i64) and `AlertSink::parse` (`stdout` / `file:PATH` / `webhook:URL` / `syslog` / `syslog:TAG`, first-match in order, remainder taken verbatim so `file:` → empty path) and `parse_alert_sinks` (collect every `--alert-to <spec>`, parsing each and short-circuiting on the first bad spec like Rust's `?`; a trailing `--alert-to` with no value is skipped, and no flags → the empty list — the watch-time default-to-stdout lives in `cmd_watch`), plus the generic `--flag` scanners shared across the CLI (`parse_flag_value` → the arg after the **first** `flag`, or `#f` even when the flag is last; `has_flag` → membership; `is_json_format` → the first `--format` that has a value decides, a trailing `--format` is skipped). Pure string→Result/bool; the sink dispatch (stdout/file append/curl webhook/`logger` syslog) and query dispatch are the deferred I/O. secmon has no `#[test]` here so `make analyze-cli-check` asserts against the Rust source. (`format_ts`/`format_ts_iso` are chrono-calendar-coupled display helpers — deferred with the other calendar I/O.) | | `bin/collector` (CLI/hosts parse helpers) | `jsecmon/collector-cli.ss` | ✅ **untyped layer** — the pure argument/hosts parsing of the `collector` binary, with the async polling + ECIES/PSK key loading + SQLite I/O deferred: `parse_after_seq` (first `--after` value as u64, `unwrap_or(0)` so junk/negative/≥2⁶⁴ → 0), `parse_format` (→ `'json`/`'human`/`'quiet`; a per-index scan where an unknown `--format` value does **not** consume the value — differs from analyze's `is_json_format` — and the no-flag default is `quiet` when a `--db` is present else `human`), `parse_db_path`, `normalize_host` (append `:31337` unless the host already contains **any** `:`, so bare IPv6 is left as-is, faithfully), `collect_positional_hosts` (skip the four value flags **and** their values, drop other `--` args, normalize the rest), and `parse_hosts_file`'s pure contents→hosts core (trim, drop blanks/`#` comments, normalize). secmon has no `#[test]` here so `make collector-cli-check` asserts against the Rust source. | | `storage` event readers (`extract_pid` / `extract_process_name` / `build_summary`) | `jsecmon/event-summary.ss` | ✅ **untyped layer** — the pure readers that turn an event's flat JSON `data` (a hash table, as `string->json-object` yields) back into a pid / process name / one-line summary, with the SQLite query + serde plumbing left to storage. Each field is read through a **typed** getter so only a JSON value of the right type counts (`as_u64`/`as_i64`/`as_str`/`as_bool`); `extract_pid` walks pid→source_pid→spawned_pid→web_server_pid and truncates the first hit to **u32** (Rust `v as u32`, so ≥2³² wraps, and pid 0 is a real hit); `extract_process_name` walks process_name→name→exe→source_process→spawned_process. `build_summary` reproduces every per-type format with the exact `unwrap_or` defaults (`"?"`/`0`), the `process_exit` exit-code *option* (Some(0) still prints `(0)`), the nested `selinux_event` perm/class/path-vs-message branches with the 80-char message cap, and the catch-all that scans values **in sorted key order** (serde's default BTreeMap) for the first string longer than 3 chars (capped at 80) else the event type. `make event-summary-check` (43 cases) reproduces secmon's `test_extract_helpers` + `test_build_summary` and adds every per-type / typed-getter corner derived from the source. | -| `storage` IOC parsing (`detect_ioc_type` / `is_ipv4` / `parse_ioc_text`) | `jsecmon/ioc.ss` | ✅ **untyped layer** — the pure indicator classifier behind threat-list ingestion (the file read in `load_ioc_file` is the deferred I/O). `detect_ioc_type` is first-match ip→hash→domain→process: `is_ipv4` (split on `.`, exactly 4 non-empty ≤3-char all-digit groups — **no** 0–255 range check, so `999.999.999.999` is still Ip and `1.2.3.4444` is not), then IPv6 (`:` present and every char hex-or-`:`), then a 32/40/64-length all-hex Hash (MD5/SHA1/SHA256, case-insensitive), then a `.`-bearing space-free Domain, else Process. `parse_ioc_text` trims, drops blanks and `#` comments, and tags each remaining line (order preserved). `make ioc-check` reproduces the `test_ioc_type_detection` #[test] plus the is_ipv4 / parse corners. | +| `storage` IOC parsing (`detect_ioc_type` / `is_ipv4` / `parse_ioc_text`) | `jsecmon/ioc.ss` | ✅ **untyped layer** — the pure indicator classifier behind threat-list ingestion (the file read in `load_ioc_file` is the deferred I/O). `detect_ioc_type` is first-match ip→hash→domain→process: `is_ipv4` (split on `.`, exactly 4 non-empty ≤3-char all-digit groups — **no** 0–255 range check, so `999.999.999.999` is still Ip and `1.2.3.4444` is not), then IPv6 (`:` present and every char hex-or-`:`), then a 32/40/64-length all-hex Hash (MD5/SHA1/SHA256, case-insensitive), then a `.`-bearing space-free Domain, else Process. `parse_ioc_text` trims, drops blanks and `#` comments, and tags each remaining line (order preserved). `ioc_type_json_fields` / `ioc_type_column_fields` expose the per-type JSON-path and column lists `match_iocs` probes (pure routing data; Process is the only type with a plain column, `process_name`). `make ioc-check` reproduces the `test_ioc_type_detection` #[test] plus the is_ipv4 / parse / field-routing corners. | | `monitor/revshell` reverse-shell classifiers (`classify_connection` + helpers) | `jsecmon/revshell.ss` | ✅ **untyped layer** — the pure deciders of the reverse-shell monitor (connection/PID enumeration, event emission, and the dedup set stay in the monitor loop). `is_shell` / `is_revshell_tool` are **exact** lower-cased name membership; `is_c2_port` tests the 17-port C2 set; `is_legitimate_service` fires only on 443/8080/8443 and matches a **substring** of the name. `extract_addr_from_cmdline` finds `/dev/tcp/`, splits the remainder on `/`, and on ≥2 pieces returns `(addr . u16-port)` (first whitespace token of piece 1, junk/out-of-range→0), else `("unknown" . 0)`. `classify_connection` runs checks 1–4 first-match: shell→`shell-outbound`, C2-port-and-not-legit→`known-c2-port`, revshell-tool→`shell-outbound`, any `REVSHELL_PATTERNS` substring→`suspicious-redirect`, else `#f`. revshell.rs has no #[test], so `make revshell-check` (47 cases) **is** the spec. | | `monitor/cron` scheduled-task path tables + classifier | `jsecmon/cron.ss` | ✅ **untyped layer** — the pure pieces of the cron / systemd-timer / periodic monitor (the baseline walk + change detection stay in the loop). secmon keys `CRON_PATHS` / `PERIODIC_PATHS` off `#[cfg(target_os)]`, so `cron-paths` / `periodic-paths` are functions of a platform symbol (`'linux` / `'freebsd` / `'other`) reproducing the three cfg arms verbatim. `is-systemd-or-periodic-path` is the routing predicate `baseline_all` uses to decide whether a `PERIODIC_PATHS` entry is a systemd unit dir vs another cron-like dir — a plain **substring** test for `"systemd"` OR `"periodic"`. cron.rs has no #[test], so `make cron-check` asserts the full tables + the classifier and **is** the spec. | | `monitor/logtamper` log-tamper decision core | `jsecmon/logtamper.ss` | ✅ **untyped layer** — the pure pieces of the log-tampering monitor (the `fs::metadata` polling + size/mtime tracking map stay in the loop). Exposes the `SYSTEM_LOGS` / `HISTORY_FILES` constant tables and `TRUNCATION_THRESHOLD` (1000). `is-history-file` is any-`HISTORY_FILES`-**substring**. `classify-tamper old-size new-size old-mtime new-mtime path` reproduces `check_tampering`'s Ok-arm in push order: size dropped by **>** threshold → `history-cleared` (if a history file) else `truncated`; mtime went backwards and `> 0` → `timestamp-modified` (both can fire for one file). The `deleted` case is the `fs::metadata` Err arm (deferred I/O). logtamper.rs has no #[test], so `make logtamper-check` **is** the spec. | --- a/examples/ioc_check.ss +++ b/examples/ioc_check.ss @@ -64,6 +64,22 @@ (check "empty text -> empty" (parse-ioc-text "") '()) (check "only comments -> empty" (parse-ioc-text "# x\n#y\n") '()) +;; ── ioc-type-json-fields / ioc-type-column-fields (match_iocs routing) ─────── +(displayln "field routing:") +(check "ip json fields" + (ioc-type-json-fields 'ip) + '("$.remote_addr" "$.local_addr" "$.server_addr" "$.target_host" "$.remote_host")) +(check "domain json fields" (ioc-type-json-fields 'domain) '("$.query_name")) +(check "hash json fields" (ioc-type-json-fields 'hash) '("$.old_hash" "$.new_hash")) +(check "process json fields" + (ioc-type-json-fields 'process) + '("$.name" "$.exe" "$.source_process" "$.spawned_process")) +(check "unknown json fields -> empty" (ioc-type-json-fields 'bogus) '()) +(check "process column fields" (ioc-type-column-fields 'process) '("process_name")) +(check "ip column fields -> empty" (ioc-type-column-fields 'ip) '()) +(check "domain column fields -> empty" (ioc-type-column-fields 'domain) '()) +(check "hash column fields -> empty" (ioc-type-column-fields 'hash) '()) + (newline) (if (= fails 0) (displayln "OK: ioc matches secmon's storage/mod.rs IOC detection.") --- a/jsecmon/ioc.ss +++ b/jsecmon/ioc.ss @@ -3,9 +3,11 @@ ;;; ;;; The pure indicator-of-compromise helpers used to ingest threat lists, with ;;; the file read left to the storage layer (load_ioc_file's fs::read): -;;; is-ipv4 : string -> #t | #f -;;; detect-ioc-type : value -> 'ip | 'hash | 'domain | 'process -;;; parse-ioc-text : text -> ((value type) …) +;;; is-ipv4 : string -> #t | #f +;;; detect-ioc-type : value -> 'ip | 'hash | 'domain | 'process +;;; parse-ioc-text : text -> ((value type) …) +;;; ioc-type-json-fields : type -> (json-path …) (match_iocs routing) +;;; ioc-type-column-fields: type -> (column …) (process only) ;;; An Ioc is rendered here as a two-element list (value type-symbol); the four ;;; IocType variants map to the symbols 'ip / 'hash / 'domain / 'process. ;;; @@ -26,7 +28,8 @@ ;;; reproduces every assertion and adds the is-ipv4 / parse-ioc-text corners). (library (jsecmon ioc) - (export is-ipv4 detect-ioc-type parse-ioc-text) + (export is-ipv4 detect-ioc-type parse-ioc-text + ioc-type-json-fields ioc-type-column-fields) (import (except (chezscheme) make-hash-table hash-table? sort sort! @@ -79,4 +82,20 @@ (and (not (string-empty? t)) (not (string-prefix? "#" t)) (list t (detect-ioc-type t))))) - (string-split text #\newline)))) + (string-split text #\newline))) + + ;; The JSON-path / column field lists match_iocs probes for each IOC type + ;; (storage/mod.rs match_iocs) — pure routing data, not logic. Process is the + ;; only type that also matches a plain column ("process_name"). + (def (ioc-type-json-fields ioc-type) + (cond + ((eq? ioc-type 'ip) + '("$.remote_addr" "$.local_addr" "$.server_addr" "$.target_host" "$.remote_host")) + ((eq? ioc-type 'domain) '("$.query_name")) + ((eq? ioc-type 'hash) '("$.old_hash" "$.new_hash")) + ((eq? ioc-type 'process) + '("$.name" "$.exe" "$.source_process" "$.spawned_process")) + (else '()))) + + (def (ioc-type-column-fields ioc-type) + (if (eq? ioc-type 'process) '("process_name") '())))