add docs
ober
adc0c5a7f3a7ee76bd7d685a5f53ec7802be1984
new file mode 100644 --- /dev/null +++ b/docs/distrib.md @@ -0,0 +1,513 @@ +# Distributed Computing Roadmap for Jerboa + +Ship work from one jerboa program to other jerboa programs on remote servers. + +## Existing Foundation + +Jerboa already has: + +| Layer | Module | What It Does | +|-------|--------|-------------| +| Transport | `(std net tcp-raw)` | POSIX sockets, fd-based, EINTR retry | +| Transport | `(std net tcp)` | Gerbil-compatible ports over TCP | +| Framing | `(std actor transport)` | 4-byte length + FASL body, cookie auth | +| Actors | `(std actor core)` | spawn, send, link, monitor, remote refs | +| Protocol | `(std actor protocol)` | ask/reply RPC, fire-and-forget tell | +| Supervision | `(std actor supervisor)` | OTP-style restart trees | +| Registry | `(std actor registry)` | Global named actor lookup | +| Scheduler | `(std actor scheduler)` | Work-stealing M:N thread pool | +| Clustering | `(std actor cluster)` | Node join/leave, remote registry | +| Distributed | `(std actor distributed)` | Location-transparent dsend, process groups | +| CRDTs | `(std actor crdt)` | G-Counter, PN-Counter, OR-Set, LWW-Register | +| Checkpoint | `(std actor checkpoint)` | FASL-based actor state snapshots | +| Serialization | `(std fasl)` | Binary serialization, 1000x faster than text | +| Persistence | `(std persist closure)` | Data checkpoint/resume via FASL | +| gRPC | `(std net grpc)` | S-expression RPC over TCP | +| HTTP | `(std net httpd)` | HTTP server with routing | +| Pools | `(std net pool)` | Generic connection pooling | +| Zero-Copy | `(std net zero-copy)` | Pre-allocated buffer pools | +| STM | `(std concur stm)` | Software transactional memory | +| Async | `(std concur async-await)` | Promise-based concurrency | +| Structured | `(std concur structured)` | Lexically-scoped task lifetimes | + +**The bones are production-grade.** What's missing is the glue that turns "actors can send messages" into "programs can ship work to remote servers." + +--- + +## 10 Features to Build + +### 1. `(std net code-ship)` — Remote Code Shipping + +The biggest gap. Distributed actors send *messages* (data), not *code*. To send work to a remote jerboa instance, you need to ship a computation. + +Chez's `compile` + FASL can serialize compiled code objects. A remote node receives a FASL bytevector, loads it, runs it. This is the Ray/Spark model. + +```scheme +;; Ship a lambda + data to a remote node, get result back +(define result + (remote-eval node + '(lambda (data) + (import (std text csv)) + (let ([rows (csv-parse data)]) + (length (filter anomalous? rows)))) + my-csv-data)) + +;; Ship a named function (already deployed on worker) +(remote-call node 'process-batch batch-42) + +;; Ship compiled code object via FASL +(let ([code (compile '(lambda (x) (* x x)))]) + (remote-eval/compiled node code 42)) +;; => 1764 +``` + +**Implementation:** +- Serialize S-expression via FASL, ship over TCP transport +- Remote node `eval`s in a sandboxed environment +- Result serialized back via FASL +- Timeout + error propagation +- Optional: ship pre-compiled code objects (Chez `compile` output) + +**Chez leverage:** `compile` for runtime compilation, `eval` for execution, FASL for serialization of both code and data. + +--- + +### 2. `(std net worker-pool)` — Worker Pool Service + +A daemon that listens for work requests. Start `jerboa-worker` on N machines, submit work from your main program. + +```scheme +;; Start worker daemon (on each server) +(define worker (start-worker-daemon + #:port 9090 + #:threads 8 + #:imports '((std text csv) (std crypto digest)))) + +;; Client: connect to pool of workers +(define pool (connect-worker-pool + '("srv1:9090" "srv2:9090" "srv3:9090" "srv4:9090"))) + +;; Submit work — automatic load balancing +(define results + (distributed-map pool + (lambda (chunk) (heavy-computation chunk)) + (chunk-data big-dataset 100))) +;; 100 chunks distributed across 4 servers, results collected + +;; Single remote eval +(define result (pool-eval pool '(+ 1 2 3))) + +;; Pool status +(pool-status pool) +;; => ((srv1 #:load 0.85 #:tasks 12) +;; (srv2 #:load 0.42 #:tasks 6) +;; (srv3 #:load 0.91 #:tasks 14) +;; (srv4 #:load 0.30 #:tasks 4)) +``` + +**Implementation:** +- Worker daemon: TCP server + thread pool + FASL framing +- Client: connection pool to all workers, round-robin or least-loaded dispatch +- Heartbeat: periodic ping, remove dead workers from pool +- Retry: if worker dies mid-task, re-submit to another +- Backpressure: workers report queue depth, client routes accordingly + +**Protocol (over existing transport framing):** +``` +Client → Worker: (submit task-id thunk-fasl data-fasl timeout-ms) +Worker → Client: (result task-id value-fasl) +Worker → Client: (error task-id message) +Client → Worker: (ping) +Worker → Client: (pong #:load 0.85 #:queue-depth 12) +Client → Worker: (cancel task-id) +``` + +--- + +### 3. `(std net task-queue)` — Persistent Task Queues + +Durable work queues. Submit tasks, workers pull them. If a worker dies, the task gets reassigned. Like Celery/Sidekiq but in Scheme. + +```scheme +;; Broker (central coordinator) +(define broker (start-task-broker #:port 9091 #:persist "/var/jerboa/tasks/")) + +;; Producer: submit tasks +(task-submit! broker 'process-report {:file "big.csv" :format 'csv}) +(task-submit! broker 'send-email {:to "alice@x.com" :subject "Done"}) + +;; Consumer: pull and execute tasks +(task-worker broker + (lambda (task) + (case (task-type task) + [(process-report) (process-csv (task-data task))] + [(send-email) (send-mail (task-data task))]))) + +;; Task status +(task-status broker task-id) +;; => 'running | 'pending | 'completed | 'failed | 'retrying + +;; Retry policy +(task-submit! broker 'flaky-api data + #:retries 3 + #:retry-delay 5000 ;; ms + #:timeout 30000) ;; ms + +;; Scheduled tasks (run at specific time) +(task-schedule! broker 'nightly-report data + #:at (make-time 'time-utc 0 1711065600)) ;; specific timestamp + +;; Priority queues +(task-submit! broker 'urgent-alert data #:priority 'high) +``` + +**Implementation:** +- Broker: TCP server, in-memory queue + FASL persistence to disk +- At-least-once delivery: tasks acked after completion, re-queued on timeout +- Visibility timeout: claimed tasks invisible to other workers for N seconds +- Dead letter queue: tasks that fail after max retries +- FASL persistence: append-only log, periodic compaction + +--- + +### 4. `(std actor consensus)` — Raft Consensus + +Distributed agreement. Leader election, replicated log, linearizable reads. Foundation for distributed locks, config management, primary/replica. + +```scheme +;; Start a Raft cluster +(define node (make-raft-node + #:id "node-1" + #:peers '("node-2:9100" "node-3:9100") + #:port 9100 + #:state-machine (lambda (state cmd) + (case (car cmd) + [(set) (cons (cons (cadr cmd) (caddr cmd)) state)] + [(get) (assq (cadr cmd) state)])))) + +(raft-start! node) + +;; Propose a command (goes through leader) +(raft-propose! node '(set counter 42)) + +;; Read (linearizable — goes through leader) +(raft-read node '(get counter)) +;; => (counter . 42) + +;; Leader info +(raft-leader node) ;; => "node-2" +(raft-state node) ;; => 'follower | 'candidate | 'leader + +;; Distributed lock built on Raft +(with-distributed-lock cluster "my-lock" + (do-critical-section)) +``` + +**Implementation:** +- Leader election via randomized timeouts +- Log replication with majority quorum +- Snapshot + log compaction for bounded memory +- Pre-vote extension to prevent disruptions +- ~500 lines for core Raft, ~200 for transport integration + +--- + +### 5. `(std net pubsub)` — Publish/Subscribe Messaging + +Broadcast patterns across nodes. Topic-based routing with wildcards. + +```scheme +;; Broker +(define broker (start-pubsub-broker #:port 9092)) + +;; Publisher (on any node) +(define pub (pubsub-connect "broker:9092")) +(publish! pub "system.metrics.cpu" {:host "srv1" :value 0.85}) +(publish! pub "system.metrics.mem" {:host "srv1" :value 0.62}) +(publish! pub "app.events.login" {:user "alice"}) + +;; Subscriber (on any node) +(define sub (pubsub-connect "broker:9092")) +(subscribe! sub "system.metrics.*" + (lambda (topic msg) + (when (> (hashtable-ref msg 'value 0) 0.9) + (alert! topic msg)))) + +(subscribe! sub "app.events.#" ;; # matches multiple levels + (lambda (topic msg) (log-event topic msg))) + +;; Unsubscribe +(unsubscribe! sub "system.metrics.*") + +;; Fan-out: all subscribers get every matching message +;; Fan-in: multiple publishers on same topic +``` + +**Implementation:** +- Broker: topic trie for O(log n) wildcard matching +- Subscribers: persistent TCP connections, push-based delivery +- QoS levels: at-most-once (fast) or at-least-once (ack-based) +- Retention: optionally persist last N messages per topic + +--- + +### 6. `(std net dht)` — Distributed Hash Table + +Consistent hashing for sharding data across nodes. Automatic rebalancing. + +```scheme +;; Create DHT cluster +(define dht (join-dht '("node1:9200" "node2:9200" "node3:9200"))) + +;; Store/retrieve (key → responsible node determined by hash ring) +(dht-put! dht "user:alice" {:name "Alice" :age 30}) +(dht-get dht "user:alice") +;; => {:name "Alice" :age 30} + +;; Key is hashed to a position on the ring +;; Each node owns a range of the ring +;; Replication factor: store on N consecutive nodes +(dht-put! dht "user:bob" data #:replicas 3) + +;; When nodes join/leave, only 1/N keys need to move +(dht-join! dht "node4:9200") ;; automatic rebalancing +(dht-leave! dht "node2:9200") ;; keys redistributed + +;; Range queries (if keys are ordered) +(dht-range dht "user:a" "user:m") +``` + +**Implementation:** +- Consistent hashing with virtual nodes (128 vnodes per physical node) +- Gossip protocol for membership +- Sloppy quorum for availability (read R of N, write W of N) +- Hinted handoff for temporary failures +- Anti-entropy via Merkle trees + +--- + +### 7. `(std stream distributed)` — Distributed Stream Processing + +Process infinite event streams across nodes. Windowed aggregations, joins, exactly-once semantics. + +```scheme +;; Define a processing topology +(define topology + (stream-topology "click-analytics" + ;; Source: read from pubsub topic + (source "clicks" (pubsub-source "events.clicks")) + + ;; Filter + (processor "valid-clicks" '("clicks") + (lambda (event) (and (event-url event) event))) + + ;; Key-by for partitioning + (partition "by-url" '("valid-clicks") + (lambda (event) (event-url event))) + + ;; Windowed aggregation (60-second tumbling windows) + (window "counts" '("by-url") + #:type 'tumbling + #:size 60 + (lambda (window events) + {:url (window-key window) + :count (length events) + :period (window-start window)})) + + ;; Sink: write to another topic + (sink "output" '("counts") + (pubsub-sink "analytics.url-counts")))) + +;; Deploy across cluster +(stream-deploy! cluster topology #:parallelism 4) +;; Each partition runs on a different node +;; Automatic failover: if a node dies, partitions reassigned + +;; Monitor +(stream-metrics topology) +;; => ((clicks #:throughput 15000/s #:lag 42) +;; (counts #:throughput 250/s #:windows-open 180)) +``` + +**Implementation:** +- DAG of processors, partitioned by key +- Exactly-once via idempotent writes + offset tracking +- Watermarks for out-of-order event handling +- Checkpoint state to FASL periodically +- Built on existing pubsub + worker pool + +--- + +### 8. `(std net discovery)` — Service Discovery + +Nodes announce capabilities, others discover them. No hardcoded addresses. + +```scheme +;; Service registration (on each server) +(service-register! discovery "image-processor" + #:host (hostname) + #:port 9090 + #:capacity 8 + #:tags '(gpu cuda) + #:health-check (lambda () (gpu-available?))) + +(service-register! discovery "csv-cruncher" + #:host (hostname) + #:port 9091 + #:capacity 16 + #:tags '(cpu high-memory)) + +;; Service discovery (from client) +(define svc (service-discover discovery "image-processor")) +;; => (#:host "gpu-srv1" #:port 9090 #:load 0.3) + +;; Discover with constraints +(define svcs (service-discover-all discovery "image-processor" + #:tags '(gpu) + #:max-load 0.8)) + +;; Automatic load-balanced calls +(define result + (service-call discovery "image-processor" 'resize + image 800 600)) +;; Picks least-loaded healthy instance automatically + +;; Watch for changes +(service-watch! discovery "csv-cruncher" + (lambda (event svc) + (case event + [(joined) (printf "New cruncher: ~a~n" svc)] + [(left) (printf "Cruncher down: ~a~n" svc)] + [(unhealthy) (printf "Cruncher sick: ~a~n" svc)]))) +``` + +**Implementation:** +- Registry: centralized (simple) or gossip-based (resilient) +- Health checks: periodic TCP connect + custom health function +- TTL: services expire if not refreshed +- DNS-SD compatible naming (optional) +- Built on existing actor cluster + gRPC + +--- + +### 9. `(std debug distributed-trace)` — Distributed Tracing + +When a request flows across 5 servers, trace it end-to-end. OpenTelemetry-compatible spans. + +```scheme +;; Start a trace +(with-trace "process-order" {:order-id 12345} + ;; Span 1: validate (local) + (with-span "validate" + (validate-order order)) + + ;; Span 2: charge (remote — trace context propagated automatically) + (with-span "charge-payment" + (remote-call payment-svc 'charge order)) + + ;; Span 3: ship (remote) + (with-span "ship-order" + (remote-call shipping-svc 'ship order))) + +;; Trace context flows through actor messages automatically +;; Each span records: start-time, end-time, parent-span, attributes + +;; Collect traces +(trace-export! collector traces) +;; Outputs Jaeger/Zipkin-compatible format + +;; Trace sampling +(set-trace-sampler! (rate-sampler 0.01)) ;; sample 1% of requests +``` + +**Implementation:** +- Thread-local trace context (trace-id, span-id, parent-span-id) +- Automatic propagation through actor send/dsend +- W3C Trace Context headers for HTTP +- Span collector with configurable export (stdout, file, network) +- Sampling strategies: always, never, rate-based, parent-based + +--- + +### 10. `(std actor migrate)` — Live Actor Migration + +Move a running actor (state + mailbox) from one node to another without downtime. + +```scheme +;; Migrate actor to another node +(migrate-actor! actor-ref target-node) +;; 1. Pause actor on source (drain current message) +;; 2. Serialize state + pending mailbox via FASL +;; 3. Ship to target node over transport +;; 4. Restore actor on target, resume processing +;; 5. Update registry — all refs transparently redirect +;; 6. Future messages route to new location + +;; Bulk migration (rebalancing) +(rebalance-actors! cluster + #:strategy 'least-loaded + #:max-concurrent 5) + +;; Migration with handoff period (no message loss) +(migrate-actor! actor-ref target-node + #:mode 'seamless ;; buffer messages during transfer + #:timeout 30000) ;; abort if takes > 30s + +;; Auto-migration on node shutdown +(on-node-drain node + (lambda (actors) + (distribute-actors! actors remaining-nodes))) +``` + +**Implementation:** +- Pause: set actor flag, let current behavior complete +- Serialize: FASL for state, drain mailbox to list, FASL that too +- Transfer: ship over existing transport layer (4-byte length + FASL) +- Restore: create actor on target, inject state + mailbox, register +- Redirect: update cluster registry, source node forwards stale messages +- Depends on: checkpointing (already exists), transport (already exists) + +--- + +## Implementation Priority + +Build in this order — each builds on the previous: + +| Priority | Feature | Module | Dependencies | Effort | +|----------|---------|--------|--------------|--------| +| **P1** | Worker Pool | `(std net worker-pool)` | tcp, fasl, transport | ~400 lines | +| **P1** | Service Discovery | `(std net discovery)` | tcp, actor/registry | ~300 lines | +| **P1** | Task Queue | `(std net task-queue)` | worker-pool, fasl | ~400 lines | +| **P2** | Pub/Sub | `(std net pubsub)` | tcp, fasl | ~350 lines | +| **P2** | Code Shipping | `(std net code-ship)` | worker-pool, compile | ~250 lines | +| **P2** | Distributed Tracing | `(std debug distributed-trace)` | actor/protocol | ~300 lines | +| **P3** | DHT | `(std net dht)` | tcp, discovery | ~500 lines | +| **P3** | Raft Consensus | `(std actor consensus)` | tcp, fasl | ~500 lines | +| **P3** | Stream Processing | `(std stream distributed)` | pubsub, task-queue | ~600 lines | +| **P3** | Actor Migration | `(std actor migrate)` | checkpoint, transport | ~350 lines | + +## The End Goal + +```scheme +;; Your main program +(import (std net worker-pool) + (std net discovery) + (std net task-queue)) + +;; Find all available compute workers on the network +(define pool (discover-workers "compute-cluster")) + +;; Ship 1000 tasks across all workers — automatic load balancing, +;; retry on failure, results collected in order +(define results + (distributed-map pool + (lambda (chunk) + (import (std text csv)) + (analyze-data (csv-parse chunk))) + (chunk-file "huge-dataset.csv" 1000))) + +(printf "Processed ~a chunks across ~a workers~n" + (length results) (pool-size pool)) +``` + +That's the vision: write normal Scheme, run it across a fleet of jerboa servers. new file mode 100644 --- /dev/null +++ b/docs/fuzzing.md @@ -0,0 +1,613 @@ +# Jerboa Fuzzing Strategy + +A comprehensive plan for discovering crashes, hangs, memory exhaustion, and logic bugs across Jerboa's parsing, networking, and security surfaces. + +--- + +## Table of Contents + +1. [What Fuzzing Finds](#what-fuzzing-finds) +2. [Fuzzing Architecture for Chez Scheme](#fuzzing-architecture-for-chez-scheme) +3. [Target Inventory](#target-inventory) +4. [Target Details](#target-details) +5. [Seed Corpus Strategy](#seed-corpus-strategy) +6. [Harness Design](#harness-design) +7. [Bug Oracles](#bug-oracles) +8. [Infrastructure](#infrastructure) +9. [Triage and Regression](#triage-and-regression) +10. [Implementation Roadmap](#implementation-roadmap) + +--- + +## What Fuzzing Finds + +Fuzzing is uniquely effective at finding bugs that humans and code review miss — the weird corner cases that only emerge from millions of random inputs. In Jerboa specifically, fuzzing targets these bug classes: + +### Crashes and Unhandled Exceptions + +| Bug Class | Where It Hides in Jerboa | Example | +|-----------|--------------------------|---------| +| **Stack overflow from unbounded recursion** | Reader (nested lists/comments), JSON (nested objects/arrays), DNS (compression pointer loops) | `((((((((...1000 deep...))))))))` overflows the Chez stack | +| **Bytevector out-of-bounds** | HTTP/2 frame decode, WebSocket frame decode, DNS response parsing | Frame header says 100 bytes of payload but only 50 bytes exist | +| **Invalid character conversions** | JSON `\uXXXX` escapes, UTF-8 decoder, Hex decoder | `\uDEAD` (lone surrogate) crashes `integer->char` | +| **Assertion failures in parsing state machines** | Reader hash dispatch, CSV quote handling, regex compilation | `#u8(not-a-number)` hits an unguarded branch | +| **Division by zero / arithmetic errors** | Format string processing, HTTP/2 HPACK integer decoding | Format directive with zero-width field | + +### Denial of Service (Hangs and Resource Exhaustion) + +| Bug Class | Where It Hides in Jerboa | Example | +|-----------|--------------------------|---------| +| **Infinite loops** | DNS compression pointers (cycle), regex backtracking (ReDoS), reader block comments | DNS name at offset X with compression pointer back to X | +| **Memory exhaustion via allocation** | HTTP/2 frame with 16MB length field, WebSocket with 2^63 payload length, JSON with million-element arrays | `ws-frame-decode` reads 8-byte length, calls `make-bytevector` with 2^63 | +| **CPU exhaustion via backtracking** | Pregexp engine on pathological patterns, deeply nested schema validation | `(a+)+$` matched against `"aaaaaaaaaaaaaaaaaaaab"` | +| **File descriptor / resource leaks** | Parsers that open ports but don't close on malformed input | `read-json` on a port where the first byte is invalid — does the port get closed? | + +### Logic Bugs and Silent Data Corruption + +| Bug Class | Where It Hides in Jerboa | Example | +|-----------|--------------------------|---------| +| **Silent wrong output** | Base64 decoder accepting invalid characters (returns -1, used as value), hex decoder silently dropping odd final byte | `base64-decode "aGVsbG8@@@"` produces garbage instead of raising an error | +| **Truncated parse without error** | CSV parser dropping the last field when quote is unterminated, JSON accepting trailing garbage | `{"a":1}GARBAGE` parses as `{"a":1}` — no error | +| **Type confusion at boundaries** | FFI layer accepting wrong Scheme types, config system accepting non-string keys | `sqlite-exec` with a bytevector where a string is expected | +| **Differential bugs** | Reader producing different ASTs than Gerbil's reader for the same input | `#;(foo) bar` — does Jerboa handle datum comments identically to Gerbil? | + +### Security-Specific Bugs + +| Bug Class | Where It Hides in Jerboa | Example | +|-----------|--------------------------|---------| +| **Sandbox escape** | `restricted-eval` with crafted syntax objects, continuation capture across sandbox boundary | `(call/cc (lambda (k) k))` captured inside sandbox, invoked outside | +| **Capability forgery** | Currently vectors — any code can construct one (V2 in security.md) | `(vector 'capability 999 'filesystem '((read . #t)))` | +| **Injection via format strings** | `format` called with user-controlled first argument | User input containing `~a` or `~s` directives | +| **Path traversal** | Router parameter extraction, config file paths, sanitize-path edge cases | `:id` param set to `../../etc/passwd` | +| **Integer overflow in size calculations** | HTTP/2 payload length, WebSocket frame length, buffer allocation | Length fields that overflow fixnum range | + +--- + +## Fuzzing Architecture for Chez Scheme + +Chez Scheme is garbage-collected and memory-safe in pure Scheme code, so traditional C fuzzing tools (AFL, libFuzzer) don't directly apply. We need a hybrid approach. + +### Approach 1: Scheme-Level Property-Based Fuzzing (Primary) + +Write Scheme harnesses that generate random inputs and feed them to parsing functions. This catches the majority of bugs: unhandled exceptions, infinite loops, memory bombs, and logic errors. + +```scheme +;; Generic fuzzing harness pattern +(import (jerboa prelude) + (std test)) + +(define (fuzz-target parse-fn input-generator iterations) + (let loop ([i 0]) + (when (< i iterations) + (let ([input (input-generator)]) + (guard (exn [#t (void)]) ;; any exception is OK — crashes are not + (with-time-limit 5 ;; seconds — catches infinite loops + (parse-fn input))) + (loop (+ i 1)))))) +``` + +**Generators needed:** +- Random bytevectors (uniform random bytes) +- Mutated valid inputs (bit flips, byte insertions/deletions, boundary values) +- Grammar-based generators (structurally valid but semantically broken) + +### Approach 2: C-Level Fuzzing for FFI Code (Secondary) + +For modules that call C via FFI (YAML via libyaml, crypto via libcrypto, SQLite, PCRE2), fuzz the C functions directly using AFL++ or libFuzzer with the C shared libraries. + +```c +// Example: fuzz libyaml through chez-yaml's entry point +#include <yaml.h> +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { + yaml_parser_t parser; + yaml_event_t event; + yaml_parser_initialize(&parser); + yaml_parser_set_input_string(&parser, data, size); + while (yaml_parser_parse(&parser, &event)) { + yaml_event_delete(&event); + if (event.type == YAML_STREAM_END_EVENT) break; + } + yaml_parser_delete(&parser); + return 0; +} +``` + +### Approach 3: Differential Fuzzing (Targeted) + +Compare Jerboa's output against Gerbil's output for the same input. Any divergence is a bug in one or both. + +```scheme +;; Compare reader output +(let ([input (generate-random-sexp-string)]) + (let ([jerboa-result (guard (e [#t 'error]) (jerboa-read-string input))] + [gerbil-result (guard (e [#t 'error]) (gerbil-read-string input))]) ;; via subprocess + (unless (equal? jerboa-result gerbil-result) + (report-differential-bug input jerboa-result gerbil-result)))) +``` + +--- + +## Target Inventory + +Targets ordered by priority — a product of attack surface exposure and bug likelihood. + +| Priority | Module | Entry Point | Input Source | Bug Types Expected | +|----------|--------|-------------|-------------|-------------------| +| **P0** | `jerboa/reader` | `jerboa-read`, `jerboa-read-string` | Source files, REPL, `eval` | Stack overflow, hangs, wrong AST | +| **P0** | `std/text/json` | `read-json`, `string->json-object` | HTTP bodies, config files, APIs | Stack overflow, memory, invalid Unicode | +| **P0** | `std/net/http2` | `http2-frame-decode` | Network (untrusted) | Memory exhaustion, OOB, frame confusion | +| **P0** | `std/net/websocket` | `ws-frame-decode` | Network (untrusted) | Memory exhaustion, OOB | +| **P0** | `std/net/dns` | `dns-decode-response` | Network (untrusted) | Infinite loop, OOB, truncation | +| **P1** | `std/pregexp` | `pregexp`, `pregexp-match` | User-provided patterns | ReDoS, stack overflow, invalid escapes | +| **P1** | `std/text/csv` | `read-csv`, `parse-csv-line` | Uploaded files | Unterminated quotes, field explosion | +| **P1** | `std/text/base64` | `base64-decode` | HTTP headers, encoded payloads | Silent wrong output, malformed padding | +| **P1** | `std/text/xml` | `xml-read`, `sxml-parse` | API responses, config | XXE (if entity expansion), recursion | +| **P1** | `std/security/restrict` | `restricted-eval`, `restricted-eval-string` | User-submitted code | Sandbox escape | +| **P2** | `std/text/hex` | `hex-decode` | Encoded data | Odd-length, invalid chars | +| **P2** | `std/text/yaml` | `yaml-load`, `yaml-load-string` | Config files | C library bugs, billion laughs | +| **P2** | `std/format` | `format` | User-controlled format strings | Format injection, arity mismatch | +| **P2** | `std/net/router` | `router-match`, `parse-pattern` | HTTP request paths | Path traversal, segment explosion | +| **P2** | `std/schema` | `validate` | Untrusted input shapes | Recursion bomb, type confusion | +| **P2** | `std/config` | `load-config`, `ht-path-get` | Config files | Nesting bomb, env injection | +| **P3** | `std/text/utf8` | `utf8-decode` | Any text processing | Invalid sequences, bounds | +| **P3** | `std/crypto/digest` | `md5`, `sha256` | Any data | Shell injection (V1 — pre-fix) | +| **P3** | `std/db/sqlite` | `sqlite-exec`, `sqlite-query` | User queries | SQL injection (if not parameterized) | +| **P3** | `std/actor/transport` | message deserialization | Network (inter-node) | Forgery, replay, type confusion | +| **P3** | `jerbuild.ss` | compiler pipeline | Malicious `.ss` files | Compiler crash, infinite expansion | + +--- + +## Target Details + +### T1. Gerbil Reader — `jerboa/reader.sls` + +**Why it's P0**: The reader processes every piece of Scheme source code. If it crashes on malformed input, it affects the REPL, the compiler, `eval`, and any system that reads user-provided S-expressions. + +**Attack surface**: +- `jerboa-read-string` — primary entry for string input +- `jerboa-read` — port-based, used by file loading +- `jerboa-read-all` — reads multiple forms (loops over `jerboa-read`) + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| `((((...1000+...))))` | Nested list recursion depth in `read-list` | Stack overflow — no depth limit | +| `#\| #\| #\| ... #\| nested 1000+ ... \|# \|# \|#` | Block comment nesting in `skip-block-comment!` | Stack overflow | +| `#u8(not numbers here)` | Hash dispatch for bytevector literals | Unhandled parse error | +| `"unterminated string` | EOF inside string literal | Graceful error vs crash | +| `[{(]})` | Mismatched delimiters | Delimiter tracking correctness | +| `#;#;#;#;(((form)))` | Datum comment chaining | Correct skip behavior | +| `#!eof #!void #!bwp` | Chez-specific hash-bang tokens | Handled or rejected cleanly | +| `1/0`, `+nan.0`, `+inf.0` | Special numeric literals | Parsed correctly | +| `\x0;\x1;\x7f;` in identifiers | Control characters | Reader behavior | +| `:keyword`, `key:` | Gerbil keyword syntax | Correct keyword/symbol distinction | +| `"""heredoc\n...\n"""` | Heredoc string syntax | Delimiter matching, EOF handling | + +**Oracle**: Compare against `(read (open-input-string input))` from Chez and against Gerbil's reader for differential testing. + +### T2. JSON Parser — `std/text/json.sls` + +**Why it's P0**: JSON is the primary data interchange format. Any server accepting JSON from the network will hit this parser with untrusted input. + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| `{"a":{"b":{"c":...}}}` 10000 deep | Object recursion | Stack overflow | +| `[[[[...]]]]` 10000 deep | Array recursion | Stack overflow | +| `"\uD800"` | Lone high surrogate | `integer->char` crash | +| `"\uDFFF"` | Lone low surrogate | `integer->char` crash | +| `"\uD800\uDC00"` | Surrogate pair | Correct UTF-16 decoding? | +| `1e999999999` | Number overflow | Bignum or inf? | +| `0.0000...0001` (1000 zeros) | Precision exhaustion | Hang or memory | +| `{"a":1}{"b":2}` | Multiple root values | Trailing garbage accepted? | +| `[1,2,3,]` | Trailing comma | Error or silent accept | +| `"\/"`, `"\b"`, `"\f"` | All JSON escapes | Correct character | +| `"\u0000"` | Null byte in string | Embedded NUL handling | +| 100MB string value | Memory | Allocation limit | + +**Oracle**: Compare output against Python's `json.loads()` or jq. Any parse success/failure disagreement is a bug. + +### T3. HTTP/2 Frame Decoder — `std/net/http2.sls` + +**Why it's P0**: HTTP/2 frames come directly from the network. An attacker controls every byte. + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| Bytevector < 9 bytes | Minimum frame header size | OOB `bytevector-u8-ref` | +| Length field = 0xFFFFFF (16MB) | Max payload allocation | Memory exhaustion | +| Length field > actual bytes | Payload/header mismatch | OOB read or short read | +| Frame type = 0xFF | Unknown frame type | Silent accept or clean error | +| Stream ID with reserved bit set | Bit 0 of stream ID | Masking correctness | +| HPACK index > 61 | Static table bounds | OOB in static table lookup | +| HPACK integer overflow | Multi-byte integer encoding | Arithmetic overflow | +| CONTINUATION frame without HEADERS | Frame sequencing | State machine correctness | +| SETTINGS frame with unknown IDs | Settings parsing | Ignored or crash | +| DATA frame with PADDED flag + pad length > payload | Padding arithmetic | Negative/underflow | + +### T4. WebSocket Frame Decoder — `std/net/websocket.sls` + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| Bytevector of 0 bytes | Minimum size | OOB | +| Bytevector of 1 byte | Only FIN/opcode, no length | OOB on length byte | +| Length = 126, bytevector = 3 bytes | Extended 16-bit length undersize | OOB | +| Length = 127, bytevector = 5 bytes | Extended 64-bit length undersize | OOB | +| 64-bit length = 2^63 - 1 | Maximum payload | Memory exhaustion | +| 64-bit length = 2^63 (MSB set) | Negative length in signed interpretation | Signedness bug | +| Mask bit set, data too short for mask key | Mask key read | OOB | +| Mask XOR with payload shorter than claimed | Unmasking loop | OOB | +| RSV bits set | Reserved extension bits | Ignored or error | + +### T5. DNS Wire Format Parser — `std/net/dns.sls` + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| Bytevector < 12 bytes | Minimum DNS header | OOB | +| Compression pointer to self | `offset → offset` | Infinite loop | +| Compression pointer cycle | `A → B → A` | Infinite loop | +| Compression pointer past end of message | Out-of-bounds offset | OOB | +| Label length = 255 (max) | Name length limit | Allocation | +| QDCOUNT = 65535, no question data | Count/data mismatch | OOB or hang | +| A record with RDLENGTH = 3 (needs 4) | Short record data | OOB | +| AAAA record with RDLENGTH = 0 | Zero-length IPv6 | OOB | +| TXT record with length > RDLENGTH | Internal length mismatch | OOB | +| All-zero message | Minimal valid? | Behavior check | + +### T6. Regex Engine — `std/pregexp.sls` + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| `(a+)+$` vs `"aaa...ab"` | Catastrophic backtracking (ReDoS) | CPU exhaustion | +| `(a\|a)*$` vs `"aaa...ab"` | Exponential matching | CPU exhaustion | +| `(.+)*$` | Nested quantifiers | CPU exhaustion | +| `[[:nonexistent:]]` | Invalid POSIX class | Error handling | +| `\99` | Non-existent backreference | OOB or error | +| `(?:` (unterminated) | Incomplete group | Error handling | +| 100KB pattern string | Pattern compilation | Memory/time | +| `[^]` | Empty negated class | Behavior | +| `\p{Lu}` | Unicode property (if supported) | Feature support | + +### T7. Sandbox — `std/security/restrict.sls` + +**Specific fuzz vectors**: + +| Vector | What It Tests | Expected Bug | +|--------|--------------|-------------| +| `(eval '(open-input-file "/etc/passwd"))` | Blocked binding access | Escape | +| `(call/cc (lambda (k) k))` | Continuation capture | Escape via continuation | +| `(interaction-environment)` | Environment access | Escape to full env | +| `(compile '(system "id"))` | Compile + eval | Bypass via compilation | +| `(record-type-descriptor ...)` | RTD access | Internal access | +| `(with-exception-handler ...)` chains | Exception handler manipulation | Control flow escape | +| `(parameterize ...)` with internal params | Parameter mutation | State escape | +| `(define-syntax ...)` with `syntax-case` | Macro that references blocked bindings | Indirect access | +| `(load "malicious.ss")` | File loading | Should be blocked | +| `(foreign-procedure ...)` | Direct FFI | Should be blocked | + +--- + +## Seed Corpus Strategy + +Every fuzzer is only as good as its starting corpus. For each target: + +### Reader +- All files in `tests/` — valid Gerbil source +- Gerbil's own test suite reader tests +- Edge case files: empty, single character, BOM, all-whitespace +- Files from popular Gerbil projects + +### JSON +- RFC 8259 test vectors +- JSONTestSuite (github.com/nst/JSONTestSuite — 300+ edge cases) +- Valid JSON from real APIs (GitHub, etc.) +- json.org examples + +### Network Protocols (HTTP/2, WebSocket, DNS) +- Captured pcap data decoded to raw frames +- RFC test vectors where available +- Wireshark-generated malformed frames +- h2spec test frames for HTTP/2 + +### Regex +- Patterns from real codebases +- ReDoS pattern databases (e.g., from snyk advisory DB) +- POSIX regex test suites + +### Sandbox +- Known Chez Scheme sandbox escape techniques +- CTF challenge solutions for Scheme sandboxes +- Gerbil's own restricted-eval tests as baseline + +--- + +## Harness Design + +### Standard Harness Template + +Each fuzz target gets a harness file in `tests/fuzz/fuzz-<target>.ss`: + +```scheme +(import (jerboa prelude) + (std test)) + +;; Configuration +(define *iterations* (or (getenv-number "FUZZ_ITERATIONS") 100000)) +(define *max-input-size* (or (getenv-number "FUZZ_MAX_SIZE") 65536)) +(define *timeout-seconds* 5) + +;; Input generation +(define (random-bytes n) + (let ([bv (make-bytevector n)]) + (do ([i 0 (+ i 1)]) + ((= i n) bv) + (bytevector-u8-set! bv i (random 256))))) + +(define (random-input) + (let ([size (+ 1 (random *max-input-size*))]) + (utf8->string (random-bytes size)))) ;; will produce invalid UTF-8 — that's intentional + +;; Mutator: flip random bits in a valid input +(define (mutate-string s) + (let* ([bv (string->utf8 s)] + [pos (random (bytevector-length bv))] + [bit (random 8)]) + (bytevector-u8-set! bv pos + (fxlogxor (bytevector-u8-ref bv pos) (fxsll 1 bit))) + (guard (e [#t s]) ;; if invalid UTF-8, return original + (utf8->string bv)))) + +;; Harness +(define (fuzz-once parse-fn input) + (guard (exn [#t (void)]) ;; any Scheme exception is acceptable + (with-time-limit *timeout-seconds* + (parse-fn input)))) + +(define (run-fuzz name parse-fn gen-fn) + (display (format "Fuzzing ~a for ~a iterations...\n" name *iterations*)) + (let loop ([i 0] [crashes 0]) + (if (>= i *iterations*) + (begin + (display (format "Done. ~a iterations, ~a timeouts/crashes\n" i crashes)) + crashes) + (let ([input (gen-fn)]) + (let ([ok? (fuzz-once parse-fn input)]) + (loop (+ i 1) (if (eq? ok? (void)) crashes (+ crashes 1)))))))) +``` + +### Bytevector Harness (for network protocols) + +```scheme +;; For HTTP/2, WebSocket, DNS — input is raw bytes, not strings +(define (random-bytevector max-size) + (random-bytes (+ 1 (random max-size)))) + +(define (mutate-bytevector bv) + (let* ([copy (bytevector-copy bv)] + [pos (random (bytevector-length copy))]) + ;; Random mutation: flip, insert, delete, or set to boundary value + (case (random 4) + [(0) (bytevector-u8-set! copy pos (fxlogxor (bytevector-u8-ref copy pos) (fxsll 1 (random 8))))] + [(1) (bytevector-u8-set! copy pos 0)] ;; null + [(2) (bytevector-u8-set! copy pos 255)] ;; max byte + [(3) (bytevector-u8-set! copy pos (random 256))]) ;; random + copy)) +``` + +### Sandbox Harness (special — measures escape, not crash) + +```scheme +;; The oracle is different: success means the sandbox HELD. +;; A "bug" is when sandboxed code accesses something it shouldn't. + +(define (fuzz-sandbox input-expr) + (let ([result (guard (e [#t 'exception]) + (restricted-eval input-expr))]) + ;; Check for signs of escape + (when (and (not (eq? result 'exception)) + (or (port? result) + (procedure? result) ;; might be a captured continuation + (and (string? result) + (string-contains result "/etc/")))) + (report-sandbox-escape input-expr result)))) +``` + +--- + +## Bug Oracles + +A fuzzer that only checks "did it crash?" misses half the bugs. Each target needs specific oracles. + +### Crash Oracle (all targets) +Any uncaught exception that isn't a well-formed condition with a message is a bug. Chez `&assertion` with a clear message is fine. Segfault is always a bug. + +### Hang Oracle (all targets) +Time limit per input. Reader, JSON, DNS, and regex are the highest-risk targets. Default: 5 seconds per input for parsers, 30 seconds for regex (backtracking is inherently slow). + +### Memory Oracle (network parsers) +Track `(current-memory-bytes)` before and after. A single input that causes >100MB allocation is a memory bomb bug. + +### Differential Oracle (reader, JSON) +Compare output against a reference implementation. Divergence on well-formed input is always a bug. Divergence on malformed input should be logged and triaged. + +### Roundtrip Oracle (JSON, base64, hex) +`decode(encode(x))` should equal `x`. `encode(decode(valid-input))` should equal the canonical form of `valid-input`. + +### Idempotence Oracle (reader) +`read(write(read(input)))` should equal `read(input)` for valid inputs. + +### Sandbox Escape Oracle +Any capability gained that isn't in the 29-binding safe set is a critical security bug. + +--- + +## Infrastructure + +### Directory Layout + +``` +tests/fuzz/ +├── harness/ +│ ├── fuzz-reader.ss +│ ├── fuzz-json.ss +│ ├── fuzz-http2.ss +│ ├── fuzz-websocket.ss +│ ├── fuzz-dns.ss +│ ├── fuzz-csv.ss +│ ├── fuzz-base64.ss +│ ├── fuzz-pregexp.ss +│ ├── fuzz-sandbox.ss +│ └── fuzz-all.ss ;; runs all harnesses +├── corpus/ +│ ├── reader/ ;; seed inputs per target +│ ├── json/ +│ ├── http2/ +│ ├── websocket/ +│ ├── dns/ +│ └── pregexp/ +├── crashes/ ;; reproducer inputs that triggered bugs +│ └── <target>-<hash>.input +├── coverage/ ;; coverage data (if Chez supports it) +└── README.md ;; how to run +``` +