Update documentation to reflect pure Scheme YAML implementation
ober
1d7c46a06e0a647ec06ffaa08f3408ff0cd8fc4d
--- a/README.md +++ b/README.md @@ -100,6 +100,7 @@ scheme --libdirs lib --script your-file.ss | `(std text utf8)` | UTF-8 utilities | | `(std text csv)` | CSV read/write | | `(std text xml)` | SXML → XML serialization | +| `(std text yaml)` | YAML load/dump with roundtrip support (pure Scheme, preserves comments/ordering/styles) | | `(std os env)` | `getenv`, `setenv`, `unsetenv` | | `(std os temporaries)` | Temporary file creation | | `(std os signal)` | POSIX signal constants + handlers | @@ -142,7 +143,6 @@ scheme --libdirs lib --script your-file.ss | `(std net httpd)` | [chez-https](https://github.com/ober/chez-https) | `httpd-start`, `httpd-route`, `http-respond-json`, etc. | | `(std net ssl)` | [chez-ssl](https://github.com/ober/chez-ssl) | `ssl-connect`, `tcp-connect`, `tcp-listen`, TLS/TCP networking | | `(std compress zlib)` | [chez-zlib](https://github.com/ober/chez-zlib) | `gzip-bytevector`, `gunzip-bytevector`, `deflate-bytevector` | -| `(std text yaml)` | [chez-yaml](https://github.com/ober/chez-yaml) | `yaml-load`, `yaml-dump`, `yaml-load-string`, `yaml-dump-string` | | `(std db leveldb)` | [chez-leveldb](https://github.com/ober/chez-leveldb) | `leveldb-open`, `leveldb-put`, `leveldb-get`, iterators, batches | | `(std db sqlite)` | [chez-sqlite](https://github.com/ober/chez-sqlite) | `sqlite-open`, `sqlite-query`, `sqlite-eval`, prepared statements | | `(std db postgresql)` | [chez-postgresql](https://github.com/ober/chez-postgresql) | `pg-connect`, `pg-query`, `pg-eval`, parameterized queries | @@ -184,7 +184,7 @@ Runs 289 core tests across reader, core macros, runtime, standard library, FFI, ## Requirements - [Chez Scheme](https://cisco.github.io/ChezScheme/) 10.x (stock, unmodified) -- Optional: [chez-*](https://github.com/ober) libraries for networking, compression, PCRE2, YAML, LevelDB, SQLite, PostgreSQL, epoll, inotify, crypto +- Optional: [chez-*](https://github.com/ober) libraries for networking, compression, PCRE2, LevelDB, SQLite, PostgreSQL, epoll, inotify, crypto ## Project Structure @@ -223,7 +223,11 @@ lib/ utf8.sls # :std/text/utf8 csv.sls # :std/text/csv xml.sls # :std/text/xml - yaml.sls # :std/text/yaml (wraps chez-yaml) + yaml.sls # :std/text/yaml (pure Scheme, roundtrip) + yaml/ + nodes.sls # AST record types + reader.sls # YAML parser + writer.sls # YAML emitter os/ path.sls # :std/os/path env.sls # :std/os/env @@ -267,6 +271,7 @@ tests/ test-modules.ss # 8 module path tests test-expanded-stdlib.ss # 76 expanded stdlib tests test-wrappers.ss # 27 wrapper module tests + test-yaml-roundtrip.ss # 58 YAML roundtrip tests ``` ## What Gerbil Code Works --- a/docs/architecture-split.md +++ b/docs/architecture-split.md @@ -63,9 +63,14 @@ These are pure C glue — no logic beyond marshaling: | Library | Why chez-* | |---------|-----------| -| chez-yaml | Pure parser — no concurrency, no resource management, just string → data | | chez-r7rs | Standards compliance layer — must be stock Chez by definition | +### Replaced by pure Scheme (no FFI needed) + +| Library | Replacement | Notes | +|---------|------------|-------| +| chez-yaml | `(std text yaml)` — pure Scheme | Roundtrip support (preserves comments, key ordering, scalar styles). No C dependency. Eliminates libyaml attack surface. | + ### Migrated to jerboa (completed) | Library | FFI shim in chez-* | Logic in jerboa | --- a/docs/native-rust.md +++ b/docs/native-rust.md @@ -22,13 +22,13 @@ Replace Jerboa's 10+ C library dependencies with a single Rust shared library. E ## Motivation -Jerboa currently links 10+ C shared libraries for crypto, TLS, compression, regex, databases, YAML, and OS integration. Each C library is an independent attack surface: its own CVE history, its own build system, its own ABI compatibility concerns. +Jerboa currently links 9+ C shared libraries for crypto, TLS, compression, regex, databases, and OS integration. (YAML was previously a C dependency but is now pure Scheme.) Each C library is an independent attack surface: its own CVE history, its own build system, its own ABI compatibility concerns. Replacing them with a single Rust shared library (`libjerboa_native.so`) provides: 1. **Memory safety** across all native code — buffer overflows, use-after-free, and double-free become compile-time errors 2. **A single dependency** instead of 10+ — one build, one audit surface, one update path -3. **Solved vulnerability classes** — ReDoS eliminated (Rust regex uses NFA, not backtracking), timing-safe crypto by default (ring), YAML parsing without C memory bugs (unsafe-libyaml) +3. **Solved vulnerability classes** — ReDoS eliminated (Rust regex uses NFA, not backtracking), timing-safe crypto by default (ring). (YAML parsing is now pure Scheme — no C/Rust needed.) 4. **Stable C ABI** — Rust's `extern "C"` produces `.so` files that Chez Scheme loads identically to C libraries 5. **Reproducible builds** — Cargo.lock pins every transitive dependency with exact hashes @@ -58,7 +58,7 @@ Replacing them with a single Rust shared library (`libjerboa_native.so`) provide | **libsqlite3.so** | chez-sqlite | `std/db/sqlite` | SQLite database | | **libpq.so** | chez-postgresql | `std/db/postgresql` | PostgreSQL client | | **libleveldb.so** | chez-leveldb | `std/db/leveldb` | LevelDB key-value store | -| **libyaml.so** | chez-yaml | `std/text/yaml` | YAML parsing | +| ~~**libyaml.so**~~ | ~~chez-yaml~~ | `std/text/yaml` | ~~YAML parsing~~ — **Eliminated**: now pure Scheme with roundtrip support | | **libcurl.so** (or libssl) | chez-https | `std/net/request` | HTTPS client | ### Custom C Code @@ -142,14 +142,15 @@ pub extern "C" fn jerboa_inflate( } ``` -### YAML: libyaml → unsafe-libyaml +### ~~YAML: libyaml → unsafe-libyaml~~ — SUPERSEDED -| Feature | Current (libyaml) | Replacement (unsafe-libyaml) | Notes | -|---------|-------------------|------------------------------|-------| -| YAML 1.1 parsing | C implementation | Mechanical Rust translation of same C code | Same parser, memory-safe | -| Event-based API | `yaml_parser_parse()` | `unsafe_libyaml::Parser` | Same event model | +**This swap is no longer needed.** `(std text yaml)` is now a pure Scheme implementation with full roundtrip support (preserves comments, key ordering, scalar styles, block/flow collection styles). No C dependency, no FFI — the entire libyaml attack surface has been eliminated without needing a Rust replacement. -`unsafe-libyaml` is a c2rust translation of libyaml, cleaned up by David Tolnay (serde maintainer). It's the exact same parser logic, byte-for-byte compatible, but compiled as Rust with bounds checking. Used by `serde_yaml` which is the standard Rust YAML library. +The pure Scheme implementation provides: +- Simple API: `yaml-load`, `yaml-dump` (backward-compatible with old chez-yaml interface) +- Roundtrip API: `yaml-read`, `yaml-write` (returns/consumes AST nodes with comment and style metadata) +- Node manipulation: `yaml-mapping-ref`, `yaml-mapping-set!`, `yaml-ref`, `yaml-set!` +- Security: `*yaml-max-input-size*` (1MB) and `*yaml-max-depth*` (512) limits ### Databases: libsqlite3 → rusqlite, libpq → rust-postgres @@ -186,7 +187,6 @@ jerboa-native-rs/ │ ├── tls.rs # rustls-ffi: connect, accept, read, write │ ├── regex.rs # regex crate: compile, match, find, replace │ ├── compress.rs # flate2: deflate, inflate (with size limits) -│ ├── yaml.rs # serde_yaml: parse string/file to Scheme-friendly format │ ├── sqlite.rs # rusqlite: open, prepare, bind, step, finalize │ ├── postgres.rs # rust-postgres: connect, query, execute │ ├── secure_mem.rs # mlock, guard pages, explicit_bzero, DONTDUMP @@ -218,7 +218,7 @@ rustls-pemfile = "2" webpki-roots = "0.26" regex = "1" flate2 = "1" -unsafe-libyaml = "0.2" +# unsafe-libyaml — no longer needed: (std text yaml) is pure Scheme rusqlite = { version = "0.32", features = ["bundled"] } postgres = "0.19" landlock = "0.4" @@ -675,7 +675,7 @@ Jerboa side: |-----------|----------|-----| | **ReDoS** | Rust regex crate | NFA engine — linear time guaranteed, backtracking impossible | | **Zlib decompression bombs** | flate2 (compress.rs) | Output size limit parameter in Rust wrapper | -| **YAML C parser bugs** | unsafe-libyaml | Same parser, memory-safe Rust compilation | +| ~~**YAML C parser bugs**~~ | ~~unsafe-libyaml~~ | **Eliminated** — `(std text yaml)` is now pure Scheme, no C/Rust needed | | **FFI null pointer deref** | Rust wrappers | Every function validates inputs and checks returns | | **FFI type confusion** | Rust type system | Rust compiler rejects type mismatches | | **Secret material in GC heap** | secure_mem.rs | mlock'd region outside GC, wiped on free | @@ -852,7 +852,7 @@ Rust: `src/epoll.rs`, `src/inotify_native.rs`, `src/landlock.rs`. Scheme: `lib/s | Metric | Before (C libraries) | After (Rust native) | |--------|---------------------|---------------------| | Shared libraries loaded | 10+ | 1 | -| C code in trust boundary | ~500K lines (OpenSSL + libyaml + SQLite + ...) | 0 (Rust only; C primitives inside ring are audited asm) | +| C code in trust boundary | ~500K lines (OpenSSL + SQLite + ...; libyaml already eliminated via pure Scheme) | 0 (Rust only; C primitives inside ring are audited asm) | | Memory safety CVEs possible | Yes — every C library | No — Rust compiler prevents them | | ReDoS possible | Yes — PCRE2 backtracks | No — Rust regex uses NFA | | Secret wiping guarantee | No — GC copies | Yes — secure region is outside GC | --- a/docs/security-reference.md +++ b/docs/security-reference.md @@ -263,7 +263,8 @@ Phases 1-4 are implemented and tested (42 tests in `tests/test-security2-parsers | HTTP/2 | frame size cap | 16 KB | Per-frame | | WebSocket | payload cap | 64 MB | Per-message | | Zlib | decompression limit | configurable | Output size | -| YAML | input size limit | configurable | Total input | +| YAML | input size limit | configurable (`*yaml-max-input-size*`, 10 MB) | Total input | +| YAML | nesting depth limit | configurable (`*yaml-max-depth*`, 512) | Recursion depth | | Actor messages | `*max-message-size*` | 1 MB | Per-message | ### Other hardening --- a/libraries.md +++ b/libraries.md @@ -28,7 +28,6 @@ Gerbil has ~438 `:std/*` modules. Jerboa currently implements 51. This document | [chez-https](https://github.com/ober/chez-https) | HTTP client+server | `(std net request)`, `(std net httpd)` | | [chez-zlib](https://github.com/ober/chez-zlib) | zlib compression | `(std compress zlib)` | | [chez-pcre2](https://github.com/ober/chez-pcre2) | PCRE2 regex | `(std pcre2)` | -| [chez-yaml](https://github.com/ober/chez-yaml) | YAML parser | `(std text yaml)` | | [chez-leveldb](https://github.com/ober/chez-leveldb) | LevelDB | `(std db leveldb)` | ### Completed (New) @@ -41,6 +40,12 @@ Gerbil has ~438 `:std/*` modules. Jerboa currently implements 51. This document | [chez-sqlite](https://github.com/ober/chez-sqlite) | SQLite3 | `(std db sqlite)` | Done | | [chez-postgresql](https://github.com/ober/chez-postgresql) | libpq | `(std db postgresql)` | Done | +### Replaced (No Longer Needed) + +| Library | Replaced By | Notes | +|---------|------------|-------| +| [chez-yaml](https://github.com/ober/chez-yaml) | Pure Scheme `(std text yaml)` | Roundtrip support: preserves comments, key ordering, scalar styles. No C dependency. 58 tests. | + ## Pure Scheme Modules (No External Deps) ### High Priority — Commonly Used