docs: add bundling-chez plan
ober
988b51e45444fbbdb6b6439ba15c1985b21a2d86
new file mode 100644 --- /dev/null +++ b/docs/bundling-chez.md @@ -0,0 +1,310 @@ +# Bundling Chez + +A plan to make Jerboa self-contained — no system Chez Scheme required on +the build or development machine. The model is Gerbil's recent shift from +"Gambit must be installed" to "Gambit ships with Gerbil." + +## Thesis + +Today, Jerboa is split-bundled: + +- The **shipped binary** (`make binary` output) is fully self-contained. + `petite.boot`, `scheme.boot`, the Chez kernel (`libkernel.a`), and a + WPO-compiled entry program are linked into one ELF/Mach-O file. End + users need no Chez install. +- The **build** is not self-contained. `support/build-binary.sh` globs + `/usr/local/lib /usr/lib /usr/lib64 /opt/homebrew/lib /opt/local/lib` + for `csv*/<machine-type>/` and fails if Chez isn't installed. +- The **dev driver** (`bin/jerboa`) calls `${SCHEME:-scheme}` — needs + system Chez to run. +- The **musl static pipeline** (`support/musl-chez-build.sh`) builds Chez + from source, but expects the source at `$HOME/mine/ChezScheme` — not + vendored, not a submodule, not part of the Jerboa repo. + +The goal: close the gap so `git clone jerboa && make` produces a working +Jerboa with no system Chez anywhere. + +## What's already in place + +A surprising amount. The C side of embedding is largely solved. + +### `support/build-binary.sh` (181 lines) + +The shipped-binary pipeline. Already does the hardest part: + +``` +[1/4] WPO compile → entry.wp.so +[2/4] Embed boots → petite_boot.h, scheme_boot.h, program_boot.h (as C arrays) +[3/4] Generate main.c → calls Sscheme_init / Sregister_boot_file_bytes / Sscheme_program +[4/4] Link → $CC main.c libkernel.a [liblz4.a libz.a] $OS_LIBS +``` + +Cross-platform: FreeBSD/Darwin/Linux branches for compiler and link flags. +Output: a 4.2M Mach-O on macOS arm64, ELF on Linux. Static-linkable on +Linux/musl. + +### `support/jerboa-embed.{c,h}` (459 lines total) + +C ABI for embedding Jerboa as a library: + +```c +jerboa_t *j = jerboa_new(NULL); +jerboa_eval(j, "(def x 42)"); +int64_t v = jerboa_get_int(j, "x"); +jerboa_destroy(j); +``` + +Functions: `jerboa_new/destroy`, `jerboa_eval[_safe]`, `jerboa_get_{int, +double,string,bool}`, `jerboa_call_{int,string}`, `jerboa_error_free`. +Configurable boot file path and lib dirs. Already supports multiple +independent instances. + +### `support/musl-chez-build.sh` + +Builds Chez from source against musl libc using Chez's native `--static` +flag (v10.4.0+): + +- Adds `-static` to LDFLAGS +- Disables dynamic loading (no dlopen) +- Disables curses/x11/iconv +- Embeds boot files via `static_boot_init()` +- Provides `main.o` for downstream static builds + +Expects source at `$HOME/mine/ChezScheme`. Installs to `/opt/chez-musl`. + +## The gap + +What separates "today" from "Gerbil-style fully bundled": + +1. **Chez source isn't in the repo.** Both the dev workflow and the musl + pipeline assume Chez exists somewhere else. Closing this gap means + either (a) git-submodule the canonical Chez source, or (b) vendor a + trimmed snapshot. + +2. **`make` doesn't build Chez.** A user with no system Chez can't run + `make binary` today — step 4 needs `libkernel.a`. We need a `make + bootstrap` (or `make chez`) target that builds Chez before anything + else, and downstream targets that depend on it. + +3. **`bin/jerboa` calls system `scheme`.** The dev driver assumes + `scheme` is on `$PATH`. After bootstrap, it should point at the + built-from-source binary unless overridden. + +4. **`--static` is musl-only in current usage.** Chez's `--static` flag + works on glibc/macOS too, but `musl-chez-build.sh` is the only script + exercising it. The general bundling path should generalize. + +5. **Tests assume system Chez.** Every test target in the Makefile is + `$(SCHEME) --libdirs $(LIBDIRS) --script tests/...`. After bootstrap, + `$(SCHEME)` should default to the bundled `scheme`. + +## Plan + +### Stage 0 — vendor Chez + +Add Chez Scheme as a git submodule at `vendor/ChezScheme/` (or pin a +specific tag and vendor as a tarball — see "Open questions" below). +Pin to a known-good Chez version (current target: v10.4.0+ for `--static` +support). + +**Acceptance**: +- `git clone --recursive jerboa` (or `make vendor`) gets the Chez source +- `vendor/ChezScheme/` is a valid Chez source tree +- License files are preserved alongside the source +- A `LICENSE-CHEZ` (or similar) is added at the Jerboa root for + attribution + +**Effort**: half a day. + +### Stage 1 — `make chez` target + +Add a `make chez` target that builds Chez from `vendor/ChezScheme/` and +installs into `build/chez/` (in-tree, not `/opt/`). Should work on +Darwin/Linux/FreeBSD without `sudo`. + +Sketch: +```makefile +CHEZ_BUILD_DIR := $(CURDIR)/build/chez +CHEZ_BIN := $(CHEZ_BUILD_DIR)/bin/scheme +CHEZ_CSV_DIR := $(CHEZ_BUILD_DIR)/lib/csv*/$(MACHINE_TYPE) + +chez: $(CHEZ_BIN) + +$(CHEZ_BIN): + cd vendor/ChezScheme && \ + ./configure --installprefix=$(CHEZ_BUILD_DIR) --threads && \ + $(MAKE) -j$(JOBS) && \ + $(MAKE) install +``` + +**Acceptance**: +- `make chez` works on macOS/Linux/FreeBSD with no `sudo`, no system + Chez install +- Builds reproducibly (same Chez tag → byte-identical kernel) +- Cleaned by `make clean` (without nuking the source submodule) +- A separate `make chez-clean` for full Chez rebuild + +**Effort**: 1–2 days, mostly bashing through configure-script quirks per +platform. + +### Stage 2 — wire bundled Chez into everything + +Switch `SCHEME` and the CSV-lookup logic to prefer the in-tree build: + +- **`Makefile`**: `SCHEME ?= $(CHEZ_BIN)` (was `SCHEME = scheme`). + Falls back to system `scheme` if `$(CHEZ_BIN)` doesn't exist *and* + `JERBOA_USE_SYSTEM_CHEZ=1` is set. Default behavior: require bundled. +- **`bin/jerboa`**: same logic — prefer `$JERBOA_HOME/build/chez/bin/scheme`, + fall back to `$SCHEME` env, fall back to system. +- **`support/build-binary.sh`**: check `$JERBOA_HOME/build/chez/lib/csv*/` + *before* globbing system prefixes. Already structured for this. +- **`make binary`** depends on `chez` so bootstrap is automatic. + +**Acceptance**: +- `git clone jerboa && make binary` works on a machine with no Chez + installed (only a C compiler + musl-gcc on Linux for static). +- `make test` runs against bundled Chez. +- `JERBOA_USE_SYSTEM_CHEZ=1 make test` runs against system Chez (escape + hatch for Jerboa-on-Jerboa development, where the bundled Chez might + be the thing being tested). + +**Effort**: 2–3 days. + +### Stage 3 — `--static` everywhere + +Generalize `musl-chez-build.sh` so `--static` builds work on all platforms, +not just musl/Linux. On macOS this means a Chez built without dynamic +loading (limits FFI, but produces truly hermetic binaries). Make this an +opt-in mode (`make chez STATIC=1`). + +**Acceptance**: +- `make chez STATIC=1` produces a non-dlopen-capable Chez on macOS +- `make binary STATIC=1` links against it +- `file jerboa-bin` reports "statically linked" where the platform allows + (Linux/musl always; macOS partial — Mach-O semi-static) + +**Effort**: 3–5 days. Mostly testing across platforms. + +### Stage 4 — distribution story + +Once the above works, decide how Jerboa is shipped: + +- **Tarball with vendored Chez source**: `jerboa-X.Y.Z.tar.gz` includes + `vendor/ChezScheme/`. User runs `./configure && make`. Same as Gerbil. +- **Tarball with pre-built Chez per arch**: heavier downloads, but no C + toolchain needed for Jerboa-only changes during dev. +- **Both**: source tarball + pre-built tarballs for common arches. + +This is a release-engineering decision, not a code one. Open question +below. + +## Risks and non-goals + +### Risks + +**Chez version drift.** Vendoring pins a Chez version. Bug fixes +upstream don't propagate until we re-pin. Mitigation: track upstream +tags in CI; re-pin quarterly or on critical fixes. + +**Submodule UX.** `git submodule` is notoriously confusing. Forgetting +`--recursive` on clone, or missing `git submodule update` after a pull, +produces cryptic build errors. Mitigation: `make` target checks the +submodule and prints a clear error. + +**Build-time inflation.** First `make` will now compile Chez (~5 min on +modern hardware). Mitigation: ccache; check `$(CHEZ_BIN)` existence +before re-running; document the expected time. + +**Cross-compilation complexity.** A bundled Chez means we own +cross-compilation of Chez, not just Jerboa. The musl pipeline already +does this; extending to other targets is non-trivial. + +### Non-goals + +- **Forking Chez.** We track upstream. Any patches go through + `vendor/ChezScheme-patches/*.patch` applied during `make chez`. No + long-lived fork. +- **Removing the system-Chez option.** `JERBOA_USE_SYSTEM_CHEZ=1` stays + forever, both for distros that already package Chez and for developers + iterating on Chez itself. +- **Embedding Chez as a `.dylib`/`.so` (Gerbil-style).** Chez's static + linking model is already cleaner than Gambit's dynamic one. We bundle + the *source* and build it; we don't ship a precompiled shared lib that + callers link against. (The C embedding API in `jerboa-embed.h` is a + different thing — that's Jerboa-as-library, not Chez-as-library.) +- **Replacing Chez.** This plan keeps Chez. Replacing it (e.g., with the + WASM backend or a Jerboa-hosted compiler) is a separate question. + +## License considerations + +Chez Scheme is currently Apache 2.0 (since the Cisco open-sourcing in +2016). Vendoring is permitted with attribution. Action items: + +- Verify the current Chez license at the pinned tag +- Add `LICENSE-CHEZ` at the Jerboa repo root +- Add an attribution line to the README and to `--version` output of + `jerboa-bin` +- Confirm any third-party dependencies of Chez itself (lz4, zlib) are + compatibly licensed for redistribution + +This needs a careful read before Stage 0 lands; license compliance is +not theoretical. + +## Open questions + +1. **Submodule vs. vendored snapshot.** Submodules give a clean upstream + relationship; vendored snapshots give offline-buildability and full + control. Gerbil chose the bundle-everything route. Recommendation: + start with submodule, switch to vendored snapshot if submodule UX is + too painful. + +2. **Build time tradeoff.** Building Chez adds ~5 minutes to the first + build. Acceptable for `make binary`, painful for `make test` in CI. + Should CI cache `build/chez/` aggressively? Probably yes, but the + cache key needs to include the Chez source hash, not just the Jerboa + commit. + +3. **`docker-build` interaction.** The Docker pipeline already builds + Chez from source inside the image. Stage 2 needs to either reuse + that build or rebuild — neither is free. Probably: Docker uses its + own Chez, native builds use `vendor/ChezScheme/`. Two paths, kept + consistent by pinning the same Chez tag. + +4. **What about `jerboa-native-rs/`?** The Rust backend currently + coexists with the Chez one. Is the bundled-Chez plan orthogonal + (Rust stays separate) or does the Rust backend eventually replace + Chez? Probably orthogonal for now; revisit when the Rust backend is + more mature. + +5. **Cross-arch story.** A user on x86_64 running `make binary` produces + an x86_64 binary. What about cross-builds (x86_64 host → + aarch64-linux target)? Chez supports cross-compilation but it's + fiddly. Defer until needed. + +## Relationship to `jerboa-on-jerboa.md` + +These two plans are **orthogonal**: + +- `jerboa-on-jerboa.md`: rewrite the *source* of the stdlib and tooling + in idiomatic Jerboa `.ss`. Doesn't change what runtime ships. +- `bundling-chez.md` (this doc): bundle the Chez *runtime* into the + Jerboa repo. Doesn't change what language the source is in. + +You can do either independently. Doing both produces a Jerboa where the +language is idiomatic Jerboa *and* the runtime is shipped with no +external dependencies — that's the "real language" milestone. + +## How to start + +If picking this up cold, the smallest useful first step is: + +1. `git submodule add https://github.com/cisco/ChezScheme vendor/ChezScheme` +2. Pin to the latest stable tag (e.g., `git checkout v10.4.0` inside the + submodule). +3. Write a minimal `make chez` target that builds it into `build/chez/`. +4. Verify `build/chez/bin/scheme -q -e '(display (machine-type))'` works. +5. Open a PR. + +That gets us to "Jerboa can be built without a system Chez install" on +the contributor's machine, which is the single highest-value step. The +rest is wiring.