docs: fully reconcile bundling-chez.md with reality
ober
d8bdce1eaffc3080f08d3b0d64baa4204925bbd0
--- a/docs/bundling-chez.md +++ b/docs/bundling-chez.md @@ -1,55 +1,89 @@ # Bundling Chez -> **Status note:** Chez is now vendored in-tree at `vendor/ChezScheme/` and the -> `Makefile` builds it from there, so the "plan to vendor Chez" framing below is -> partly overtaken by events — treat the "Today" / "gap" sections as historical. -> The musl build scripts now default to `vendor/ChezScheme/`. See -> [`chez-fork.md`](chez-fork.md) and [`Chez-changes.md`](Chez-changes.md). - -A plan to make Jerboa self-contained — no system Chez Scheme required on -the build or development machine. The model is Gerbil's recent shift from -"Gambit must be installed" to "Gambit ships with Gerbil." - -## Thesis - -Today, Jerboa is split-bundled: - -- The **shipped binary** (`make binary` output) is fully self-contained. - `petite.boot`, `scheme.boot`, the Chez kernel (`libkernel.a`), and a - WPO-compiled entry program are linked into one ELF/Mach-O file. End - users need no Chez install. -- The **build** is not self-contained. `support/build-binary.sh` globs - `/usr/local/lib /usr/lib /usr/lib64 /opt/homebrew/lib /opt/local/lib` - for `csv*/<machine-type>/` and fails if Chez isn't installed. -- The **dev driver** (`bin/jerboa`) calls `${SCHEME:-scheme}` — needs - system Chez to run. -- The **musl static pipeline** (`support/musl-chez-build.sh`) builds Chez - from source, defaulting to the in-tree `vendor/ChezScheme/` (override with - an explicit path arg). - -The goal: close the gap so `git clone jerboa && make` produces a working -Jerboa with no system Chez anywhere. - -## What's already in place - -A surprising amount. The C side of embedding is largely solved. - -### `support/build-binary.sh` (181 lines) - -The shipped-binary pipeline. Already does the hardest part: +Jerboa is self-contained: `git clone && make binary` produces a working Jerboa +with **no system Chez Scheme** required. Chez is vendored in-tree and built as +part of the normal build. This doc records how that works and what's left. + +> The model mirrors Gerbil's shift from "Gambit must be installed" to "Gambit +> ships with Gerbil." Jerboa took the **vendored-snapshot** route (not a +> submodule) — see [`chez-fork.md`](chez-fork.md) and +> [`../vendor/ChezScheme/UPSTREAM.md`](../vendor/ChezScheme/UPSTREAM.md). + +## Status + +| Area | State | +| --- | --- | +| Vendor Chez source | **Done** — flat snapshot at `vendor/ChezScheme/` (cisco base `4bde3b35`) | +| `make chez` builds it | **Done** — installs to `./.chez/`, no `sudo`, no system Chez | +| Wired into the build | **Done** — `build:`/`binary:` depend on `chez`; `SCHEME` defaults to `.chez/bin/scheme` | +| Cross-compilation | **Done** — `make chez-cross` for Linux/FreeBSD/macOS-x86 targets | +| Docker | **Done** — image builds the same `vendor/ChezScheme` (glibc + a `--static` musl variant) | +| `--static` musl | **Done** — `support/musl-chez-build*.sh`, Docker musl stage | +| `--static` everywhere | **Partial** — a hermetic-build knob beyond musl/Linux isn't generalized | +| Licensing / attribution | **Open** — no root `LICENSE-CHEZ`, no README / `--version` attribution yet | +| Distribution story | **Open** — release-engineering decision | + +## How it works today + +`make binary` (target deps `chez build`) bootstraps everything — no `scheme` on +`$PATH` needed: + +1. **`make chez`** builds `vendor/ChezScheme/` into `./.chez/`: + ``` + configure --pb --as-is # bootstrap kernel + make bootquick XM=<machine> # native boot files + configure --installprefix=$(CHEZ_PREFIX) … # CHEZ_PREFIX = ./.chez + make && make install + ``` + Scratch build dir: `build/chez/`. Install prefix: `./.chez/` (`CHEZ_PREFIX`). + +2. **`SCHEME ?= $(CHEZ_PREFIX)/bin/scheme`** — every Make target, and the + `bin/jerboa` dev driver (`SCHEME="${SCHEME:-$JERBOA_HOME/.chez/bin/scheme}"`), + default to the in-tree Chez. + +3. **`support/build-binary.sh`** searches `$JERBOA_CHEZ_PREFIX/lib/csv*` + (default `./.chez/lib`) **before** the system prefixes, then WPO-compiles, + embeds boot files as C arrays, generates `main.c`, and links the kernel into + one self-contained `jerboa-bin` (Mach-O / ELF; static-linkable on Linux/musl). + +**Using a system or custom Chez** is an override, not a flag: set +`SCHEME=/path/to/scheme` (Make targets + `bin/jerboa`) and +`JERBOA_CHEZ_PREFIX=/usr` (`build-binary.sh`). The old plan's +`JERBOA_USE_SYSTEM_CHEZ=1` switch was never added — these overrides cover it. + +### Cross-compilation + +`make chez-cross CHEZ_TARGET_MACHINE=<m>` builds target boot files, +`libkernel.a`, and an `xpatch` into `./.chez-cross-<m>/` + `build/chez-cross-<m>/`; +`make jerboa-cross` / `binary-cross` then emit a foreign-target binary. Wired +targets include `ta6le`/`tarm64le` (Linux), `ta6fb` (FreeBSD), and `ta6osx` +(macOS x86_64). This resolved the original "cross-arch" open question, though +each target still needs its cross-CC plumbed (`CROSS_CC=…`). + +### Docker + +`Dockerfile` copies `vendor/ChezScheme` into the build context and builds it +twice: a glibc Chez at `/usr/local` (`--threads --enable-harden --disable-x11`) +and a `--static CC=musl-gcc` variant for fully static binaries. Native builds +and Docker share one Chez source. + +## What's already in place (C embedding) + +The shipped-binary pipeline solves the hard part of embedding. + +### `support/build-binary.sh` ``` [1/4] WPO compile → entry.wp.so -[2/4] Embed boots → petite_boot.h, scheme_boot.h, program_boot.h (as C arrays) -[3/4] Generate main.c → calls Sscheme_init / Sregister_boot_file_bytes / Sscheme_program +[2/4] Embed boots → petite_boot.h, scheme_boot.h, program_boot.h (C arrays) +[3/4] Generate main.c → Sscheme_init / Sregister_boot_file_bytes / Sscheme_program [4/4] Link → $CC main.c libkernel.a [liblz4.a libz.a] $OS_LIBS ``` -Cross-platform: FreeBSD/Darwin/Linux branches for compiler and link flags. -Output: a 4.2M Mach-O on macOS arm64, ELF on Linux. Static-linkable on -Linux/musl. +Cross-platform (FreeBSD/Darwin/Linux branches). Output: ~4.2M Mach-O on macOS +arm64, ELF on Linux, static-linkable on Linux/musl. -### `support/jerboa-embed.{c,h}` (459 lines total) +### `support/jerboa-embed.{c,h}` C ABI for embedding Jerboa as a library: @@ -60,257 +94,69 @@ int64_t v = jerboa_get_int(j, "x"); jerboa_destroy(j); ``` -Functions: `jerboa_new/destroy`, `jerboa_eval[_safe]`, `jerboa_get_{int, -double,string,bool}`, `jerboa_call_{int,string}`, `jerboa_error_free`. -Configurable boot file path and lib dirs. Already supports multiple -independent instances. - -### `support/musl-chez-build.sh` - -Builds Chez from source against musl libc using Chez's native `--static` -flag (v10.4.0+): - -- Adds `-static` to LDFLAGS -- Disables dynamic loading (no dlopen) -- Disables curses/x11/iconv -- Embeds boot files via `static_boot_init()` -- Provides `main.o` for downstream static builds - -Defaults to the in-tree `vendor/ChezScheme/`. Installs to `/opt/chez-musl`. - -## The gap - -What separates "today" from "Gerbil-style fully bundled": - -1. **Chez source isn't in the repo.** Both the dev workflow and the musl - pipeline assume Chez exists somewhere else. Closing this gap means - either (a) git-submodule the canonical Chez source, or (b) vendor a - trimmed snapshot. - -2. **`make` doesn't build Chez.** A user with no system Chez can't run - `make binary` today — step 4 needs `libkernel.a`. We need a `make - bootstrap` (or `make chez`) target that builds Chez before anything - else, and downstream targets that depend on it. - -3. **`bin/jerboa` calls system `scheme`.** The dev driver assumes - `scheme` is on `$PATH`. After bootstrap, it should point at the - built-from-source binary unless overridden. - -4. **`--static` is musl-only in current usage.** Chez's `--static` flag - works on glibc/macOS too, but `musl-chez-build.sh` is the only script - exercising it. The general bundling path should generalize. - -5. **Tests assume system Chez.** Every test target in the Makefile is - `$(SCHEME) --libdirs $(LIBDIRS) --script tests/...`. After bootstrap, - `$(SCHEME)` should default to the bundled `scheme`. - -## Plan - -### Stage 0 — vendor Chez - -Add Chez Scheme as a git submodule at `vendor/ChezScheme/` (or pin a -specific tag and vendor as a tarball — see "Open questions" below). -Pin to a known-good Chez version (current target: v10.4.0+ for `--static` -support). - -**Acceptance**: -- `git clone --recursive jerboa` (or `make vendor`) gets the Chez source -- `vendor/ChezScheme/` is a valid Chez source tree -- License files are preserved alongside the source -- A `LICENSE-CHEZ` (or similar) is added at the Jerboa root for - attribution - -**Effort**: half a day. - -### Stage 1 — `make chez` target - -Add a `make chez` target that builds Chez from `vendor/ChezScheme/` and -installs into `build/chez/` (in-tree, not `/opt/`). Should work on -Darwin/Linux/FreeBSD without `sudo`. - -Sketch: -```makefile -CHEZ_BUILD_DIR := $(CURDIR)/build/chez -CHEZ_BIN := $(CHEZ_BUILD_DIR)/bin/scheme -CHEZ_CSV_DIR := $(CHEZ_BUILD_DIR)/lib/csv*/$(MACHINE_TYPE) - -chez: $(CHEZ_BIN) - -$(CHEZ_BIN): - cd vendor/ChezScheme && \ - ./configure --installprefix=$(CHEZ_BUILD_DIR) --threads && \ - $(MAKE) -j$(JOBS) && \ - $(MAKE) install -``` - -**Acceptance**: -- `make chez` works on macOS/Linux/FreeBSD with no `sudo`, no system - Chez install -- Builds reproducibly (same Chez tag → byte-identical kernel) -- Cleaned by `make clean` (without nuking the source submodule) -- A separate `make chez-clean` for full Chez rebuild - -**Effort**: 1–2 days, mostly bashing through configure-script quirks per -platform. - -### Stage 2 — wire bundled Chez into everything - -Switch `SCHEME` and the CSV-lookup logic to prefer the in-tree build: - -- **`Makefile`**: `SCHEME ?= $(CHEZ_BIN)` (was `SCHEME = scheme`). - Falls back to system `scheme` if `$(CHEZ_BIN)` doesn't exist *and* - `JERBOA_USE_SYSTEM_CHEZ=1` is set. Default behavior: require bundled. -- **`bin/jerboa`**: same logic — prefer `$JERBOA_HOME/build/chez/bin/scheme`, - fall back to `$SCHEME` env, fall back to system. -- **`support/build-binary.sh`**: check `$JERBOA_HOME/build/chez/lib/csv*/` - *before* globbing system prefixes. Already structured for this. -- **`make binary`** depends on `chez` so bootstrap is automatic. +`jerboa_new/destroy`, `jerboa_eval[_safe]`, `jerboa_get_{int,double,string,bool}`, +`jerboa_call_{int,string}`, `jerboa_error_free`. Configurable boot path + lib +dirs; supports multiple independent instances. -**Acceptance**: -- `git clone jerboa && make binary` works on a machine with no Chez - installed (only a C compiler + musl-gcc on Linux for static). -- `make test` runs against bundled Chez. -- `JERBOA_USE_SYSTEM_CHEZ=1 make test` runs against system Chez (escape - hatch for Jerboa-on-Jerboa development, where the bundled Chez might - be the thing being tested). +## Remaining work -**Effort**: 2–3 days. +1. **Licensing / attribution (open).** `vendor/ChezScheme/LICENSE` + `NOTICE` + (Apache 2.0) ship with the snapshot, but there is still no root-level + `LICENSE-CHEZ`, no Chez attribution line in the README, and none in + `jerboa-bin --version`. Also confirm lz4/zlib (vendored under + `vendor/ChezScheme/`) are cleared for redistribution. Compliance, not + theory — settle it before any tagged release. -### Stage 3 — `--static` everywhere +2. **Generalize `--static` (partial).** Static, hermetic Chez is wired for + musl/Linux (`musl-chez-build*.sh`, the Docker musl stage). A single + cross-platform knob (e.g. `make chez STATIC=1`, including a non-dlopen macOS + build) doesn't exist yet. -Generalize `musl-chez-build.sh` so `--static` builds work on all platforms, -not just musl/Linux. On macOS this means a Chez built without dynamic -loading (limits FFI, but produces truly hermetic binaries). Make this an -opt-in mode (`make chez STATIC=1`). - -**Acceptance**: -- `make chez STATIC=1` produces a non-dlopen-capable Chez on macOS -- `make binary STATIC=1` links against it -- `file jerboa-bin` reports "statically linked" where the platform allows - (Linux/musl always; macOS partial — Mach-O semi-static) - -**Effort**: 3–5 days. Mostly testing across platforms. - -### Stage 4 — distribution story - -Once the above works, decide how Jerboa is shipped: - -- **Tarball with vendored Chez source**: `jerboa-X.Y.Z.tar.gz` includes - `vendor/ChezScheme/`. User runs `./configure && make`. Same as Gerbil. -- **Tarball with pre-built Chez per arch**: heavier downloads, but no C - toolchain needed for Jerboa-only changes during dev. -- **Both**: source tarball + pre-built tarballs for common arches. - -This is a release-engineering decision, not a code one. Open question -below. +3. **Distribution story (open).** Decide what ships: a source tarball with + `vendor/ChezScheme/` (à la Gerbil), pre-built per-arch tarballs, or both. + Release-engineering, not code. ## Risks and non-goals ### Risks -**Chez version drift.** Vendoring pins a Chez version. Bug fixes -upstream don't propagate until we re-pin. Mitigation: track upstream -tags in CI; re-pin quarterly or on critical fixes. +- **Chez version drift.** The vendored snapshot pins a Chez base; upstream cisco + fixes don't propagate until re-vendored. Resync recipe in + `vendor/ChezScheme/UPSTREAM.md`; track cisco tags in CI. +- **Build-time inflation.** The first `make` compiles Chez (~5 min). Mitigated + because `./.chez/` persists — later builds skip it when present. CI should + cache `.chez/`, keyed on the vendored Chez hash. +- **Cross-compilation complexity.** Owning a bundled Chez means owning its + cross-compilation; largely handled by `make chez-cross`, but each target needs + its cross-CC. -**Submodule UX.** `git submodule` is notoriously confusing. Forgetting -`--recursive` on clone, or missing `git submodule update` after a pull, -produces cryptic build errors. Mitigation: `make` target checks the -submodule and prints a clear error. - -**Build-time inflation.** First `make` will now compile Chez (~5 min on -modern hardware). Mitigation: ccache; check `$(CHEZ_BIN)` existence -before re-running; document the expected time. - -**Cross-compilation complexity.** A bundled Chez means we own -cross-compilation of Chez, not just Jerboa. The musl pipeline already -does this; extending to other targets is non-trivial. +(The original "submodule UX" risk is gone — Jerboa vendors a flat snapshot, not +a submodule.) ### Non-goals -- **Forking Chez.** We track upstream. Any patches go through - `vendor/ChezScheme-patches/*.patch` applied during `make chez`. No - long-lived fork. -- **Removing the system-Chez option.** `JERBOA_USE_SYSTEM_CHEZ=1` stays - forever, both for distros that already package Chez and for developers - iterating on Chez itself. -- **Embedding Chez as a `.dylib`/`.so` (Gerbil-style).** Chez's static - linking model is already cleaner than Gambit's dynamic one. We bundle - the *source* and build it; we don't ship a precompiled shared lib that - callers link against. (The C embedding API in `jerboa-embed.h` is a - different thing — that's Jerboa-as-library, not Chez-as-library.) -- **Replacing Chez.** This plan keeps Chez. Replacing it (e.g., with the - WASM backend or a Jerboa-hosted compiler) is a separate question. - -## License considerations - -Chez Scheme is currently Apache 2.0 (since the Cisco open-sourcing in -2016). Vendoring is permitted with attribution. Action items: - -- Verify the current Chez license at the pinned tag -- Add `LICENSE-CHEZ` at the Jerboa repo root -- Add an attribution line to the README and to `--version` output of - `jerboa-bin` -- Confirm any third-party dependencies of Chez itself (lz4, zlib) are - compatibly licensed for redistribution - -This needs a careful read before Stage 0 lands; license compliance is -not theoretical. - -## Open questions - -1. **Submodule vs. vendored snapshot.** Submodules give a clean upstream - relationship; vendored snapshots give offline-buildability and full - control. Gerbil chose the bundle-everything route. Recommendation: - start with submodule, switch to vendored snapshot if submodule UX is - too painful. - -2. **Build time tradeoff.** Building Chez adds ~5 minutes to the first - build. Acceptable for `make binary`, painful for `make test` in CI. - Should CI cache `build/chez/` aggressively? Probably yes, but the - cache key needs to include the Chez source hash, not just the Jerboa - commit. - -3. **`docker-build` interaction.** The Docker pipeline already builds - Chez from source inside the image. Stage 2 needs to either reuse - that build or rebuild — neither is free. Probably: Docker uses its - own Chez, native builds use `vendor/ChezScheme/`. Two paths, kept - consistent by pinning the same Chez tag. - -4. **What about `jerboa-native-rs/`?** The Rust backend currently - coexists with the Chez one. Is the bundled-Chez plan orthogonal - (Rust stays separate) or does the Rust backend eventually replace - Chez? Probably orthogonal for now; revisit when the Rust backend is - more mature. - -5. **Cross-arch story.** A user on x86_64 running `make binary` produces - an x86_64 binary. What about cross-builds (x86_64 host → - aarch64-linux target)? Chez supports cross-compilation but it's - fiddly. Defer until needed. +- **A long-lived *external* fork.** Jerboa does maintain a small in-tree fork of + Chez (`vendor/ChezScheme/`; divergence recorded in + `vendor/ChezScheme-patches/`), but there is no separate upstream fork repo, and + the patch series is a *record*, not applied at build time — the vendored tree + already contains the changes. See [`chez-fork.md`](chez-fork.md). +- **Removing the override to a system/custom Chez.** `SCHEME=` and + `JERBOA_CHEZ_PREFIX=` stay — for distros that package Chez, and for + Jerboa-on-Jerboa work where the bundled Chez is itself under test. +- **Embedding Chez as a `.dylib`/`.so` (Gerbil-style).** We bundle the *source* + and build it; we don't ship a precompiled shared lib. (`jerboa-embed.h` is + Jerboa-as-library — a different thing.) +- **Replacing Chez.** Out of scope here (a WASM backend or Jerboa-hosted + compiler is a separate question; cf. `jerboa-native-rs/`). ## Relationship to `jerboa-on-jerboa.md` -These two plans are **orthogonal**: - -- `jerboa-on-jerboa.md`: rewrite the *source* of the stdlib and tooling - in idiomatic Jerboa `.ss`. Doesn't change what runtime ships. -- `bundling-chez.md` (this doc): bundle the Chez *runtime* into the - Jerboa repo. Doesn't change what language the source is in. - -You can do either independently. Doing both produces a Jerboa where the -language is idiomatic Jerboa *and* the runtime is shipped with no -external dependencies — that's the "real language" milestone. - -## How to start - -If picking this up cold, the smallest useful first step is: +Orthogonal: -1. `git submodule add https://github.com/cisco/ChezScheme vendor/ChezScheme` -2. Pin to the latest stable tag (e.g., `git checkout v10.4.0` inside the - submodule). -3. Write a minimal `make chez` target that builds it into `build/chez/`. -4. Verify `build/chez/bin/scheme -q -e '(display (machine-type))'` works. -5. Open a PR. +- `jerboa-on-jerboa.md` — rewrite the stdlib/tooling *source* in idiomatic + Jerboa `.ss`. Doesn't change what runtime ships. +- `bundling-chez.md` (this doc) — bundle the Chez *runtime* into the repo. + Doesn't change what language the source is in. -That gets us to "Jerboa can be built without a system Chez install" on -the contributor's machine, which is the single highest-value step. The -rest is wiring. +Both done → the language is idiomatic Jerboa *and* the runtime ships with no +external dependency.