updates
ober
a1dadf00146b78b9b1de6688ce9ec2883769aa3a
--- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,6 @@ # Jerboa Documentation -Updated 2026-05-23. +Updated 2026-06-04. All documents live flat in `docs/`. Historical planning notes are under [`archive/`](archive/). The full generated symbol reference is @@ -85,6 +85,7 @@ New here? Start with [quickstart.md](quickstart.md), then [tutorial.md](tutorial - [static-binary-gotchas.md](static-binary-gotchas.md) — real bugs and guard patterns in musl static builds - [compiling-gerbil-projects.md](compiling-gerbil-projects.md) — Gerbil→Chez translation and compatibility shims - [packages.md](packages.md) — semver package manager +- [jpkg-plan.md](jpkg-plan.md) — secure-by-default package manager plan - [single-file-packages.md](single-file-packages.md) — self-contained script format with dependencies - [optimization.md](optimization.md) — Chez compiler tuning (WPO, `cp0` parameters) - [cp0-passes.md](cp0-passes.md) — user-defined `cp0` optimization passes new file mode 100644 --- /dev/null +++ b/docs/jpkg-plan.md @@ -0,0 +1,539 @@ +# jpkg Package Manager Plan + +## Purpose + +`jpkg` is the Jerboa package manager. It should be installed as a symlink or +hardlink to the `jerboa` binary and selected by multicall dispatch, with +`jerboa pkg ...` kept as an equivalent spelling. + +The goal is not "npm for Jerboa". The goal is a secure-by-default package +system with a small trusted base, reproducible artifacts, signed metadata, +explicit policy, and no implicit package code execution during install. + +## Design Goals + +- One toolchain binary: `jerboa`, `jpkg`, and `jerboa pkg` share code and state. +- Keep the productive parts of Gerbil `gxpkg`: project templates, dependency + management, local linking, build/clean, search, package directories, list, + retag/reindex, and package-scoped command environments. +- Prefer verified immutable artifacts over cloning arbitrary Git repositories. +- Make package install a data operation: resolve, fetch, verify, extract, link. +- Treat signatures as necessary but insufficient. Validation must also cover + manifests, archives, provenance, policy, advisory status, and build behavior. +- Keep official registry metadata mirrorable and cacheable. +- Support offline and self-hosted registries without weakening defaults for the + official registry. + +## Non-Goals + +- No arbitrary lifecycle scripts on install. +- No mutable package versions. +- No unsigned packages from the public registry. +- No hidden network access during builds. +- No global trust in all maintainers of all transitive dependencies. +- No attempt to prove that signed code is non-malicious. `jpkg` should reduce + ambiguity and blast radius, not pretend authorship is equivalent to safety. + +## Threat Model + +`jpkg` should defend against: + +- Compromised or malicious mirrors. +- Stale, rollback, or freeze attacks against registry metadata. +- Artifact substitution after dependency resolution. +- Package archive path traversal, symlink, hardlink, device-file, and permission + attacks. +- Typosquatting and dependency confusion across registries. +- Stolen long-lived publish tokens. +- Accidental installation of native/FFI code or build-time network behavior. +- Maintainer account compromise, where possible, through provenance, + transparency, threshold ownership for critical packages, and revocation/yank + mechanisms. + +`jpkg` cannot fully defend against a trusted maintainer intentionally publishing +malicious source. It can make that action attributable, auditable, harder to +hide, easier to constrain, and easier to revoke from new resolutions. + +## External Standards To Reuse + +- TUF for registry metadata roles, threshold signatures, key rotation, mirrors, + rollback protection, and freeze protection. +- Sigstore for keyless package signing tied to OIDC identity and transparency + logging. +- in-toto attestations and SLSA provenance for source, commit, build workflow, + builder identity, and materials. +- SPDX or CycloneDX SBOM attestations for dependency and license inventory. +- OSV format for vulnerability and advisory data. +- OpenSSF Scorecard-style repository checks as advisory signals, not default + hard gates. + +## Command Surface + +The initial command set should preserve the useful `gxpkg` workflows while +adding verification-first operations: + +```text +jpkg init create jpkg.sexp for the current project +jpkg new NAME create a new package template +jpkg add PKG[@VERSION] add dependency and update lockfile +jpkg remove PKG remove dependency and update lockfile +jpkg install install exactly what jpkg.lock describes +jpkg update [PKG ...] resolve newer allowed versions +jpkg uninstall PKG remove from the project environment +jpkg link PKG PATH link a local development checkout +jpkg unlink PKG remove local development link +jpkg build [PKG ...] build under policy-controlled sandbox +jpkg clean [PKG ...] remove build outputs +jpkg pack create deterministic local .jpkg artifact +jpkg verify verify manifest, lock, artifacts, signatures +jpkg audit check advisories, yanks, policy drift +jpkg publish sign, attest, and publish an artifact +jpkg search QUERY ... search configured package directories +jpkg dir add|remove|list manage registry/package-directory list +jpkg list list installed packages +jpkg env -- COMMAND ... run command with package environment +jpkg policy inspect or explain active policy +``` + +`jerboa new`, `jerboa build`, `jerboa deps`, and similar aliases can continue to +dispatch to `jpkg` for compatibility with the `gxpkg` mental model. + +## Project Files + +### `jpkg.sexp` + +The source manifest should be a Jerboa-readable data file, not executable code. +Recommended shape: + +```scheme +(package + (name "@scope/name") + (version "0.1.0") + (description "short text") + (license "Apache-2.0") + (source "https://github.com/scope/name") + (jerboa ">=0.1.0") + (modules ((root "src") (exports ((name main) (name util))))) + (dependencies + (("@scope/dep" "^1.2.0"))) + (dev-dependencies ()) + (capabilities ())) +``` + +The manifest parser must reject unknown fields by default, reject duplicate +fields, normalize names and versions, and produce a canonical JSON +representation for signing, registry metadata, and lockfile inclusion. + +### `jpkg.lock` + +The lockfile is the project security boundary. It records the exact resolved +graph: + +- package name and version +- registry identity and TUF root identity +- artifact digest and size +- manifest digest +- source repository and commit +- Sigstore bundle or signature reference +- in-toto/SLSA provenance digest +- SBOM digest, when present +- yanked/advisory status at resolution time +- local policy exceptions, with reason and timestamp + +`jpkg install` must install exactly the lockfile graph. It should not silently +resolve new versions. + +## Package Identity + +Use scoped package names: + +```text +@scope/name +@scope/collection.subpackage +``` + +The official registry owns scope assignment. A self-hosted registry may define +its own scopes, but dependency resolution must prevent dependency confusion: +once a package name is resolved from a registry in a lockfile, the same name +cannot be substituted from a different registry without an explicit lock update. + +Module imports remain Jerboa module imports. Package names identify +distribution units, ownership, and dependency resolution; module names identify +source-level imports. + +## Artifact Format + +Use `.jpkg` for immutable package artifacts. Recommended underlying format: + +- deterministic `tar.zst` once Jerboa has solid zstd support, or + deterministic `tar.gz` initially +- sorted paths +- normalized uid/gid, uname/gname, mtimes, and modes +- UTF-8 paths only +- regular files and directories only by default +- no absolute paths +- no `..` components +- no symlinks, hardlinks, device files, sockets, fifos, extended attributes, or + setuid/setgid bits +- maximum file size, file count, path length, and expanded archive size limits +- mandatory embedded normalized manifest + +The signed subject is the artifact digest, size, normalized manifest digest, +and package identity. Do not sign loosely interpreted source trees. + +## Local State + +Suggested state layout: + +```text +~/.jerboa/pkg/ + roots/ trusted TUF roots by registry + registries/ cached metadata and registry config + store/sha256/.. content-addressed verified artifacts + src/ optional unpacked source cache + envs/ project or global package environments + links/ local development links + advisories/ cached OSV/advisory data +``` + +Project environments should be cheap views over the content-addressed store, +not mutable copies of packages. + +## Registry Model + +Start with a static, mirrorable registry. Dynamic APIs can be added later for +search and publishing. + +```text +metadata/root.json +metadata/timestamp.json +metadata/snapshot.json +metadata/targets.json +metadata/delegations/@scope.json +packages/@scope/name/1.2.3/release.json +blobs/sha256/<digest> +advisories/osv/*.json +``` + +TUF roles: + +- `root`: offline keys, threshold signed, rotates all other keys +- `targets`: package release metadata and delegated namespaces +- `snapshot`: consistent view of target metadata +- `timestamp`: short-lived freshness marker +- delegated scope roles: per-scope or per-organization publish authority + +Mirrors may be untrusted. The client trusts TUF metadata and verified artifact +digests, not HTTPS alone. + +## Publishing + +Official registry publishing should use trusted publishing by default: + +- Maintainer triggers a release from an approved source repository workflow. +- CI builds the deterministic `.jpkg` artifact. +- CI emits SLSA provenance and SBOM attestations. +- The artifact is signed with Sigstore keyless signing. +- The registry verifies OIDC identity, package ownership, artifact digest, + provenance subject, and policy. +- The registry adds signed TUF target metadata. +- The registry appends publish data to a transparency log or publishes enough + data for independent monitoring. + +Self-hosted and offline registries may use Ed25519 keys instead of Sigstore, +but the client policy must record that trust root distinctly. + +## Install Verification Pipeline + +`jpkg install` should run these checks in order: + +1. Load project policy and trusted registry roots. +2. Refresh TUF metadata with rollback and freeze protection. +3. Resolve or read the exact lockfile graph. +4. Fetch artifacts by digest. +5. Check digest and size before opening archives. +6. Validate normalized manifest against registry target metadata. +7. Verify package signature or Sigstore bundle. +8. Verify provenance subject matches artifact digest. +9. Verify provenance identity matches package policy. +10. Verify SBOM/advisory/yank status against active policy. +11. Validate archive structure before extraction. +12. Extract into a staging directory with safe permissions. +13. Move into the content-addressed store atomically. +14. Link the project environment. + +Any failed check stops installation. Recovery should leave no partially trusted +package in the store. + +## Build Security + +Installation must not run package code. Building is a separate operation with +explicit policy. + +Default build sandbox: + +- read-only package sources and dependencies +- writable build directory only +- no network +- no ambient access to `$HOME` +- no inherited secrets except explicit allowlist +- no native/FFI compilation unless the package declares that capability and the + project policy allows it +- deterministic environment variables +- fixed toolchain identity + +Platform backends: + +- Linux: Landlock plus process isolation where available +- macOS: sandbox-exec/Seatbelt profile where available +- FreeBSD: Capsicum where feasible +- fallback: clear warning and stricter policy for operations that cannot be + sandboxed + +## Capability Declarations + +Packages that need sharp edges must declare them: + +```scheme +(capabilities + (native-code reason: "sqlite extension") + (network build: #f test: #t) + (ffi libraries: ("sqlite3")) + (executables ("bin/tool"))) +``` + +The default policy rejects undeclared capabilities and rejects declared native +or network capabilities unless the project explicitly permits them. + +## Policy Modes + +Recommended built-in modes: + +- `default`: official registry, TUF verified, signed release, provenance + present, no install scripts, no native build, no build network. +- `strict`: default plus source/workflow identity pinning, recursive provenance + checks, no local/git dependencies, no policy exceptions. +- `dev`: permits local links and path dependencies, records them as + non-reproducible in the lockfile. +- `offline`: requires already cached metadata/artifacts and valid freshness or + explicit offline override. +- `unsafe`: explicit command-line opt-in only; records exceptions in the + lockfile and never becomes a default. + +## Dependency Resolution + +Use deterministic resolution with: + +- semantic versions for compatibility constraints +- immutable selected versions in the lockfile +- registry-scoped package identity +- yanked versions excluded from new resolutions unless explicitly pinned +- no implicit pre-release selection +- conflict explanations that show the dependency chain + +PubGrub-style conflict reporting is a good target because it gives useful +human-readable explanations when constraints cannot be satisfied. + +## Advisories And Yanking + +Yanking means "do not select for new resolutions"; it must not remove existing +artifacts or break reproducibility. + +Advisories should use OSV-shaped records and support: + +- affected package/version ranges +- severity +- patched versions +- aliases/CVEs/GHSA identifiers +- policy action: warn, block new installs, or block all installs + +`jpkg audit` checks the current lockfile against the latest advisory metadata. + +## Search And Package Directories + +Keep the `gxpkg dir` idea, but make directories verified registries or signed +package indexes. A package directory can provide search metadata, but package +installation still comes from TUF-verified release metadata and immutable +artifacts. + +Search ranking should prefer exact names, verified scopes, recent maintained +versions, non-yanked releases, provenance availability, and compatible Jerboa +versions. Security signals should be visible but not presented as absolute +safety claims. + +## Local Development + +`jpkg link` should support local iteration without pretending it is +reproducible: + +- links are project-local by default +- linked packages are recorded in `jpkg.lock` as local overrides +- `jpkg verify --strict` fails if local links are present +- `jpkg publish` refuses to publish with unresolved local links +- `jpkg pack` can include local source only after manifest validation + +## Compatibility With gxpkg Ideas + +Keep: + +- `new` for project templates +- `deps` behavior through `add`, `remove`, `install`, and `update` +- `install`, `update`, `uninstall` +- `link` and `unlink` +- `build` and `clean` +- `search` and package directories +- `list` +- `env` +- retag/reindex behavior, renamed internally as environment indexing + +Change: + +- Git repository names are dev inputs, not the default trust model. +- Tags are not sufficient release identity. +- Builds are explicit and sandboxed. +- Install never executes package code. +- Global install is de-emphasized in favor of project lockfiles. + +## Implementation Handoff + +A follow-up implementer should treat this document as the design authority, not +as permission to implement every phase at once. The first useful slice is Phase +0 plus the local parts of Phase 1. + +Non-negotiable constraints: + +- Do not add install-time lifecycle scripts. +- Do not fetch and run Git repositories as the default package model. +- Do not invent custom signing, provenance, or registry freshness protocols. +- Do not accept unknown manifest fields silently. +- Do not extract package archives before validating digest, size, and archive + structure. +- Do not weaken policy behavior to make early demos easier. +- Do not write `.sls` user-facing package code; Jerboa package examples and + templates use `.ss`. + +First implementation slice: + +- Add `jpkg` multicall dispatch and `jerboa pkg` alias. +- Install or document the `jpkg -> jerboa` symlink in the binary install path. +- Add command stubs with stable help text for the full command surface. +- Implement `jpkg init` for a minimal `jpkg.sexp`. +- Implement a strict manifest parser and validator. +- Implement deterministic local `jpkg pack`. +- Implement local `jpkg verify` for manifest and archive structure. +- Add tests for manifest rejection, duplicate fields, archive path traversal, + symlink entries, absolute paths, and deterministic packing. + +Acceptance criteria for the first slice: + +- `jpkg --help` and `jerboa pkg --help` show the same package-manager command + surface. +- `jpkg init` creates a valid non-executable manifest. +- `jpkg pack` produces byte-identical artifacts for unchanged input. +- `jpkg verify` rejects malformed manifests and unsafe archives before + extraction. +- No network registry code exists until the local artifact path is tested. +- The docs stay updated with any schema or command changes. + +## Implementation Roadmap + +### Phase 0: RFC and CLI dispatch + +- Add `jpkg` multicall dispatch to the Jerboa binary. +- Install `jpkg` as a symlink or hardlink next to `jerboa`. +- Add `jerboa pkg` alias. +- Add command stubs and help text. +- Decide final manifest syntax. + +### Phase 1: Local package primitives + +- Implement `jpkg init`. +- Implement manifest parser and schema validator. +- Implement deterministic `jpkg pack`. +- Implement archive validator. +- Implement content-addressed local store. +- Implement `jpkg verify` for local artifacts. + +### Phase 2: Lockfile and resolver + +- Implement `jpkg add`, `remove`, `install`, and `update` against a local test + registry. +- Implement deterministic lockfile generation. +- Implement dependency conflict explanations. +- Implement project environments over the store. + +### Phase 3: Static registry with TUF + +- Define registry layout and metadata schemas. +- Implement TUF client verification. +- Implement static registry generator for tests. +- Implement untrusted mirror support. +- Add rollback/freeze tests. + +### Phase 4: Signing and provenance + +- Add Sigstore bundle verification. +- Add Ed25519 offline signature support for self-hosted registries. +- Add in-toto/SLSA provenance verification. +- Add registry-side publish validation. +- Add `jpkg publish` against a staging registry. + +### Phase 5: Sandbox builds + +- Implement policy parser. +- Implement Linux Landlock backend first. +- Add macOS and FreeBSD backends where feasible. +- Add native/FFI capability declarations. +- Make build logs and build attestations reproducible inputs. + +### Phase 6: Audit, advisories, and search + +- Add OSV advisory ingestion. +- Implement `jpkg audit`. +- Implement yanking semantics. +- Implement package search indexes. +- Add package-directory management. + +### Phase 7: Federation and hardening + +- Add registry delegation for scopes. +- Add namespace transfer and threshold-maintainer policies. +- Add transparency monitoring. +- Add reproducible rebuild verification for selected packages. +- Add security review tooling for popular packages. + +## Initial Engineering Decisions + +Recommended defaults for the first implementation: + +- Use `jpkg.sexp` for the source manifest. +- Use canonical JSON internally for signed registry metadata. +- Use `.jpkg` as deterministic `tar.gz` initially. +- Use TUF from the first networked registry prototype. +- Require lockfiles for project installs. +- Allow local path dependencies only in `dev` mode. +- Do not implement install scripts. +- Do not support unsigned public-registry packages. + +## Open Questions + +- Should official package names require `@scope/name`, or should unscoped names + be reserved for Jerboa core packages? +- Should source-only packages be the only official format at first, or should + prebuilt native artifacts be allowed behind a stricter capability gate? +- Should the official registry require Sigstore exclusively, or allow + threshold Ed25519 packages for maintainers without supported OIDC workflows? +- Should `jpkg.sexp` replace any existing package declaration mechanism, or + should Jerboa support a small compatibility reader for Gerbil-style + `gerbil.pkg` during migration? +- What is the minimum supported sandbox behavior on platforms without a strong + kernel sandbox? + +## References + +- The Update Framework: https://theupdateframework.io/ +- Sigstore: https://docs.sigstore.dev/ +- SLSA: https://slsa.dev/spec/ +- in-toto attestations: https://in-toto.io/ +- OSV schema: https://ossf.github.io/osv-schema/ +- OpenSSF Scorecard: https://scorecard.dev/