Add robustness roadmap
ober
3e34c0d38071a8437c359812293393e9e6288faf
new file mode 100644 --- /dev/null +++ b/docs/robust.md @@ -0,0 +1,208 @@ +# Robustness Roadmap + +## Goal + +Make invalid editor state fail early in Scheme, before it crosses into +Qt/Scintilla FFI and becomes a native crash. + +This repo already has the right foundation: crash logging, a TCP debug REPL, +static gdb burn-in, and automated stress actions. The next step is to tighten +the feedback loop so structural bugs are caught by focused checks and short +stress profiles, not by late-stage segfault diagnosis. + +## Priorities + +### 1. Window tree invariants + +Add explicit validation around Qt window-tree state. + +Recommended checks: + +- Every split tree node has a valid, known shape. +- Every leaf contains exactly one edit window. +- The active edit window is present in the current split tree. +- No edit window appears twice in the same tree. +- Every edit window has a non-null editor widget. +- Every splitter has a valid Qt handle. +- Scheme split-tree children match the corresponding Qt splitter children. +- Deleted windows are not reachable from the active tree. + +Recommended entry points: + +- After `split-window`, `split-window-right`, and `split-window-below` +- After `delete-window` and `delete-other-windows` +- After `balance-windows` +- Before and after rebuilding Qt splitter layout +- At the end of each window stress phase + +Keep the checks enabled in debug and stress modes first. If the runtime cost is +small, leave the cheap shape checks enabled in normal development builds. + +### 2. Generated-code drift checks + +The `lib/**/*.sls` files are generated from `src/**/*.ss`, but they are checked +in. That is workable, but only if drift is impossible to miss. + +Add a target that verifies a clean generated tree: + +```make +check-generated: + $(MAKE) build + git diff --exit-code -- lib +``` + +Use it in CI and in any pre-commit path that touches Scheme source. This catches +missing regeneration, accidental edits to generated files, and nondeterministic +jerbuild output. + +### 3. Focused stress profiles + +Keep the long burn-in test, but split the stress driver into shorter named +profiles. The all-in-one test is good at finding crashes; smaller profiles are +better at making failures actionable. + +Recommended targets: + +```make +stress-window +stress-window-static +stress-buffer +stress-buffer-static +stress-edit +stress-edit-static +stress-file +stress-file-static +stress-chaos +stress-chaos-static +``` + +Suggested profile ownership: + +- `stress-window`: split, delete, balance, other-window, active-window checks +- `stress-buffer`: open, switch, kill, restore scratch/current buffer +- `stress-edit`: insert, delete, kill/yank, undo, mark/region +- `stress-file`: temp-file churn, async reads, reloads, buffer/file association +- `stress-chaos`: mixed random operations, equivalent to the current burn-in + +The shortest profile that reproduces a crash should become the first regression +test for that bug. + +### 4. Crash-log freshness + +The stress targets currently print `~/.jemacs-crash.log` if it exists. That can +surface an old crash report and confuse the result. + +Before starting a stress run: + +- Record whether `~/.jemacs-crash.log` exists. +- Record its mtime if present. +- Clear or archive `stress-test.log`. + +After the run: + +- Print the crash log only if its mtime changed during the run. +- Fail the target if a new crash log was written. +- Report "no new crash log" when the file is unchanged. + +This makes burn-in output trustworthy even on machines with old crash artifacts. + +### 5. FFI guardrails + +FFI-heavy GUI code needs cheap boundary checks. + +Add wrappers or assertions for: + +- Null Qt handles before every mutating Qt call. +- Object ownership before widget insertion/removal/reparenting. +- Splitter index bounds before `insert-widget` and `add-widget`. +- Scintilla widget validity before `send-message`. +- Thread affinity for Qt operations. + +Prefer failing with a Scheme condition that names the invalid object and command +over allowing a bad pointer to reach C++. + +### 6. Makefile simplification + +The Makefile is useful, but it is carrying build, static build, Qt setup, gdb, +Xvfb, REPL orchestration, and stress-test policy. + +Keep the public targets in `Makefile`, but move multi-step orchestration into +small scripts under `scripts/`. + +Good candidates: + +- Starting jemacs-qt under Xvfb and waiting for the REPL port file +- Running static gdb burn-in +- Checking crash-log freshness +- Running a stress profile and collecting the tail summary + +The Makefile should read like an index of workflows. The scripts should own the +procedural details. + +### 7. Known-good developer loop + +Document and preserve a short loop for risky Qt/window/editor changes. + +Recommended minimum loop: + +```bash +make build +make test-tier2 +make test-qt +make stress-window-static +``` + +For broader editor command work: + +```bash +make build +make test-functional +make test-org +make stress-chaos-static +``` + +For generated-code changes: + +```bash +make build +make check-generated +``` + +These loops should be short enough to run often, and strict enough to catch the +common state-corruption cases. + +### 8. Command registry audits + +The command surface is spread across many `commands-*` files. Add a registry +audit test that checks: + +- Duplicate command names +- Aliases pointing at missing commands +- Registered symbols whose procedures are unbound +- Commands documented as implemented but missing from the registry +- Commands in the registry with no smoke-test coverage + +If possible, generate parity or command-list docs from the live registry rather +than maintaining command lists by hand. + +## Suggested Implementation Order + +1. Add `validate-window-tree!` and run it in window commands during stress mode. +2. Add `stress-window-static` as the smallest gdb-backed window reproducer. +3. Add crash-log freshness handling to stress targets. +4. Add `check-generated`. +5. Add FFI null-handle and ownership assertions around splitter/widget calls. +6. Split the remaining stress profiles. +7. Move stress orchestration into scripts. +8. Add command registry audits. + +## Success Criteria + +A robust workflow should make these statements true: + +- A malformed split tree fails with a Scheme error naming the bad node. +- A stale crash log is never mistaken for a fresh stress-test failure. +- Regenerating `lib/` from `src/` is deterministic or the diff is visible. +- Window regressions can be reproduced with a short focused target. +- Native crash logs are reserved for real FFI/runtime failures, not ordinary + editor state mistakes.