add docs
ober
3718a9f417441f30a49a1e3cb9eae0a2372fe2e9
new file mode 100644 --- /dev/null +++ b/docs/improve-kimi3.md @@ -0,0 +1,259 @@ +# jemacs-qt — Last-Mile Handoff (improve-kimi3) + +Goal: make the **Qt frontend** (`src/jerboa-emacs/qt/`) good enough to replace GNU Emacs +as a daily driver. This document is the implementation brief. Work top-down; each +section is self-contained. **Verify every claim against the cited file:line before +editing — line numbers drift.** + +## 0. Operational cheat sheet (read first) + +- Build: `make build` (jerbuild translates `src/**/*.ss` → `lib/**/*.sls`; **never edit `lib/`**). +- Run Qt interpreted: `make run-qt`. Static binary: `make static-qt` (Docker, Linux). +- Tests must run via the Makefile (it sets `LIBDIRS`, `LD_LIBRARY_PATH`/`DYLD_*`, and `PRELOAD_ENV` shims). `make test` = tier0/2/3/4/5 + org + extra + command-registry audit. +- Qt logic tests (offscreen): `make test-qt`. Live-app behavioral suite (Xvfb + debug REPL): `make test-behavioral`. xdotool e2e: `make test-qt-e2e`. Stress: `make stress-{window,buffer,edit,file,chaos}`. +- Debug REPL: `jemacs-qt --repl 0` writes `PORT=`/`TOKEN=` to `~/.jerboa-repl-port` (override with `JEMACS_REPL_PORT_FILE`). Protocol: token line, then `(N eval "EXPR")`. REPL exposes `send-keys!`, `send-keys-async!`, `screenshot!`, `app-state`, `wait-echo!`, `test-reset!`, `qt-validate-window-tree!` (bound in `qt/app.ss:2182-2273`). +- Crash/self-telemetry: `~/.jemacs-crash.log` (SIGSEGV handler + 64-entry FFI ring buffer in `support/vendor-overrides/qt_shim.cpp`), `~/.jemacs-errors.log` (command errors), `JEMACS_VERBOSE_LOG`. +- Gotcha: Chez `--script` hangs after ~590 top-level forms when Qt threads are active — keep test files small or split (reason `test-qt-part2` mechanism exists). +- Gotcha: `gsh-capture`/`command-substitute` deadlocks from secondary threads (Chez port-registry lock). Use `run-process-interruptible` (`subprocess.ss`). +- Gotcha: jerbuild mangles `guard [else ...]` — use `with-catch`. +- Editing `.ss` files: follow repo `AGENTS.md` — use jerboa-mcp balanced tools, never raw edit/sed. + +## 1. P0 — correctness bugs (fix these first) + +1. **macOS event-pump gap (likely THE blocker on this machine — verify interactively first).** + On macOS the C++ shim runs no `exec()`; a comment claims the Scheme poll loop pumps + `processEvents` every 50 ms (`support/vendor-overrides/qt_shim.cpp:699-701`), but + `qt-app-exec!` (`vendor/jerboa-qt/src/jerboa-qt/qt.ss:851-863`) only calls the tick + + `is-running`, and neither `qt-drain-pending-callbacks!` (`qt/sci-shim.ss:313`) nor + `master-timer-tick!` (`async.ss:240`) calls `qt-app-process-events!`. Steady-state + pumping happens only inside minibuffer reads (`qt/echo.ss:492,543,593,659,716`), PTY + busy-poll (`qt/app.ss:683`), automation waits, and splits. Automation never catches + this because synthetic keys use `sendEvent` directly. **Fix:** call + `qt-app-process-events!` each poll iteration on `__APPLE__` (or always — it is cheap), + then re-run `make test-qt` + a manual typing session. +2. **Tree-sitter reparse keyed on text *length*.** `qt/app.ss:2286-2292`: an edit that + keeps buffer length constant never re-highlights. Use Scintilla's modification + counter (`SCI_GETMODIFY`/`SCI_GETUNDOCOLLECTION` or a monotonic edit counter bumped in + the self-insert path) as the version key. +3. **Narrowing silently truncates candidates at 45.** `qt/echo.ss:229-234`: + `*mb-filtered*` is set to `take filtered (* *mb-max-visible* 3)` = 45 items; matches + beyond #45 are unselectable (M-x has ~1,776 commands). Keep the full filtered list in + `*mb-filtered*`; only cap the *widget* rows and page/scroll the list. +4. **Fake scrolling commands.** `scroll-up`/`scroll-down` are 20-iteration cursor-move + loops (`qt/commands-core.ss:683-699`); `recenter` is `SCI_SCROLLCARET` + (`qt/sci-shim.ss:476-477`). Implement real viewport paging (`SCI_LINESONSCREEN`, + `SCI_SETFIRSTVISIBLELINE`) and true recenter (current line → middle of viewport). +5. **`toggle-line-numbers` is a lie.** Line-number-area API is all stubs returning `#f` + (`qt/sci-shim.ss:559-563`); the margin is unconditionally created + (`qt/window.ss:504-506`). The command echoes ON/OFF but changes nothing + (`qt/commands-edit.ss:1322-1332`); `display-line-numbers-mode` inherits the bug. + Implement via `SCI_SETMARGINWIDTHN` (width 0 = off). +6. **Key handler is outside the error net.** Only `execute-command!` is wrapped in + `with-catch` (`core.ss:1557-1599`); an error in self-insert/chord/refresh logic + escapes into the drain trampoline. Wrap the whole `key-handler` (`qt/app.ss:1076-1571`). +7. **Callback ring buffer drops events silently when full** (8192 entries, + `support/vendor-overrides/qt_chez_shim.c:50-51`). At minimum log drops to + `~/.jemacs-errors.log`; better: drain more aggressively or grow the ring. +8. **Key-fidelity bugs.** `C-Return` indistinguishable from `Return` — both map to + `"C-m"` (`qt/keymap.ss:74-75`, note the dead `(if ctrl? "C-m" "C-m")`); dead `cond` + clause at `qt/keymap.ss:96`; `C-h` bound to `backward-delete-char` then rebound as + help prefix (`core.ss:522` vs `core.ss:601`) — delete the dead first binding. +9. **LSP write failures swallowed.** `with-catch … (void)` around every + `lsp-write-message` (`qt/lsp-client.ss:169-171,194-196,275-277`) — a dead server is a + silent no-op. Detect write failure → mark server dead → echo-area message. +10. **Startup TRACE spam.** Ten unconditional `(display "TRACE …")` in `qt-main` + (`qt/app.ss:2323-2370`). Gate behind `JEMACS_DEBUG` env var. + +## 2. P1 — daily-driver gaps (GNU Emacs parity that matters) + +1. **Mouse support is zero at the Scheme level.** No mouse-press filter exists in the + C++ shim (`qt_shim.cpp:593-638`); Scintilla-native click/drag works but doesn't + refresh modeline/tabs and never reaches the kill ring. Add a mouse-event filter → + ring buffer → Scheme handlers: click-to-move-point + refresh, drag-select → kill-ring + sync, modeline/tab clicks, `mouse-yank-primary` (X11 PRIMARY). Unblocks + `xterm-mouse-mode` (currently echo stub). +2. **No context menus anywhere.** Add right-click menu in editor (cut/copy/paste, + LSP actions, spell) — `context-menu-mode` is an echo stub (`qt/commands-parity5.ss:121`). +3. **Drag & drop FFI exists but is unused** (`qt.ss:1854-1861`, DropFilter in shim). + Wire drop-file → `find-file`, drop-text → insert. +4. **No minibuffer history.** `savehist` data loads at startup (`qt/app.ss:1023`) but + the minibuffer key handler has no M-p/M-n (`qt/echo.ss:421-442`). Implement + per-command history rings (use `:std/misc/ringbuf`, §4). +5. **IME deliberately crippled:** `QT_IM_MODULE=compose` (`qt/app.ss:2340`) — CJK input + impossible. The original sin is a Scintilla `ImSurroundingText` assertion; fix the + stale-position query in the shim, then re-enable real IM modules per-platform. +6. **Single status bar for the whole frame** (`qt/modeline.ss:84,137`). Per-window + modelines: render a one-line modeline widget at the bottom of each split leaf. +7. **Frames are virtual** (window-config swap in one OS window, + `qt/commands-shell.ss:975-1030`). Real `C-x 5 2` needs a second `QMainWindow` sharing + the buffer table — large but well-contained; the split-tree validator pattern + (`qt/window.ss:416-437`) is the template. +8. **~400 echo-only toggle/stub commands** make the advertised surface unreliable: + `make-toggle-command`/`make-stub-command` (`qt/commands-parity3.ss:77-85,521-532`, + ~360 names at :98-508, ~340 more in parity4, 63 modes in `commands-parity5.ss:108-174`). + Registration order means stubs can shadow real implementations + (`qt/commands.ss:2024-2030`). Task: (a) build a CI audit listing every registered + command whose body is an echo stub; (b) for each: implement, downgrade to + "not available" error, or delete. Priority real ones: `blink-cursor-mode` + (`SCI_SETCARETPERIOD 0/toggle`), `winner-mode`, `recentf-mode`, `electric-indent-mode`, + `global-display-line-numbers-mode` (with P0.5), `so-long-mode` (below). +9. **`so-long-mode` is inert.** `check-so-long!` (`qt/commands-shell.ss:427-441`) is + imported (`qt/commands.ss:53`) but **never called**; the flag is never read. Call it + on find-file/revert; when triggered, disable tree-sitter/lexers + decorations for + that buffer. +10. **`run-with-timer` is a stub** ("not available interactively", + `qt/commands-parity5.ss:1332-1333`) but `schedule-periodic!`/`async.ss` already + provides the machinery — wire a one-shot/repeating user-facing timer to it. +11. **Snippets destroy undo history.** Expansion replaces the whole buffer text + (`qt/snippets.ss:44,161` via `qt-plain-text-edit-set-text!`). Replace with + targeted `SCI_REPLACETARGET`/`SCI_INSERTTEXT` edits. +12. **Helm is shallow:** sources flattened, default action only, no action menu / + marked candidates / persistent action (`qt/helm-qt.ss:28-93`); `helm-occur` results + don't jump to line (`qt/helm-commands.ss:28-43`). Implement TAB action menu + + RET default action + jump-to-match. +13. **LSP: single server, no timeouts.** Auto-starts only `jerboa-lsp` + (`qt/commands-lsp.ss:65-80`, one global `*lsp-server-command*`); timeout machinery + neutered (`qt/lsp-client.ss:174-184`). Add per-language server-command alist + + restart-on-crash (use `:std/misc/retry` or `:std/circuit`, §4). For timeouts, + `:std/safe-timeout`'s engine-based `with-timeout` runs on the same thread — verify + SMP safety before adopting, else enforce deadlines in the existing 50 ms poll loop. +14. **Auto-save blocks the UI** (all modified buffers snapshotted synchronously every + 30 s, `qt/app.ss:1855-1866`). Move writes to `spawn-worker` (`async.ss:172`) with a + UI-thread completion. +15. **Tab bar staleness:** refresh cache key is only `(current-buffer . count)` + (`qt/app.ss:541-544`) — modified `*` markers and renames go stale. Include + modified-flag + name in the key. Add close buttons / middle-click close. +16. **which-key is unreadable** (one-line echo dump after 0.5 s, `qt/app.ss:1987-1999`). + Render a real popup list (reuse the narrowing QListWidget). +17. **Org highlighting is a synchronous full-buffer pass on the UI thread** + (`qt/highlight.ss:1348-1355`) — stalls on big files. Make it incremental + (visible-range first) or worker-thread + apply. +18. **Multiple cursors:** Scintilla multi-selection is enabled (`qt/window.ss:527-530`) + but no commands use it. `toggle-global-multiple-cursors` is an echo stub. + +## 3. Performance hot spots + +1. **Full-buffer FFI text fetch on every keystroke** — the biggest perf bug. + `qt-refresh-after-key!` (`qt/app.ss:823-845`) → `qt-update-visual-decorations!` + fetches entire buffer text across FFI (`qt/highlight.ss:1426`) then brace-scans it + (`qt/highlight.ss:1387-1416`); self-insert fetches full text *again* for auto-pair + (`qt/app.ss:1340`) and again for abbrev (`qt/app.ss:1367`). On Linux each fetch is a + BlockingQueuedConnection round-trip. Fix: (a) cache text per buffer with + modification-counter invalidation (same counter as P0.2); (b) brace-match only a + window around point; (c) throttle decorations to the 50 ms tick, not per key. +2. **Narrowing rebuilds the QListWidget per typed char** (`qt/echo.ss:223-247`). With + P0.3 fixed, keep a persistent model and only diff visible rows; consider a + `:std/misc/trie` index for the 1,776-command M-x corpus. +3. **File-watch stats every buffer file every 5 s** (`qt/app.ss:1891-1926`). Replace + with `:std/os/inotify` (Linux) / `:std/os/kqueue` (macOS) — both vendored, unused + (§4). Fall back to polling. +4. **Per-keystroke modeline/tab/title/echo redraws** (`qt/app.ss:834-843`) — coalesce + into the 50 ms tick (the terminal pipeline already does exactly this: + 33 ms throttle + row-diff, `qt/app.ss:96-101,651-760`; copy that pattern). +5. No large-file strategy: no `SCI_SETLAYOUTCACHE`, no chunking; so-long inert (P1.9). + +## 4. Unused Jerboa stdlib features (robustness/perf/correctness wins) + +Vendored jerboa is **1,089 commits behind** `~/mine/jerboa` (vendor @ 37426d9, +2026-06-14; upstream @ 1e43b3df, 2026-07-29). Bump `dependencies.lock` deliberately — +most modules below already exist in the vendored copy, so no bump is required unless +noted. Currently used: sugar, srfi/13, sort, misc/{string,process,ports,channel,list, +completion,rwlock,memo,wg,shuffle,rbtree,pqueue,number,barrier,atom}, text/{json,hex, +diff,glob}, crypto/digest, format, net/{request,uri}, pregexp, srfi/{1,19}, stm, os/ +{signal,fdio}, iter, engine, amb, repl, native-loader. + +| Problem in jemacs | Unused module (vendored unless noted) | What it gives | +|---|---|---| +| Hand-rolled JSON-RPC + Content-Length framing in `qt/lsp-client.ss` | `:std/net/json-rpc` | Spec-correct request/response/error parse incl. batch | +| LSP timeouts neutered (P1.13) | `:std/safe-timeout` (`with-timeout`, engine-based, same-thread) | Preemptive timeouts without spawning threads | +| File-watch polling (§3.3) | `:std/os/inotify`, `:std/os/kqueue` | Event-driven file change (note: inotify needs `jerboa_inotify_shim`) | +| **Local reimplementation of LRU cache** in `chez-powers.ss:80-98` (comment claims vendored module breaks static builds; the vendored and upstream files are byte-identical — re-test whether the claim is still true) | `:std/misc/lru-cache` | Dedupe ~40 lines; also use for brace-scan cache, git-branch memo (hand-rolled in `qt/modeline.ss:17-33`), narrowing results | +| Hand-rolled ring vectors (key lossage, perf commit 2026-07-22) | `:std/misc/ringbuf` | O(1) fixed rings; also model for the C-side callback ring (P0.7) | +| Hand-rolled debounces (LSP didChange 1 s, tree-sitter 150 ms, eldoc 300 ms) | `:std/misc/rate-limiter` (token bucket, non-blocking `try-acquire`) | Uniform, testable throttling | +| Fuzzy match scans 1,776 commands per char | `:std/misc/trie` | Prefix index for M-x / completion corpora | +| Multi-pattern search (occur, grep highlight, isearch-all) | `:std/text/aho-corasick` | One linear pass for N literals | +| Hand-rolled test runners per file | `:std/test` | Unified runner, better CI reporting | +| Hand-rolled `verbose-log!` everywhere (17 call sites in `qt/app.ss` alone) + TRACE spam (P0.10) | `:std/log` / `:std/logger` | Leveled, filterable logging; kill the `display "TRACE"` lines | +| No benchmarks anywhere | `:std/profile` (`with-profile`, `time-it`) | Startup-time + per-keystroke-cost benchmarks (§5.4) | +| LSP/server crash = silent death (P0.9) | `:std/misc/retry`, `:std/circuit` | Restart policy + circuit breaker | +| org-babel temp files (past P0 temp-race security fix) | `:std/os/temp` (`mkstemp`-based, auto-cleanup) | Secure temp files by construction | +| `persist.ss` session writes not atomic/locked | `:std/os/flock`; `:std/os/secure-output` (**upstream only — needs vendor bump**) | Locking; atomic descriptor-relative replacement writes | +| Window-tree invariants hand-rolled (`qt-validate-window-tree!`) | `:std/misc/validate`, `:std/schema` | Declarative invariant specs, reusable for buffer/FFI handles | +| Modal state = flag soup (isearch, query-replace, chords, vterm-copy) | `:std/misc/state-machine` | Explicit, testable modal transitions | +| FFI handle lifecycle (Qt objects, Scintilla docs) relies on guardians alone | `:std/misc/with-destroy`, `:std/resource` | Deterministic acquire/release scoping | +| Server/process health (LSP, vterm reaper) | `:std/health`, `:std/os/supervise` | Liveness checks, supervised restarts | +| Token/id generation | `:std/misc/uuid` | Session/tab ids | +| Mail compose shells out to msmtp | `:std/net/smtp` | Direct SMTP | +| EWW "not usable for modern web" | `:std/text/xml`, `:std/text/html-parse` | Real HTML parsing (JS remains out of scope) | +| Caches that must not leak (LSP, file content) | `:std/misc/weak` | Weak references | +| Secrets in logs (release docs demand redaction review) | `:std/os/secure-output` (upstream only) + redaction pass in `:std/log` | Redaction-aware logging | +| Subprocess reads with hand timeouts | `:std/misc/timeout`, `:std/net/timeout` | Deadline wrappers | + +Also underused prelude features: `with-resource`/`unwind-protect` (3 uses total — apply +to FFI handle scopes), the Result type (`ok`/`err`/`try-result`/`->?` — zero uses; good +fit for LSP calls and FFI wrappers instead of `with-catch … (void)`). + +## 5. Testing & usability-verification improvements + +1. **Fix broken/vapor targets first** (cheap, high trust): + - `make scenario-test`/`scenario-burn[-static]` reference `tests/scenario-runner.ss` — + **file never existed** (Makefile:1846-1922). Implement it: scripted editing sessions + (open → edit → split → M-x → save) replayed through the debug REPL, with + `screenshot!` capture (primitive exists: `qt/automation.ss:248-259`, zero callers). + - `make test-pty` references missing `tests/test-pty.ss` (Makefile:777-778). + - `test-qt.ss` groups 44-53 are orphaned: `tests/test-qt-part2.ss` never existed and + there is no `test-qt-part2` target (silently "SKIP"ped, Makefile:761-765). Split the + file for real (remember the ~590-form Chez hang). + - `tests/test-core.ss` has **no Makefile target** — wire it into `make test`. + - `tests/test-emacs.ss` (2,471 lines) is excluded from `make test`, `verify`, and CI — + either fix and include, or delete. + - `make test-behavioral` and `make stress-run` are **broken on macOS**: hard-coded + Linux `.so` names + literal `xvfb-run` (Makefile:1755,1925) — use `$(SHLIB_EXT)` and + `$(XVFB_RUN)` like the newer `stress-*` targets do. +2. **Key-fidelity test suite** (new): C-g/`keyboard-quit` from minibuffer + during + subprocess output, M-x completion round-trip, C-Return vs Return (P0.8), modifier + combos, chord passthrough to vterm. Layer on `test-qt-functional.sh` (xdotool) since + it's the only layer exercising real OS input — and add it to CI (currently orphaned). +3. **Screenshot goldens:** `screenshot!` exists but no comparison harness. Add golden + PNGs for: startup frame, split layout, modeline states, narrowing list, org file, + magit status. Compare with tolerance; store failures as artifacts. +4. **Startup & perf smoke in CI:** CI (`.github/workflows/ci.yml`) never launches an + editor binary. Add: launch-under-Xvfb → wait for REPL port → `app-state` → quit + (crash-on-startup gate); record startup wall time and per-keystroke cost using + `:std/profile`; RSS soak during `stress-edit` (stress currently records no memory + series). +5. **Wire invariants into behavioral phases:** `qt-validate-window-tree!` is REPL-bound + (`qt/app.ss:2130`) but `test-behavioral.ss` never calls it — call it after every + window-op phase (docs/robust.md §1). +6. **Clipboard tests:** currently only a hand-reviewed proof marker + (`scripts/editor-flow-evidence.sh:24`). Automate: set clipboard via REPL → yank → + assert; kill → read clipboard → assert. +7. **TUI backend e2e:** essentially untested end-to-end (test-tier4 excludes live + terminal). Add a PTY-driven (expect/tmux-style) smoke: launch `jemacs`, type, assert + screen contents via `vtscreen.ss`. +8. **CI coverage:** GitHub CI runs only `security/build/test/audit`; add + `test-functional`, `test-qt-e2e`, `test-behavioral` (Linux runner has xvfb already). + +## 6. Suggested implementation order + +1. **Week 1 — trust:** P0.1 (macOS pump; verify by hand first), P0.2, P0.3, P0.6, P0.9, + P0.10; §5.1 target repairs; §5.4 startup smoke. Everything here is small and + independently verifiable. +2. **Week 2 — feel:** P0.4, P0.5, P0.7, P0.8; P1.4 (minibuffer history), P1.9 (so-long), + P1.10 (run-with-timer), P1.15 (tab bar); §3.1 + §3.4 (per-keystroke perf). +3. **Week 3 — surface:** P1.1–P1.3 (mouse, context menu, DnD), P1.8 stub purge + CI + audit, P1.11 (snippet undo), P1.12 (helm actions), P1.16 (which-key). +4. **Week 4 — depth:** P1.13 (LSP multi-server/restart), P1.14 (async auto-save), + P1.17 (org highlighting), §3.3 (inotify), §5.2/5.3/5.6/5.7 test layers, + vendor bump evaluation (§4 preamble). +5. **Later:** P1.5 per-window modeline, P1.6 real frames, P1.7 IME, P1.18 multiple + cursors, EWW html-parse upgrade. + +## 7. Doc-drift cleanup (do alongside, cheap) + +- `docs/jemacs-vs-emacs.md` internal contradictions: persistent undo (§5 table vs + summary line 192), tree-sitter (50-features #36 "DONE" vs Tier-3 gap line 1016), + super-to-meta (§45 :1089 vs :1093), LSP multi-server (:551-553), orphaned TODO at :554. +- `docs/stress-test.md:230-247` shows an outdated `stress-run` recipe. +- `plan.md` is fully superseded (Qt port done) — mark it historical. +- README "Source Layout" omits `qt/` submodule descriptions and `chez-powers.ss`.