v3 pipeline: 3-stage CPT→SFT→DPO on RunPod axolotl + docs
ober
c30b2025456cc36934aad24b7b207b89be9828f2
--- a/.gitignore +++ b/.gitignore @@ -16,10 +16,12 @@ llama.cpp/ # Pipeline + adapter outputs (multi-GB binary) runpod-pipeline-final/ jerboa-mlx-4bit-v2/ +jerboa-mlx-6bit-v3/ mlx_data/ mlx_data_v1/ mlx_adapters/ mlx_adapters_v1/ +gguf/ # Logs *.log @@ -29,6 +31,7 @@ mlx_adapters_v1/ .together_state_v2.json .together_state_v3.json .runpod_state.json +.runpod_abliterate_state.json # Editor / agent state .claude/ new file mode 100644 --- /dev/null +++ b/.gitsafe.json @@ -0,0 +1,19 @@ +{ + "severity": "high", + "exclude": [ + "*.lock", + "go.sum", + "*.md", + "vendor/**", + "node_modules/**", + "*.min.js", + "*.min.css", + "training_data*.json", + "training_data*.jsonl", + "cpt_corpus*.jsonl", + "sft_v3*.jsonl", + "dpo_pairs*.jsonl", + "delta_sft.jsonl", + "eval_*.json" + ] +} --- a/README.md +++ b/README.md @@ -1,219 +1,142 @@ # Jerboa Scheme LoRA -A fine-tuned Qwen 2.5 7B model that knows Jerboa Scheme. Run locally with Ollama, host on RunPod, or pull from the registry. +A fine-tune of **Qwen3-Coder-30B-A3B-Instruct** that knows [Jerboa](https://github.com/jaimef/jerboa) — a Chez-Scheme-based dialect with a Gerbil-flavored prelude. The LoRA teaches the base model Jerboa's module syntax (`(std foo)`, `(jerboa prelude)` — not `:std/foo`), the standard library, the actor/fiber system, the FFI, and how Jerboa diverges from Gerbil/Racket/Clojure/SRFI. -Jerboa is a Chez-Scheme-based dialect with a Gerbil-flavored prelude. This LoRA teaches the base model the Jerboa-specific module syntax (`(std foo)`, `(jerboa prelude)` — not `:std/foo`), the standard library, the actor/fiber system, the FFI, and how Jerboa diverges from Gerbil/Racket/Clojure/SRFI. +Currently shipping **v3**, trained via a 3-stage CPT → SFT → DPO pipeline on RunPod A100 80GB. -## Quick Start (Local) +## Quick Start (Mac) ```bash ollama pull jaimef/jerboa-qwen ollama run jaimef/jerboa-qwen "How do I import the Jerboa prelude and parse JSON?" ``` -## Deployment Options +Tags: `:latest` is Q8_0 (~32 GB, full fidelity). `:q4_k_m` is the smaller quant (~18 GB). Pick `:latest` if it fits. -| Option | Cost | Speed | Setup | -|--------|------|-------|-------| -| **Local Ollama (GPU)** | Free | 30–40 tok/s | `ollama pull jaimef/jerboa-qwen` | -| **Local Ollama (CPU)** | Free | 5–10 tok/s | Same as above | -| **RunPod Serverless** | $0 idle, ~$0.39/hr active | 30–40 tok/s | `./deploy_runpod.sh` | -| **Together AI Endpoint** | $6.60/hr always-on | Fast | Not recommended | +## Pipeline (v3) -### Estimated RunPod monthly costs +Three stages on one A100 80GB PCIe pod, ~5–6h wall clock, ~$11–15. See [TRAINING_PIPELINE.md](TRAINING_PIPELINE.md) for the design rationale and [TRAINING_FIX_PLAN.md](TRAINING_FIX_PLAN.md) for the v3 hyperparam corrections that fixed the v1 "DPO delta below the quantization floor" pathology. -| Usage | Hours/month | Cost/month | -|-------|-------------|------------| -| Idle (scale-to-zero) | 0 | **$0** | -| Light (1hr/day) | ~30 | **~$10** | -| Moderate (3hr/day) | ~90 | **~$31** | -| Heavy (8hr/day) | ~240 | **~$82** | +| Stage | What it learns | Data | LR / Epochs | LoRA | +|---|---|---|---|---| +| **CPT** | Token distribution of raw Jerboa source | `cpt_corpus_v3.jsonl` (~6 MB, mined from `jerboa`, `jerboa-mcp`, stdlib doc-comments) | 2.0e-5 / 2 | r=64, α=128 | +| **SFT** | Answer Jerboa questions in chat | `sft_v3_mined.jsonl` (~3.5 MB, compile-gated via `jerboa_check_syntax`) | 1.0e-4 / 2 | r=64, α=128 | +| **DPO** | Suppress Gerbil/Racket/Clojure/SRFI hallucinations | `dpo_pairs_v3.jsonl` (~300 pairs, compile-validated chosen) | 5.0e-6 / 3 | r=64, α=128 | -## Use with OpenCode - -### Option A: Local Ollama +Each stage's LoRA is merged into bf16 between stages so the next stage trains on absorbed weights, not stacked adapters. Targets attention + MLP + MoE experts (`experts.gate_up_proj`, `experts.down_proj`) — the key reason RunPod axolotl beat the previous Together-AI single-stage path, which was attention-only. -Add to `~/.config/opencode/opencode.json`: +### Run the pipeline -```json -{ - "$schema": "https://opencode.ai/config.json", - "provider": { - "ollama": { - "npm": "@ai-sdk/openai-compatible", - "name": "Ollama (local)", - "options": { - "baseURL": "http://localhost:11434/v1" - }, - "models": { - "jerboa-qwen": { - "name": "Jerboa Qwen" - } - } - } - } -} +```bash +# Provision pod + push configs/data + train + pull + tear down +JERBOA_MODE=pipeline .venv/bin/python runpod_train.py up +JERBOA_MODE=pipeline .venv/bin/python runpod_train.py push +JERBOA_MODE=pipeline .venv/bin/python runpod_train.py train +JERBOA_MODE=pipeline .venv/bin/python runpod_train.py eval # base vs trained, on-pod +JERBOA_MODE=pipeline .venv/bin/python runpod_train.py pull # → runpod-pipeline-final/ +JERBOA_MODE=pipeline .venv/bin/python runpod_train.py down ``` -Or run `./configure_opencode.sh ollama` to write that for you. +Pod state lives in `.runpod_state.json` (gitignored). Each stage is idempotent: rerunning `train` skips stages whose `.done` marker is `OK`. Killing the orchestrator does not kill the training — each stage runs in its own tmux session on the pod. -### Option B: RunPod Serverless (recommended if no local GPU) +### Build data -```json -{ - "$schema": "https://opencode.ai/config.json", - "provider": { - "runpod": { - "npm": "@ai-sdk/openai-compatible", - "name": "RunPod (serverless)", - "options": { - "baseURL": "https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1", - "apiKey": "<RUNPOD_API_KEY>" - }, - "models": { - "jerboa-qwen": { - "name": "Jerboa Qwen 7B" - } - } - } - } -} +```bash +python3 build_cpt_corpus_v2.py # → cpt_corpus_v3.jsonl +python3 build_sft_v3.py # → sft_v3_mined.jsonl (compile-gated) +python3 build_dpo_pairs_v3.py # → dpo_pairs_v3.jsonl (compile-validated) ``` -Replace `<ENDPOINT_ID>` and `<RUNPOD_API_KEY>` with your values. - -## Deploy to RunPod +The builders read from `~/mine/jerboa` (source, stdlib, tests, docs) and `~/mine/jerboa-mcp` (cookbooks, divergence, error-fixes, api-signatures) via the `jerboa_*` MCP tools. -One script handles everything: downloads merged model, uploads to HuggingFace, creates RunPod endpoint via API. +### Convert + publish ```bash -# Prerequisites -pip install together huggingface_hub -export TOGETHER_API_KEY="your-key" -export RUNPOD_API_KEY="your-key" -hf auth login - -# Deploy (after training completes) -./deploy_runpod.sh jaimef21/jerboa-qwen-7b +# MLX 6-bit bundle for Mac (loads under mlx_lm or LM Studio) +./convert_to_mlx.sh runpod-pipeline-final jerboa-mlx-6bit-v3 + +# GGUF for ollama +./build_ollama_gguf.sh # → gguf/jerboa-qwen-q8_0.gguf + q4_k_m +./push_ollama.sh jaimef # pushes :latest (q8_0); add --all to also push :q4_k_m + +# HuggingFace +./upload_hf.sh ``` -The script reads the Together AI job ID from `.together_state.json` (set by `train_together.py train`), so you don't need to paste it in. +## Evaluation -Endpoint URL: `https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1` +v3 is gated on two evals before publish; both must beat the un-fine-tuned base. -## Build from Source +| Eval | What it measures | Script | v3 result | +|---|---|---|---| +| Held-out coding | Compile-rate + idiom hits on questions never seen during training | `eval_holdout.py` | see `eval_pod_v3.json` vs `eval_pod_base.json` | +| Per-pair similarity | Chosen-vs-rejected on the DPO set (Jaccard + char-similarity, volume-invariant) | `eval_similarity.py` | see `eval_mlx_v3.json` vs `eval_mlx_v2.json` | -### 1. Generate training data +The similarity eval is the one to trust — it's volume-invariant and per-pair, so it doesn't get gamed by dataset growth. -```bash -# These should already exist on your machine: -# ~/mine/jerboa (the Jerboa source repo) -# ~/mine/jerboa-mcp (the Jerboa MCP server with cookbooks/api/divergence) +## Use with OpenCode + +### Local Ollama -python3 convert_training_data.py -# → training_data.jsonl (~9MB, 4,622 entries) +```json +{ + "$schema": "https://opencode.ai/config.json", + "provider": { + "ollama": { + "npm": "@ai-sdk/openai-compatible", + "name": "Ollama (local)", + "options": { "baseURL": "http://localhost:11434/v1" }, + "models": { "jerboa-qwen": { "name": "Jerboa Qwen" } } + } + } +} ``` -### 2. Train on Together AI (~$3, ~7 minutes) +Or run `./configure_opencode.sh ollama` to write that for you. -```bash -pip install together -export TOGETHER_API_KEY="your-key" +## Why this works where the previous approach didn't -python3 train_together.py upload -python3 train_together.py train -python3 train_together.py status -``` +The v1 pipeline was a single SFT pass on Together AI (attention-only LoRA) + a single SFT pass on MLX (attention + MLP, 4-bit base). Two structural problems: -### 3. Deploy +1. **No CPT** — the model never saw raw Jerboa source. It learned "how to answer questions about Jerboa" but not "how Jerboa code naturally looks." +2. **No DPO** — the 119 divergence entries (`hash-has-key?` → `hash-key?`, `(srfi :NN)` → `(srfi NN)`, etc.) were flattened into SFT Q/A, which only pulls the model toward the right form. DPO actively pushes it away from the wrong form. The wrong form's prior (learned from the base's massive Racket/Gerbil training data) survived essentially intact. -**Local (with GPU or slow CPU):** -```bash -./download_and_convert.sh -``` +v3 fixes both, plus the v1 DPO hyperparam pathology (LR `5e-7` × 18 steps × LoRA r=16 produced a cumulative weight update of ~2e-5 — below the BF16 precision floor and erased by Q4_K_M quantization). v3 runs LR `5e-6` / 3 epochs / r=64, α=128. -**Hosted (RunPod serverless):** -```bash -export RUNPOD_API_KEY="your-key" -hf auth login -./deploy_runpod.sh YOUR_USERNAME/jerboa-qwen-7b -``` +## v4 (in progress) -### 4. Verify +Abliterates the base model with [Heretic](https://github.com/p-e-w/heretic) on RunPod before re-running the v3 CPT → SFT → DPO pipeline on the abliterated base. Qwen's alignment is unusually stubborn — v1 plateaued at 75/100 refusals; v2 (winsorization=0.95 + wider KL target) made it worse (96/100, aborted); v3 is back to v1 baseline with `orthogonalize_direction=false` (subtract the full refusal vector, accept higher capability cost). -```bash -python3 verify_model.py \ - --base-url http://localhost:11434/v1 \ - --model jerboa-qwen -v -``` +Driver: `runpod_abliterate.py`. Driver-side automation that bypasses heretic's interactive TUI: `heretic_auto_driver.py`. Active config: `config.jerboa.v3.toml`. Target HF repo: `jaimef21/qwen3-coder-30b-a3b-abliterated-v3`. -### 5. Push to Ollama registry +## Repo layout -```bash -./push_ollama.sh YOUR_USERNAME -``` +| File | Purpose | +|---|---| +| `runpod_train.py` | RunPod pod lifecycle + staged training (`up`/`push`/`train`/`eval`/`pull`/`down`) | +| `runpod_abliterate.py` | RunPod pod lifecycle for the v4 Heretic abliteration | +| `heretic_auto_driver.py` | Monkey-patches Heretic's TUI prompts so it runs hands-off + uploads to HF | +| `axolotl_jerboa_{cpt,sft,dpo}.yaml` | Per-stage axolotl configs (r=64, α=128, MoE expert targets) | +| `build_cpt_corpus_v2.py` | Mine `~/mine/jerboa` + `~/mine/jerboa-mcp` → CPT corpus | +| `build_sft_v3.py` | Mine cookbooks + error-fixes + stdlib → SFT, compile-gated | +| `build_dpo_pairs_v3.py` | Generate divergence pairs across Gerbil/Racket/Clojure/SRFI surface variants, compile-validated | +| `eval_holdout.py` | Held-out Jerboa coding eval (compile-rate + idiom hits) | +| `eval_similarity.py` | Per-pair chosen-vs-rejected similarity eval | +| `eval_on_pod.py` | Pod-side eval runner (base + trained, parallel) | +| `eval_mlx.py` | MLX-side eval runner for local Mac comparison | +| `build_ollama_gguf.sh` | Convert merged model → GGUF Q8_0 + Q4_K_M | +| `push_ollama.sh` | Push to ollama.com (`:latest` = Q8_0; `--all` also pushes `:q4_k_m`) | +| `upload_hf.sh` | Push merged model to HuggingFace | +| `config.jerboa.v2.toml` | Heretic config for v4 abliteration | +| `convert_to_mlx.sh` | mlx_lm.convert wrapper | +| `manage_runpod.sh` | RunPod endpoint lifecycle (separate from training pods) | -## Training Data - -**4,622 entries** generated from the Jerboa source tree and the jerboa-mcp knowledge base. - -| Source | Count | Description | -|--------|-------|-------------| -| doc | 2,320 | jerboa/docs/*.md (architecture, fiber, capability, FFI, ...) + top-level docs (JERBOA-LANG, README, CLAUDE) | -| cookbook | 924 | Verified working code recipes from cookbooks.json | -| api | 626 | Per-module export catalogs from api-signatures.json | -| test | 270 | Real usage examples from jerboa/tests/test-*.ss | -| divergence | 238 | Cross-dialect "wrong → right" pairs (Racket/Gerbil/Clojure/SRFI → Jerboa) | -| errorfix | 112 | Error pattern → fix mappings | -| security | 83 | Vulnerability patterns and remediation | -| convention | 24 | Hand-written Jerboa idiom teaching examples (3× weighted) | -| std-source | 21 | Source of key stdlib modules (prelude, fiber, error, sort, ...) | -| source | 4 | Tutorial examples from jerboa/examples/ | - -### Output formats - -| File | Format | Use with | -|------|--------|----------| -| `training_data_together.jsonl` | Together AI messages | Together AI fine-tuning | -| `training_data.jsonl` | ChatML/ShareGPT | LLaMA-Factory, Axolotl, Unsloth | -| `training_data_alpaca.jsonl` | Alpaca JSONL | Unsloth, HuggingFace | - -## Scripts - -| Script | Purpose | -|--------|---------| -| `convert_training_data.py` | Generate training data from jerboa + jerboa-mcp | -| `train_together.py` | Upload, train, and monitor on Together AI | -| `download_and_convert.sh` | Download adapter, convert to GGUF, set up Ollama | -| `deploy_runpod.sh` | Upload merged model to HuggingFace, create RunPod endpoint | -| `manage_runpod.sh` | RunPod endpoint lifecycle (list, health, delete, purge) | -| `push_ollama.sh` | Tag and push model to Ollama registry | -| `configure_opencode.sh` | Generate OpenCode config for Ollama/RunPod | -| `verify_model.py` | Run 10 Jerboa-specific test prompts | -| `train_unsloth.py` | Local GPU training with Unsloth | -| `merge_and_export.py` | Merge adapter + base to GGUF (needs 32GB RAM or GPU) | -| `train_runpod.sh` | One-shot training on rented GPU | -| `Modelfile` | Ollama model definition | - -## Iterating - -To improve the model with more training data: +## Iteration loop 1. Add recipes to `~/mine/jerboa-mcp/cookbooks.json` -2. Add divergence entries to `~/mine/jerboa-mcp/divergence.json` for any new hallucinations you catch -3. `python3 convert_training_data.py` -4. `python3 train_together.py upload` -5. `python3 train_together.py train` -6. `./download_and_convert.sh` (local) or `./deploy_runpod.sh` (hosted) - -## Why a Jerboa-specific model? - -Out-of-the-box LLMs hallucinate Gerbil/Racket/Clojure/SRFI forms when asked for Jerboa code: -- `(import :std/sort)` instead of `(import (std sort))` -- `hash-has-key?` instead of `hash-key?` -- `define-condition-type` instead of the Jerboa `(std error)` helpers -- `(srfi :NN)` instead of `(srfi NN)` - -The 238 divergence entries + 626 API-signature entries + 924 cookbook recipes train the model on Jerboa's actual surface area, so generated code compiles and runs. -# jerboa-lora -# jerboa-lora +2. Add divergence entries to `~/mine/jerboa-mcp/divergence.json` for any new hallucinations caught +3. Rebuild data: `build_cpt_corpus_v2.py && build_sft_v3.py && build_dpo_pairs_v3.py` +4. Bump version suffix on output JSONLs and adapter dirs +5. Re-run the pipeline (`runpod_train.py up/push/train/eval/pull/down`) +6. Eval gate: trained must beat base on both holdout and similarity +7. Convert + push: `build_ollama_gguf.sh && push_ollama.sh && upload_hf.sh` --- a/TODO.md +++ b/TODO.md @@ -1,154 +1,32 @@ -# Jerboa Scheme LoRA Training — Together AI +# Jerboa LoRA — Status -## Status +## Shipped -- [x] Scaffold scripts (forked from gerbil-lora) -- [x] Generate training data (4,622 entries from cookbooks, docs, api signatures, divergence, tests, source) -- [ ] Upload training data to Together AI -- [ ] Start fine-tuning job -- [ ] Wait for training to complete (~7 min expected for 7B / 3 epochs) -- [ ] Download adapter and convert to GGUF -- [ ] Deploy locally with Ollama -- [ ] Push to Ollama registry (`./push_ollama.sh jaimef`) -- [ ] Deploy to RunPod serverless (`./deploy_runpod.sh`) -- [ ] Verify model with `verify_model.py` -- [ ] Connect to OpenCode +- [x] **v1** — single-stage SFT on Together AI + MLX. Plateaued at val loss 2.08, hallucinated Gerbil forms. Superseded. +- [x] **v3** — 3-stage CPT → SFT → DPO on RunPod axolotl. Beats base on held-out coding eval and similarity eval. Live as `jaimef/jerboa-qwen:latest` (Q8_0) and `:q4_k_m` on Ollama. ---- +## v4 — abliterated base (in progress) -## Training Data +- [x] Heretic upstream cloned to `~/mine/heretic` +- [x] Tuned Heretic config (`config.jerboa.v2.toml`) — wider KL target, winsorization, seeded +- [x] `runpod_abliterate.py` — pod lifecycle (up/push/install/run/down) +- [x] `heretic_auto_driver.py` — bypass Heretic's interactive TUI, auto-upload to HF +- [x] v1 abliteration attempted — plateaued at 75/100 refusals (Qwen alignment unusually stubborn) +- [x] v2 abliteration — winsorization=0.95 + kl_target=0.05 made it **worse** (96/100 refusals at trial 150); aborted at trial 153. Conclusion: winsorization removed the signal we needed to ablate. +- [ ] **v3 abliteration** — back to v1 baseline but `orthogonalize_direction=false` (subtract the full refusal vector, accept higher capability cost). Config in `config.jerboa.v3.toml`. +- [ ] HF upload to `jaimef21/qwen3-coder-30b-a3b-abliterated-v3` +- [ ] Sanity check abliterated base (refusal rate < 20%, coding ability intact) +- [ ] Re-run CPT → SFT → DPO with new base (`axolotl_jerboa_{cpt,sft,dpo}.yaml` already point at `jaimef21/qwen3-coder-30b-a3b-abliterated` — bump to `-v2`) +- [ ] Eval v4 vs v3 vs un-abliterated base on holdout + similarity +- [ ] Convert to MLX 6-bit + GGUF, push to Ollama as `:v4` and (if a clean win) promote `:latest` -Generated **4,622 training entries** in `~/mine/jerboa-lora/`: +## v5 — open questions -| File | Format | Size | -|--------------------------------|------------------------|--------| -| `training_data_together.jsonl` | Together AI (messages) | 8.7 MB | -| `training_data.jsonl` | ChatML/ShareGPT | 9.0 MB | -| `training_data_alpaca.jsonl` | Alpaca JSONL | 6.1 MB | +- Base model swap? On 128 GB Mac M5, `Qwen3-Next-80B-A3B-Instruct` at 8-bit fits comfortably (~80 GB) with same 3B active inference speed. RunPod training cost ~2–3× v3 (~$250–400 / pipeline). Decide after v4 lands. +- Bigger DPO set — current ~300 pairs. Programmatic generators in `build_dpo_pairs_v3.py` could 5× by adding more idiom-variant generators (Common Lisp `defun`/`setf`/`mapcar`, more Racket forms). -### Source breakdown +## Reference -| Source | Count | -|--------|-------| -| doc | 2,320 | -| cookbook | 924 | -| api | 626 | -| test | 270 | -| divergence | 238 | -| errorfix | 112 | -| security | 83 | -| convention | 24 | -| std-source | 21 | -| source | 4 | - -Regenerate: `python3 convert_training_data.py` - ---- - -## Step 1: Setup - -```bash -pip install together -export TOGETHER_API_KEY="your-key-here" -``` - -## Step 2: Upload - -```bash -python3 train_together.py upload -``` - -Saves `file_id` to `.together_state.json`. - -## Step 3: Train - -```bash -python3 train_together.py train -``` - -Saves `job_id` to `.together_state.json`. Training settings: -- LoRA r=16, alpha=32 -- 3 epochs, learning rate 1e-5, batch size 8 -- Base model: Qwen/Qwen2.5-7B-Instruct - -## Step 4: Wait & Status - -```bash -python3 train_together.py status -``` - -When done, the model name (e.g. `jaimef_xxxx/Qwen2.5-7B-Instruct-yyyyyyyy`) is saved to state. - ---- - -## Step 5: Deploy — Choose Your Option - -### Option A: Local Ollama (free, needs GPU for good speed) - -**No merge required** — Ollama supports LoRA adapters natively. - -```bash -./download_and_convert.sh -``` - -Or pull from the registry once published: -```bash -ollama pull jaimef/jerboa-qwen -``` - -Configure OpenCode: -```bash -./configure_opencode.sh ollama -``` - -### Option B: RunPod Serverless (scale-to-zero, ~$0.39/hr active) - -```bash -export RUNPOD_API_KEY="your-key" -hf auth login -./deploy_runpod.sh jaimef21/jerboa-qwen-7b -``` - -The script auto-reads JOB_ID from `.together_state.json`. - -Configure OpenCode: -```bash -./configure_opencode.sh runpod <ENDPOINT_ID> -``` - -### Option C: Local Unsloth training (free, needs 16GB+ GPU) - -If you have an RTX 4090 / 3090 / A100 etc.: -```bash -python3 train_unsloth.py # → ./jerboa-lora-output/ -python3 merge_and_export.py # → ./jerboa-qwen-gguf/ -ollama create jerboa-qwen -f Modelfile -``` - ---- - -## Verification - -```bash -# Local -python3 verify_model.py --base-url http://localhost:11434/v1 --model jerboa-qwen -v - -# RunPod -python3 verify_model.py \ - --base-url https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1 \ - --model jaimef21/jerboa-qwen-7b \ - --api-key $RUNPOD_API_KEY -v -``` - -10 test cases covering: prelude imports, divergence (hash-has-key? → hash-key?), fibers, JSON, try/catch, pattern matching, sort, actor system, error conditions. - ---- - -## Iteration - -To improve quality: -1. Add recipes to `~/mine/jerboa-mcp/cookbooks.json` -2. Add new divergence entries for caught hallucinations -3. `python3 convert_training_data.py` -4. `python3 train_together.py upload && python3 train_together.py train` -5. Redeploy +- Pipeline design: [TRAINING_PIPELINE.md](TRAINING_PIPELINE.md) +- v3 hyperparam corrections: [TRAINING_FIX_PLAN.md](TRAINING_FIX_PLAN.md) +- Data sources: `~/mine/jerboa` (source/stdlib/tests/docs), `~/mine/jerboa-mcp` (cookbooks, divergence, error-fixes, api-signatures) new file mode 100644 --- /dev/null +++ b/TRAINING_FIX_PLAN.md @@ -0,0 +1,136 @@ +# Jerboa-LoRA v3 Retrain Plan + +> Mirrors the corrective plan that took **crystal-lora** from a no-op v1 to a measurably-better v3 (held-out coding eval `36` vs base `9`, similarity `+0.357` vs base `+0.124` tok-lean). Same disease, same cure. + +## v1 diagnosis — same hyperparam pathology as crystal-lora v1 + +Reading the actual axolotl YAMLs and dataset sizes: + +| stage | examples | epochs | LR | LoRA r/α | issue | +|---|---|---|---|---|---| +| CPT | 499 files / ~5MB | 1 | **5.0e-6** | 16/32 | LR ~4× too low for CPT | +| SFT | 38,356 lines / training_data | 2 | 1.0e-4 | 16/32 | Reasonable LR; data is partly redundant | +| DPO | **72** (severity=error only) | 1 | **5.0e-7** | 16/32 | **LR 100× too low, dataset 5× too small, epochs 3× too few** | + +DPO cumulative weight update math (same as crystal v1): +``` +ΔW ≈ steps × lr × ||grad|| × (α/r) + ≈ 18 × 5e-7 × O(1) × 2 + ≈ 2e-5 +``` +Below the BF16 precision floor and **completely inside Q4_K_M quantization noise** (~1–2% per weight). The DPO LoRA delta was erased before Ollama ever saw it — same failure mode as crystal v1. + +**Subjective "the model is good" is mostly the SFT stage + the system prompt + jerboa-mcp tools doing the work.** The DPO stage that's supposed to actively suppress Gerbil/Racket/Clojure/SRFI hallucinations almost certainly didn't move the weights. + +## Corrective plan — Jerboa v3 + +| Stage | v1 (current) | v3 (target) | Why | +|-------|--------------|-------------|-----| +| CPT data | 499 files / ~5 MB | **30K+ records / ~30M tok** | + jerboa-mcp JSON files + docs + stdlib doc-comments | +| SFT data | 38K lines (`training_data*.jsonl`) | **mined + LLM-augmented**, compile-gated | More real, less hand-written / synthetic-stale | +| DPO data | 72 pairs (severity=error from divergence.json) | **300+ pairs** (200+ compile-validated) | 50+ idioms × multiple variants × programmatic generators | +| CPT lr | 5e-6, 1 ep | **2e-5, 2 ep** | v1 was 4× too low | +| DPO lr | 5e-7, 1 ep | **5e-6, 3 ep** | v1 delta was below BF16 floor | +| LoRA r/α | 16 / 32 | **64 / 128** | Bigger delta → survives Q8 quantization | +| Quant target | Q4_K_M | **Q8_0 (latest)** | LoRA contributions actually survive; Q4_K_M kept as secondary tag | + +## Data sources for v3 + +### CPT corpus (`build_cpt_corpus_v2.py` → `cpt_corpus_v3.jsonl`) + +| Source | Path | Why | +|---|---|---| +| Existing `~/mine/jerboa` walk | already covered in v1 | ~5MB baseline | +| **jerboa-mcp JSON data files** | `~/mine/jerboa-mcp/{cookbooks,error-fixes,features,security-rules,api-signatures,divergence,changelog}.json` | ~2.3MB structured Jerboa knowledge — flatten to text | +| **jerboa root docs** | `JERBOA-LANG.md`, `gerbil-like.md`, `libraries.md`, `secure.md`, `slang.md`, `whatchanged.md`, `implement-me.md`, `zig-ideas.md` | Authoritative language docs | +| **jerboa stdlib doc-comments** | `~/mine/jerboa/lib/std/**/*.ss`, `~/mine/jerboa/lib/jerboa/**/*.ss` | Per-module API documentation | +| **jerboa tests** | `~/mine/jerboa/tests/**/*.ss` (301 files) | Real working Jerboa code | +| **jerboa examples** | `~/mine/jerboa/examples/**/*` | Idiomatic usage | +| GitHub jerboa-related repos | `gh search repos --language=scheme jerboa` | Niche language; expect 0–10 repos | + +SHA-dedup within and across sources. One JSONL record per logical file, `text` field with `;; FILE: <path>` header (or `<!-- FILE: ... -->` for markdown). + +### SFT dataset (`build_sft_v3.py` → `sft_v3_mined.jsonl`) + +Mine from real code; compile-gate via **jerboa-mcp tools** (`jerboa_check_syntax`, `jerboa_compile_check`, `jerboa_eval`): + +1. **Cookbook recipes** — `~/mine/jerboa-mcp/cookbooks.json` has ~100 entries each with `{title, tags, imports, code, notes}`. Each → 2–3 SFT pairs: + - "How do I X in Jerboa?" → code block + imports + notes + - "Show me an example of <tag>" → code block + - "What does <function-name> do?" → notes + example +2. **Error fix entries** — `~/mine/jerboa-mcp/error-fixes.json` has 60+ `{pattern, type, message, fix, code_example}` records. Each → 1 SFT pair: + - "I got error: <message>. How do I fix it?" → fix + code_example +3. **Stdlib doc-comments** — extract `;;` blocks immediately preceding `(define …)` / `(def …)` in `lib/std/**/*.ss`. Each → 1 SFT pair templated 5 ways. +4. **Test/spec extractions** — `tests/**/*.ss`. Extract `(test-case "…" …)` blocks (or whatever Jerboa test idiom is — check first). Each → 1 SFT pair. + +Compile-gate via `jerboa_compile_check`: lenient on missing third-party imports, strict on syntax errors / unbound identifiers / wrong arity. Reject any pair whose chosen code fails strict checks. + +### SFT LLM-augmented (`build_sft_llm.py` → `sft_v3_llm.jsonl`, optional) + +Use Claude Haiku 4.5 via OpenRouter (~$15–20 for ~2000 examples). Feed each module's `jerboa_module_exports` list and ask for idiomatic Q/A pairs. Compile-gate identically. Treat as backup if mined corpus is thin. + +### DPO pairs (`build_dpo_pairs_v3.py` → `dpo_pairs_v3.jsonl`) + +Currently `build_dpo_pairs.py` uses only severity=`error` from `divergence.json` → 72 pairs. + +v3 plan: +1. Keep the 72-from-divergence baseline. +2. **Add programmatic rule generators** — one function per Gerbil/Racket/Clojure/SRFI idiom, each emitting 5–10 surface variants: + - Gerbil: `:std/foo` → `(std foo)`, `def-foo` → `defrules`/`def`, `(class …)` → `defclass`, `(defstruct (Foo bar))` (Gerbil) → Jerboa form + - Racket: `(require racket/foo)` → `(import (std foo))`, `define-struct` → `defstruct`, `define/contract` → none/`assert`, `for/list` → `map`/`for-each` + - Clojure: `(defn foo [x] …)` → `(def (foo x) …)`, `let [x …]` → `(let ((x …)) …)`, vector literal `[1 2 3]` → `(list 1 2 3)`, keyword `:foo` → `'foo` + - SRFI: `(srfi :NN)` → `(std …)`, SRFI hash-table API → Jerboa `hash-*` + - Common Lisp: `defun` → `def`, `setf` → `set!`/`hash-put!`, `mapcar` → `map` +3. **Compile-validate chosen** via `jerboa_compile_check` and `jerboa_eval`. Reject pair if chosen fails. +4. Target: ~50 idioms × ~6 variants = ~300 pairs, ~200+ compile-validated. + +### Hyperparam updates (axolotl YAMLs) + +Edit `axolotl_jerboa_{cpt,sft,dpo}.yaml`: + +```yaml +# All three: +lora_r: 64 # was 16 +lora_alpha: 128 # was 32 + +# CPT: +learning_rate: 2.0e-5 # was 5.0e-6 +num_epochs: 2 # was 1 + +# SFT: leave as is (lr=1e-4, epochs=2) + +# DPO: +learning_rate: 5.0e-6 # was 5.0e-7 +num_epochs: 3 # was 1 +``` + +## Eval gate (mandatory before publish) + +Build: +- `eval_holdout.py` — held-out Jerboa coding eval (questions never seen during training, score on idiom hits + `jerboa_compile_check` pass rate). Patterned on crystal's eval_holdout.py. +- `eval_similarity.py` — per-pair similarity to chosen vs rejected on the DPO set. Volume-invariant (Jaccard + char-similarity). The eval to trust. + +Required to publish: trained must beat `qwen3-coder:30b` base on **both** metrics. No exceptions. + +## Estimated cost & duration + +| Step | Time | $ | +|---|---|---| +| Write data builders + eval | 1–2 days | $0 | +| Run data builders locally (compile-gate is the slow part) | 2–6 hrs | $0 | +| LLM SFT augmenter (optional) | overnight | $15–20 | +| RunPod A100 80GB single-pod training (CPT + SFT + DPO + merges) | 8–15 hrs | $80–150 | +| Eval inference (local Ollama, base + trained) | 1–2 hrs | $0 | +| **Total** | **3–4 working days** | **~$100–170** | + +## Order of execution + +1. ✅ This plan doc (you are here) +2. Write `build_cpt_corpus_v2.py` — fastest win, no external deps +3. Write `build_dpo_pairs_v3.py` — programmatic, compile-gated +4. Write `build_sft_v3.py` — mining from cookbooks + error-fixes + stdlib +5. Update axolotl YAMLs (LoRA r=64/α=128, fixed LRs) +6. Build `eval_holdout.py` + `eval_similarity.py` +7. Run all builders → produce `cpt_corpus_v3.jsonl`, `sft_v3_mined.jsonl`, `dpo_pairs_v3.jsonl` +8. `runpod_train.py up && push && train && pull && down` +9. Eval gate. If trained beats base → convert to GGUF Q8_0, push to ollama.com. --- a/TRAINING_PIPELINE.md +++ b/TRAINING_PIPELINE.md @@ -4,6 +4,16 @@ This document explains the **CPT → SFT → DPO** pipeline now driving the Jerb fine-tune, why it replaces the previous single-stage SFT approach, and what each piece is contributing. +> **Update — v3 supersedes the hyperparams cited below.** The pipeline's +> *structure* is unchanged (3 stages, MoE expert targets, same RunPod A100 80GB +> hardware), but the v1 hyperparams in the per-stage sections produced a DPO +> delta below the BF16 precision floor that Q4_K_M quantization erased. v3 +> bumps **LoRA r=64, α=128** across all three stages, **CPT lr 2e-5 / 2 epochs** +> (was 5e-6 / 1), and **DPO lr 5e-6 / 3 epochs** (was 5e-7 / 1). Full +> diagnosis and arithmetic in [TRAINING_FIX_PLAN.md](TRAINING_FIX_PLAN.md). +> v3 is the version live on Ollama. v4 (abliterated base, same hyperparams) +> is in progress. + --- ## TL;DR @@ -17,9 +27,9 @@ each piece is contributing. | Suppresses Gerbil/Racket hallucinations | weak (SFT teaches right form once) | strong (DPO actively penalises wrong form) | | Uses divergence pairs as preferences | ✗ (turned into Q/A — wasted signal) | ✓ (DPO triples) | | Uses raw Jerboa source files | ✗ | ✓ (499 files, 4.3 MB) | -| Total wall clock | ~7 min (Together) / ~17 hr (MLX) | ~6 hr | -| Total cost | ~$3 (Together) / free (MLX) | ~$11 | -| Final val loss (best run) | 1.14 (MLX v1) / 2.08 (Together v3) | TBD — pipeline running now | +| Total wall clock | ~7 min (Together) / ~17 hr (MLX) | ~5–6 hr | +| Total cost | ~$3 (Together) / free (MLX) | ~$11–15 | +| Final val loss (best run) | 1.14 (MLX v1) / 2.08 (Together v3) | beats base on holdout + similarity (v3 shipped) | | Output runs locally on 48 GB Mac | ✓ | ✓ | --- --- a/axolotl_jerboa_cpt.yaml +++ b/axolotl_jerboa_cpt.yaml @@ -5,7 +5,7 @@ # this — SFT teaches "how to answer questions about Jerboa", CPT teaches # "Jerboa exists." Output of this stage is the base for stage 2 (SFT). -base_model: Qwen/Qwen3-Coder-30B-A3B-Instruct +base_model: jaimef21/qwen3-coder-30b-a3b-abliterated model_type: AutoModelForCausalLM tokenizer_type: AutoTokenizer trust_remote_code: false @@ -33,8 +33,8 @@ sequence_len: 2048 sample_packing: true pad_to_sequence_len: true -lora_r: 16 -lora_alpha: 32 +lora_r: 64 +lora_alpha: 128 lora_dropout: 0 lora_target_modules: - q_proj @@ -48,12 +48,12 @@ lora_target_parameters: - experts.gate_up_proj - experts.down_proj -gradient_accumulation_steps: 4 -micro_batch_size: 2 -num_epochs: 1 +gradient_accumulation_steps: 8 +micro_batch_size: 1 +num_epochs: 2 optimizer: paged_adamw_8bit lr_scheduler: cosine -learning_rate: 5.0e-6 +learning_rate: 2.0e-5 warmup_ratio: 0.05 weight_decay: 0.0 --- a/axolotl_jerboa_dpo.yaml +++ b/axolotl_jerboa_dpo.yaml @@ -33,8 +33,8 @@ dataset_processes: 8 sequence_len: 2048 -lora_r: 16 -lora_alpha: 32 +lora_r: 64 +lora_alpha: 128 lora_dropout: 0 lora_target_modules: - q_proj @@ -50,10 +50,10 @@ lora_target_parameters: gradient_accumulation_steps: 4 micro_batch_size: 1 -num_epochs: 1 +num_epochs: 3 optimizer: paged_adamw_8bit lr_scheduler: cosine -learning_rate: 5.0e-7 +learning_rate: 5.0e-6 warmup_ratio: 0.1 weight_decay: 0.0 max_grad_norm: 1.0 --- a/axolotl_jerboa_sft.yaml +++ b/axolotl_jerboa_sft.yaml @@ -36,8 +36,8 @@ sequence_len: 2048 sample_packing: true pad_to_sequence_len: true -lora_r: 16 -lora_alpha: 32 +lora_r: 64 +lora_alpha: 128 lora_dropout: 0 lora_target_modules: - q_proj @@ -51,8 +51,8 @@ lora_target_parameters: - experts.gate_up_proj - experts.down_proj -gradient_accumulation_steps: 4 -micro_batch_size: 2 +gradient_accumulation_steps: 8 +micro_batch_size: 1 num_epochs: 2 optimizer: paged_adamw_8bit lr_scheduler: cosine new file mode 100755 --- /dev/null +++ b/build_cpt_corpus_v2.py @@ -0,0 +1,303 @@ +#!/usr/bin/env python3 +""" +Build the v3 continued-pretraining corpus. + +Expansion over v1 (~5MB / 499 files): + 1. Walk ~/mine/jerboa for .ss/.scm/.md (same as v1) + 2. Flatten ~/mine/jerboa-mcp JSON data files into per-entry text records: + - cookbooks.json → recipe text (title + tags + imports + code + notes) + - error-fixes.json → error-fix text (type + message + explanation + fix + example) + - divergence.json → divergence text (wrong/correct pairs with examples) + - features.json → feature text + - security-rules.json → security pattern text + - api-signatures.json → API signature blocks + 3. SHA-dedup within and across sources + +Output: cpt_corpus_v3.jsonl (one JSON record per file/entry, {"text": "..."}) + +Usage: + python3 build_cpt_corpus_v2.py +""" + +import hashlib +import json +import os +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parent +JERBOA_SRC = Path.home() / "mine" / "jerboa" +JERBOA_MCP = Path.home() / "mine" / "jerboa-mcp" +OUT = REPO / "cpt_corpus_v3.jsonl" + +CODE_EXTS = {".ss", ".scm"} +DOC_EXTS = {".md"} +SKIP_DIRS = {".git", "node_modules", "build", "dist", ".cache", "target", + ".venv", "__pycache__"} +MIN_BYTES = 64 +MAX_BYTES = 256 * 1024 + + +def file_records(root: Path): + """Yield (header, text, source_tag) for each source file under root.""" + if not root.exists(): + return + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS] + for fn in filenames: + p = Path(dirpath) / fn + ext = p.suffix.lower() + if ext not in CODE_EXTS and ext not in DOC_EXTS: + continue + try: + text = p.read_text(encoding="utf-8") + except (UnicodeDecodeError, OSError): + continue + size = len(text.encode("utf-8")) + if size < MIN_BYTES or size > MAX_BYTES: + continue + rel = p.relative_to(root) + if ext in CODE_EXTS: + header = f";; FILE: {rel}\n" + tag = "code" + else: + header = f"<!-- FILE: {rel} -->\n" + tag = "doc" + yield header, text, tag + + +def cookbook_records(path: Path): + if not path.exists(): + return + data = json.loads(path.read_text()) + for r in data: + title = r.get("title", "").strip() + rid = r.get("id", "").strip() + tags = r.get("tags", []) + imports = r.get("imports", []) + code = r.get("code", "").rstrip() + notes = r.get("notes", "").strip() + if not code: + continue + parts = [f";; COOKBOOK: {rid} — {title}"] + if tags: + parts.append(f";; tags: {', '.join(tags)}") + if imports: + parts.append(f";; imports: {', '.join(imports)}") + parts.append("") + parts.append(code) + if notes: + parts.append("") + parts.append(f";; notes: {notes}") + yield "\n".join(parts) + "\n", "cookbook" + + +def error_fix_records(path: Path): + if not path.exists(): + return + data = json.loads(path.read_text()) + for r in data: + rid = r.get("id", "").strip() + typ = r.get("type", "").strip() + msg = r.get("message", "").strip() + explain = r.get("explanation", "").strip() + fix = r.get("fix", "").strip() + code = r.get("code_example", "").rstrip() + parts = [f";; ERROR-FIX: {rid} — {typ}"] + if msg: + parts.append(f";; {msg}") + if explain: + parts.append(f";; explanation: {explain}") + if fix: + parts.append(f";; fix: {fix}") + if code: + parts.append("") + parts.append(code) + yield "\n".join(parts) + "\n", "error_fix" + + +def divergence_records(path: Path): + if not path.exists(): + return + data = json.loads(path.read_text()) + entries = data["entries"] if isinstance(data, dict) else data + for r in entries: + rid = r.get("id", "").strip() + wrong = r.get("wrong", "").strip() + correct = r.get("correct", "").strip() + wrong_ex = r.get("wrong_example", "").rstrip() + correct_ex = r.get("correct_example", "").rstrip() + sev = r.get("severity", "").strip() + notes = r.get("notes", "").strip() + sources = r.get("wrong_source") or [] + if not correct_ex: + continue + parts = [ + f";; DIVERGENCE: {rid} — {sev}", + f";; wrong ({'/'.join(sources) or 'other'}): {wrong}", + f";; correct (jerboa): {correct}", + ] + if wrong_ex: + parts.append(f";; wrong-example:\n;; {wrong_ex.replace(chr(10), chr(10) + ';; ')}") + parts.append("") + parts.append(correct_ex) + if notes: + parts.append("") + parts.append(f";; notes: {notes}") + yield "\n".join(parts) + "\n", "divergence" + + +def feature_records(path: Path): + if not path.exists(): + return + data = json.loads(path.read_text()) + items = data if isinstance(data, list) else data.get("features", []) + for r in items: + rid = r.get("id", r.get("name", "")).strip() if isinstance(r, dict) else "" + if not isinstance(r, dict): + continue + text_parts = [f";; FEATURE: {rid}"] + for k, v in r.items(): + if k in ("id", "name"): + continue + if isinstance(v, str) and v.strip(): + text_parts.append(f";; {k}: {v}") + elif isinstance(v, list) and v: + text_parts.append(f";; {k}: {', '.join(map(str, v))}") + if len(text_parts) > 1: + yield "\n".join(text_parts) + "\n", "feature" + + +def security_records(path: Path): + if not path.exists(): + return + data = json.loads(path.read_text()) + items = data if isinstance(data, list) else data.get("rules", []) + for r in items: + if not isinstance(r, dict): + continue + rid = r.get("id", "").strip() + title = r.get("title", r.get("name", "")).strip() + sev = r.get("severity", "").strip() + desc = r.get("description", r.get("message", "")).strip() + fix = r.get("fix", r.get("recommendation", "")).strip() + bad = r.get("bad_example", r.get("vulnerable_example", "")).rstrip() + good = r.get("good_example", r.get("safe_example", "")).rstrip() + parts = [f";; SECURITY: {rid} — {title}"] + if sev: + parts.append(f";; severity: {sev}") + if desc: + parts.append(f";; description: {desc}") + if fix: + parts.append(f";; fix: {fix}") + if bad: + parts.append("") + parts.append(";; bad:") + parts.append(bad) + if good: + parts.append("") + parts.append(";; good:") + parts.append(good) + if len(parts) > 1: + yield "\n".join(parts) + "\n", "security"