add lora.md
ober
d0f0b5dd32836c11c0840605c611a8c183ae9468
new file mode 100644 --- /dev/null +++ b/lora.md @@ -0,0 +1,863 @@ +# LoRA + RAG: A Working Programmer's Reference + +This is a study document. It explains the math, the metrics, the moving parts, and the design considerations behind the pipeline in this repo (`gerbil-lora`) — plus what worked, what failed, and why, across three sibling projects: `~/mine/jerboa-lora`, `~/mine/crystal-lora`, and this one. + +**Audience.** You write code. You understand variance, distributions, skew, kurtosis, sampling. You're comfortable with linear algebra at the level of "matrices have shapes and multiplication is `(m×n)·(n×k) → m×k`." You are *not* fluent in optimization theory, information theory, or differential geometry. You don't need to be. The math below is presented at the level where you can reason about the trade-offs without rederiving anything. + +**How to read this.** Front-to-back the first time. After that, treat the table of contents as an index — each section is meant to stand alone. + +--- + +## Table of Contents + +1. [The big picture](#1-the-big-picture) +2. [The LoRA math](#2-the-lora-math) +3. [Stage 1 — CPT (Continued Pre-Training)](#3-stage-1--cpt-continued-pre-training) +4. [Stage 2 — SFT (Supervised Fine-Tuning)](#4-stage-2--sft-supervised-fine-tuning) +5. [Stage 3 — DPO (Direct Preference Optimization)](#5-stage-3--dpo-direct-preference-optimization) +6. [Every metric you'll see, decoded](#6-every-metric-youll-see-decoded) +7. [Hyperparameter design space](#7-hyperparameter-design-space) +8. [MoE-specific gotchas](#8-moe-specific-gotchas) +9. [Quantization and the LoRA-delta-survival problem](#9-quantization-and-the-lora-delta-survival-problem) +10. [Deployment formats](#10-deployment-formats) +11. [RAG fundamentals](#11-rag-fundamentals) +12. [Fine-tune vs RAG: when to pick what](#12-fine-tune-vs-rag-when-to-pick-what) +13. [Lessons from three projects](#13-lessons-from-three-projects) +14. [Reading list and next steps](#14-reading-list-and-next-steps) + +--- + +## 1. The big picture + +### What we're trying to do + +Take a general-purpose 30B-parameter language model (`Qwen3-Coder-30B-A3B-Instruct`) and *specialise* it — make it good at Gerbil Scheme, a niche dialect it has only seen in trace amounts. We have three knobs: + +- **Continued pre-training (CPT).** Show it more raw Gerbil source, like you'd show a junior engineer a codebase. No questions, no answers, just text. +- **Supervised fine-tuning (SFT).** Show it good Q/A pairs. "When asked X, the right answer looks like Y." +- **Direct preference optimization (DPO).** Show it pairs of (good answer, bad answer) and tell it: prefer the good one, push the bad one's probability down. + +Then we evaluate. If the trained model beats the base model on held-out questions, ship it. + +### Why not just retrain from scratch? + +A 30B model has ~30 billion parameters. Training one from scratch costs ~$10M+ in GPU-hours. We can't and shouldn't. We need to *adapt* the existing weights. + +### Why not just fine-tune all 30B parameters? + +Two reasons: + +1. **Memory.** Full fine-tuning at BF16 (16 bits per parameter) needs ~60 GB just for weights, plus another ~60 GB for gradients, plus another ~120 GB for the optimizer state (Adam keeps two moments per parameter at FP32). Total: ~240 GB. A single A100 has 80 GB. +2. **Catastrophic forgetting.** Touching every weight tends to overwrite useful general knowledge. You wanted the model to learn Gerbil and stay good at everything else. + +**LoRA** (Low-Rank Adaptation) is the answer to both. + +### What is LoRA in one paragraph + +Don't change the original weights. Instead, *add* a tiny correction `ΔW` to each layer's weight matrix, where `ΔW = B · A`, and `B` and `A` are small. If the original weight is shape `(d_out, d_in)` — say `(4096, 4096) = 16.8M params` — then `A` has shape `(r, d_in)` and `B` has shape `(d_out, r)`, where `r` (the **rank**) is small (typical: 16, 32, 64). Total parameters trained: `r × (d_in + d_out) = 32 × 8192 ≈ 262K`. **64× smaller, and we still get most of the expressiveness.** + +The intuition: most useful corrections to a pretrained weight matrix are low-rank. You aren't rewriting the function; you're nudging it in a few directions. + +--- + +## 2. The LoRA math + +### The decomposition + +Let `W₀ ∈ ℝ^(d_out × d_in)` be the frozen pretrained weight. LoRA replaces `W₀` with: + +``` +W_effective = W₀ + (α/r) · B · A +``` + +where: + +- `A ∈ ℝ^(r × d_in)`, initialized from a Gaussian (`N(0, 1/r)` typically). +- `B ∈ ℝ^(d_out × r)`, **initialized to zero**. +- `r` is the rank (16/32/64 most common). +- `α` is the *alpha* scalar — a hyperparameter independent of `r`. + +**Why is `B` initialized to zero?** So that at step 0, `B·A = 0` and the effective weight equals `W₀` exactly. The model behaves identically to the base model. Then gradient descent slowly grows `B`. + +**Why the `α/r` scaling?** Without it, doubling the rank `r` doubles the magnitude of `BA`. The `α/r` factor makes the effective learning rate of the adapter independent of `r` — so you can change `r` without re-tuning your learning rate. By convention, set `α = 2r` (so `α/r = 2`) as a starting point. We use `r=32, α=64`. + +### Rank: the actual concept + +`rank(BA)` is at most `r`. If a real `ΔW` requires rank 200 to express, then LoRA-r=32 *cannot* represent it. You'd be projecting a 200-dimensional update onto a 32-dimensional subspace. + +**Is this a problem?** In practice, mostly no. The LoRA paper showed that adaptation deltas during fine-tuning have approximately low effective rank — even when you fine-tune all parameters, the useful information in the resulting `ΔW` has rank ~8-64 for most tasks. Higher ranks help for harder adaptations (new languages, large domain shifts). + +### How rank scales with task difficulty + +From the three projects in `~/mine/*-lora`: + +| Project | Final rank | Adaptation difficulty | +|---|---|---| +| `crystal-lora v1` | 16 | Modest — Crystal is close to Ruby in syntax | +| `crystal-lora v3` | 64 | After Q4 quantization erased r=32 signal | +| `gerbil-lora v3` (this) | 32 | Distinct dialect, no surface overlap with Crystal | +| `jerboa-lora v3` | 32–64 | Larger custom dialect, more anti-idioms | + +Rule of thumb: start at `r=32`. If post-quantization (Q4_K_M) the LoRA signal vanishes, you have two choices: (a) raise `r` to 64; (b) ship a higher-fidelity quant (Q8_0). Either fixes it. Both have memory costs. + +### Target modules — which weights does LoRA touch? + +A transformer block has four big linear weight matrices in attention (`q_proj`, `k_proj`, `v_proj`, `o_proj`) and three in the MLP (`gate_proj`, `up_proj`, `down_proj`). The LoRA paper originally targeted only `q_proj` and `v_proj` — the attention projections — because that was the cheapest move. + +That's wrong for serious domain adaptation. Empirically: + +- **Attention only**: cheapest, but you cap the model's ability to learn new *content*. Together AI's hosted LoRA path only supports attention-only and that's why jerboa-lora v1 plateaued at val loss 2.08 — there's a hard ceiling. +- **Attention + MLP**: the right default. The MLP modules carry most of the model's *knowledge*. If you're teaching new domain content (Gerbil syntax, Crystal stdlib), you need the MLPs. +- **+ MoE experts**: required for Qwen3-Coder-30B-A3B. The MoE expert matrices contain most of the model's parametric knowledge. Skipping them silently is one of the most insidious failure modes in this whole pipeline (see [§8](#8-moe-specific-gotchas)). + +This repo targets all of them: +```yaml +lora_target_modules: + - q_proj + - k_proj + - v_proj + - o_proj + - gate_proj + - up_proj + - down_proj +lora_target_parameters: + - experts.gate_up_proj + - experts.down_proj +``` + +### Merging — when adaptation becomes weights + +After training, you have `W₀` (frozen) + `B·A` (the adapter, small). You can either: + +1. **Ship the adapter separately.** Loader applies `W₀ + BA` at inference. Lower disk footprint. Required for hot-swappable multi-LoRA serving. +2. **Merge.** Compute `W_new = W₀ + (α/r)·B·A` and save the result as a regular checkpoint. Looks identical to a normal model. Required for GGUF/MLX conversion. Required for stage chaining (CPT → SFT → DPO each merge before the next stage's adapter trains). + +**Why stage chaining requires merging:** if you stack adapters (CPT adapter + SFT adapter + DPO adapter), each new adapter trains *on top of* a model that still includes the prior adapter's contribution. Gradients flow through that contribution, and you can end up with adapter-on-adapter interactions that are hard to reason about. Merging between stages collapses each adapter into the frozen base, giving the next stage a clean slate. This is what `merge_method: legacy` in `axolotl_gerbil_*.yaml` does — it calls peft's `merge_and_unload()` which produces a flat BF16 checkpoint. + +(`merge_method: legacy` specifically is a workaround for an axolotl quirk: the default merge method scans for `lora_target_parameters` and assumes any non-empty list means you're using AdaLoRA, then refuses to merge with a confusing error. `legacy` bypasses that check.) + +--- + +## 3. Stage 1 — CPT (Continued Pre-Training) + +### What it is + +You hand the model raw text and ask it to predict the next token. This is exactly what the base model was originally trained on, just continued — hence the name. No structure, no Q/A format, no special tokens beyond whatever the tokenizer normally inserts. + +### Why we do it + +Three reasons, in order of importance: + +1. **Token distribution.** The model has seen Gerbil source maybe a few times in its 10T-token pretraining mix. We're going to teach it that this dialect exists and what its surface looks like. Which imports cluster with which idioms. What `;;` doc-comments above `(def ...)` look like. How `(import :std/sort)` is spelled versus how Racket spells `(require ...)`. + +2. **Prior on form.** When the model later answers "how do I sort a list in Gerbil?", it needs the *prior* that `:std/sort` is the right namespace, not `racket/sort` or `clojure.core`. CPT loads that prior into the MLP weights. SFT alone can't, because SFT shows the model good answers but doesn't train on the *substrate* the answers live in. + +3. **Regularization against catastrophic forgetting.** Counter-intuitive: more training, more careful training, *protects* the model. With a low LR (we use `2e-5`) over diverse-ish data, CPT acts as a regulariser that prepares the weights for the harder SFT/DPO that follow. + +### The loss function + +Standard autoregressive language modeling loss. For a sequence of tokens `t_1, t_2, ..., t_N`: + +``` +L = -1/N · Σ log P(t_i | t_1, ..., t_{i-1}) +``` + +Read: "minus the average log-probability the model assigns to each next token, given everything before it." + +This is cross-entropy. If the model assigns probability 0.5 to the actual next token, the loss for that token is `-log(0.5) ≈ 0.693`. If it assigns 0.9, the loss is `-log(0.9) ≈ 0.105`. Lower loss = better predictions. + +### What "low LR" means here + +CPT learning rate: `2e-5` (0.00002). + +Compare: +- SFT LR: `1e-4` (5× higher) +- DPO LR: `5e-6` (4× lower than CPT) +- Pretraining LR: typically `1e-4` to `3e-4` + +CPT's job is to nudge the distribution, not rewrite it. Too high an LR and you damage general capabilities. Too low and nothing happens. `2e-5` is a well-validated sweet spot for continued pretraining at ~10K-100K examples. + +### Data format + +Axolotl's `type: completion`: each record is `{"text": "..."}`. No prompt/response split. Just text. + +Our CPT corpus (`cpt_corpus_v3.jsonl`, 22.7 MB): +- ~3,758 records +- Walks `~/mine/gerbil`, `~/mine/gambit`, `~/mine/gerbil-mcp` +- File contents prefixed with `;; FILE: path/to/file.ss` so the model learns that imports/definitions cluster within file boundaries +- Markdown docs included +- JSON-flattened cookbook entries (each entry → one text record with metadata as a header comment) + +### What success looks like + +- **Training loss** drops steadily from initial value to plateau. Initial value depends on how close the data is to the base model's training distribution; for niche dialects, expect to start around 0.8-1.2 and settle around 0.4-0.7. +- **Validation loss** (eval_loss) follows training loss with a slight lag, and ideally doesn't diverge. If train_loss keeps dropping but eval_loss starts climbing — you're overfitting. Stop earlier or lower the LR. +- **No catastrophic NaN**. If gradients explode (grad_norm spikes to thousands), you need lower LR or gradient clipping. + +For Gerbil v3 right now: initial eval_loss `0.9356`, step-10 train_loss `0.9143` (2.3% drop after 10 steps of 794) — the signal is good. We expect train_loss in the 0.4-0.6 range by the end of 2 epochs. + +--- + +## 4. Stage 2 — SFT (Supervised Fine-Tuning) + +### What it is + +Show the model examples of `(instruction, ideal response)` pairs. Train it to produce the response when given the instruction. Format-wise, this is wrapped in a chat template — the tokenizer inserts special tokens like `<|im_start|>user`, `<|im_start|>assistant`, etc. + +### Loss function + +**Identical to CPT mathematically** — still cross-entropy on next-token prediction. The only difference: **loss is masked on the instruction tokens**. We only train the model to predict the *response*, conditional on the instruction. The instruction's tokens contribute zero gradient. + +In axolotl, this is the `train_on_inputs: false` setting (default). + +### Why mask instruction tokens? + +Two reasons: + +1. **Efficiency.** You don't have ground-truth probability for "user instructions" — they're free-form prompts. Training on them just teaches the model to mimic typical user phrasing, which it already does fine. +2. **Avoiding regression.** Training on instructions can degrade the model's general capability to respond to *novel* phrasings, because you're saying "this exact wording is canonical." Bad. + +### Data format + +Axolotl's `type: chat_template`. Each record is structured as a conversation: + +```json +{"messages": [ + {"role": "user", "content": "How do I parse JSON in Gerbil?"}, + {"role": "assistant", "content": "Import :std/text/json and use string->json-object..."} +]} +``` + +Axolotl applies the tokenizer's chat template (Qwen's, in our case) to render this into tokens. The `<|im_start|>assistant` token marks where the loss-mask flips on. + +### Hyperparameters + +| Param | Value | Why | +|---|---|---| +| LR | `1e-4` | 5× CPT. SFT has fewer examples and we want clearer behavior change. | +| Epochs | 2 | More than 1 helps the model lock in patterns; >3 starts overfitting on small datasets. | +| Micro batch | 1 | A100 80GB barely fits this with sample_len=2048. | +| Grad accum | 4 | Effective batch = 4 examples per step. | +| Warmup ratio | 0.1 | First 10% of steps linearly ramp LR from 0 to peak. Prevents early divergence. | +| LR scheduler | cosine | After warmup, decay LR following cosine curve to ~0 at end. Standard. | + +### What "warmup" does, concretely + +Adam-family optimizers have running statistics (first and second moments of gradients). At step 0 these stats are at their initial values (zeros or biased estimates) and the optimizer's update direction is noisy. Stepping at full LR with noisy updates causes the model to thrash. Warming up — starting LR near zero and ramping over 10% of training — gives the optimizer time to estimate good gradient statistics before taking big steps. + +For tiny datasets you can sometimes skip warmup. For anything serious, don't. + +### What success looks like + +- **Train loss drops faster than CPT** because LR is higher and examples are more structured. +- **Eval loss tracks train loss closely.** If eval diverges, you have too few examples or LR is too high. +- **Response quality on held-out prompts is *visibly* better.** Run the eval script (`eval_holdout.py`) between epochs if you can. + +For Gerbil v3: SFT input data is 2,391 examples (cookbooks × 2 phrasings + error fixes + resource markdown + stdlib doc-comments). Expected behavior: train loss starts in the 1.5-2.0 range (chat formatting alone adds noise), drops to 0.6-0.9 by end of epoch 2. + +--- + +## 5. Stage 3 — DPO (Direct Preference Optimization) + +This is where the math gets interesting. Pay attention. + +### The problem DPO solves + +After SFT, the model knows what good answers look like. But it still has a *prior* toward bad answers it learned during pretraining. If you ask "sort a list in Gerbil," SFT teaches it the good form `(sort lst <)`. But the bad form `(sort < lst)` (Racket argument order) might still have nontrivial probability. + +SFT can't fix this. SFT's loss only pulls probability *up* on `chosen` responses. It says nothing about pushing `rejected` probability *down*. The rejected form keeps its prior intact. + +**DPO is contrastive.** It trains the model to prefer `chosen` over `rejected` directly: `P(chosen) > P(rejected)`. + +### The math — derived gently + +DPO comes from RLHF (Reinforcement Learning from Human Feedback). RLHF's setup: + +1. Train a reward model `r(x, y)` on preference data (`(prompt, chosen, rejected)` triples). +2. Use that reward model to do PPO (a fancy RL algorithm) on the LLM, pushing it to maximize reward. + +DPO's insight: **you can collapse the reward model into the LLM itself.** The math (which I'll spare you in detail; the paper is "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" by Rafailov et al., 2023) shows that under standard RLHF assumptions, the optimal policy `π*` is related to the reference policy `π_ref` (your SFT model) by: + +``` +π*(y | x) ∝ π_ref(y | x) · exp(r(x, y) / β) +``` + +where `β` is a temperature-like parameter (KL coefficient). This says the optimal model is the reference model, "tilted" by exponentiated reward. + +If you take logarithms and rearrange, you get an *implicit* reward expressed in terms of the model itself: + +``` +r(x, y) = β · log [π(y|x) / π_ref(y|x)] +``` + +Now plug this into the Bradley-Terry preference model (`P(chosen ≻ rejected) = σ(r_chosen - r_rejected)`, where σ is sigmoid): + +``` +L_DPO = -E[ log σ(β · (log π(c|x)/π_ref(c|x) - log π(r|x)/π_ref(r|x))) ] +``` + +That's the DPO loss. Read in English: **the model should assign higher relative log-probability to the chosen response than to the rejected response, where "relative" is measured against a frozen reference model.** + +### What the parts mean + +- **`π`**: the model currently being trained. +- **`π_ref`**: a frozen copy of the SFT model. Held fixed. Provides the baseline. +- **`β` (beta)**: KL strength. High β = stay close to π_ref (conservative); low β = wander further (aggressive). Typical value `0.1`. Axolotl default. +- **`log π(c|x)`**: log-probability of the chosen response under the trained model. Computed token-by-token, summed. +- **`σ`**: sigmoid function `1/(1+e^-x)`. Squashes logits into (0, 1). + +### The implicit reward and KL penalty + +The clever thing is that the KL-divergence constraint (don't drift too far from π_ref) is *built into the loss function*. You're not adding a regularization term — the log-ratio form `log π/π_ref` *is* the KL penalty in disguise. That's why DPO is stable: it can't run away from the SFT model the way naive RL would. + +### Hyperparameters — DPO is sensitive + +| Param | Value | Why | +|---|---|---| +| LR | `5e-6` | **20× lower than SFT.** DPO can collapse the model quickly. Higher LRs → catastrophic mode collapse, where the model starts emitting garbage to lower rejected probability. | +| Epochs | 3 | DPO data is small (66 pairs in our case). 3 epochs gives enough exposure. | +| β | `0.1` | KL strength. | +| Warmup | 0.1 | Same reasoning as SFT. | + +**The LR is the danger zone.** Jerboa-lora v1 used DPO LR `5e-7` × 18 steps. Total parameter update was below the noise floor of BF16 — the gradient signal was real but the magnitudes were so small that after quantizing to Q4_K_M, the LoRA delta was rounded to zero. Result: a "trained" model that behaved identically to the base. Crystal-lora v3 corrected to `5e-6`. That's the right floor. + +If you go higher (`1e-5`+) you start collapsing the model. If you go lower (`1e-6`-) you lose the delta to quantization. Five-times-ten-to-the-minus-six is the sweet spot. Stay there. + +### Data format + +```json +{ + "prompt": "Sort a list ascending in Gerbil.", + "chosen": "```scheme\n(import :std/sort)\n(sort '(3 1 2) <)\n```", + "rejected": "```scheme\n(sort < '(3 1 2))\n```" +} +``` + +Axolotl's `rl: dpo` mode. Both `chosen` and `rejected` are rendered through the chat template; loss is computed on the assistant turn only. + +### Quality of preference pairs >> volume of preference pairs + +This is the most important fact about DPO and probably the most under-appreciated. + +- 66 high-quality, programmatically-validated pairs (this repo) **outperform** 1000 noisy pairs. +- Every `chosen` response in `dpo_pairs_v3.jsonl` is run through `gxi` (Gerbil interpreter) and rejected if it has syntax errors, missing imports, or compile failures. Runtime errors from placeholder free variables (`compute`, `ht`) are accepted because they're intentional in idiom-demo snippets. +- This validation eliminates ~20-30% of generated pairs. The remaining ones are *trusted* signal. + +Crystal-lora made the same point: compile-gating eliminated 36% of LLM-augmented Ruby-Crystal pairs as bugs. Without that filter, you train the model to prefer broken code over slightly-worse-but-correct code. Catastrophic. + +### What success looks like — the DPO metrics + +These appear in DPO training logs and nowhere else. + +- **`rewards/chosen`**: average implicit reward for chosen responses. Starts near 0, grows positive (chosen probability increases relative to π_ref). +- **`rewards/rejected`**: average implicit reward for rejected responses. Starts near 0, grows *negative* (rejected probability decreases relative to π_ref). +- **`rewards/margin`**: `rewards/chosen - rewards/rejected`. Should grow monotonically. This is the headline metric. Larger = bigger preference gap learned. +- **`rewards/accuracies`**: fraction of pairs where the model now prefers chosen over rejected (i.e., `log π(c) > log π(r)`). Should approach 1.0. If you plateau at 0.5, DPO isn't learning. + +For Gerbil v3 (when we hit it): expect `rewards/accuracies` to hit 0.85-0.95 by end of 3 epochs. `rewards/margin` should grow from ~0 to ~2-3. + +--- + +## 6. Every metric you'll see, decoded + +Training logs are dense. Here's what each number means and what range is normal. + +### Loss metrics + +| Metric | What it is | "Good" value | Watch for | +|---|---|---|---| +| `loss` / `train_loss` | Cross-entropy on the training batch, averaged | Depends on stage. CPT: 0.4-0.9. SFT: 0.5-1.5. DPO: 0.3-0.7. | Must decrease overall. NaN = blowup. | +| `eval_loss` | Same metric on held-out eval set | Slightly higher than train_loss | Diverging from train_loss = overfitting | +| `grad_norm` | L2 norm of the gradient vector before clipping | 0.5-3.0 typical | Spikes >10 = instability. Sustained >100 = LR too high. | +| `learning_rate` | Current LR after scheduler | Curves smoothly from warmup → peak → decay | Should match your scheduler config | +| `epoch` | Fractional epoch | Counts up to `num_epochs` | Just a progress indicator | + +### Perplexity (PPL) + +`PPL = exp(loss)`. It's just loss in a different unit, more interpretable: "average number of equally-likely choices the model is picking between for each token." Loss 0.69 ↔ PPL 2.0 = "model is equally torn between 2 plausible next tokens on average." + +You won't see PPL in axolotl logs directly. Compute it from eval_loss: `exp(0.93) ≈ 2.54` = base model has ~2.54-way uncertainty on Gerbil tokens. Successful CPT should drive that down to 1.7-2.0 (loss 0.5-0.7). + +### Throughput metrics + +| Metric | What it is | Typical | +|---|---|---| +| `train_runtime` | Wall clock seconds since start | accumulating | +| `train_samples_per_second` | Examples processed per second | A100 80GB on 30B model: 0.1-0.3 | +| `train_steps_per_second` | Optimizer steps per second | 0.05-0.1 typical (batch=1, accum=4 → 4 samples per step) | +| `it/s` (tqdm bar) | Iterations (forward+backward passes) per second | 0.05-0.1, same as steps_per_second | + +For our current run: 794 steps total, 17.5s per step, ETA ~3h35m for CPT. + +### DPO-specific metrics + +(Covered in §5. Repeating the table for cross-reference.) + +| Metric | Range | Direction | +|---|---|---| +| `rewards/chosen` | 0 → +2 | Up | +| `rewards/rejected` | 0 → -2 | Down | +| `rewards/margin` | 0 → 3+ | Up monotonically | +| `rewards/accuracies` | 0.5 → 0.9+ | Up toward 1.0 | + +### Memory metrics + +GPU memory is the constant constraint. Track: + +- **GPU memory used** (`nvidia-smi`): should be 70-95% of total. Below 50% = waste; you could increase micro_batch or sequence_len. +- **Out-of-memory crashes**: drop micro_batch first, then sample_len, then unfreeze gradient_checkpointing if not already on. + +For our pod: 75.8 / 80 GiB used. No room to grow batch size on this card. Would need H200 (141 GB) or 2× A100 with model-parallel sharding for bigger batches. + +### How to read tqdm progress bars in the training log + +``` +1%| | 5/794 [02:24<4:49:08, 21.99s/it] +``` + +- `1%` and `5/794`: current step / total steps. Total = `(dataset_size / effective_batch) * num_epochs`. +- `02:24`: elapsed since training started. +- `4:49:08`: ETA. +- `21.99s/it`: seconds per step *at that moment*. Smooths out as steps proceed. + +If `s/it` is *growing*, you have a problem (data loader stalling, swap, etc.). If it's stable or slowly decreasing as the dataset's longest sequences are processed first, that's normal. + +--- + +## 7. Hyperparameter design space + +### The hyperparameters you actually choose + +Out of dozens of knobs, these are the ones with real impact: + +| Knob | Range | Default | Bigger means | +|---|---|---|---| +| `learning_rate` | 1e-6 to 5e-4 | 1e-4 (SFT) | More movement, more risk of instability | +| `num_epochs` | 1-5 | 2-3 | More learning, more overfit risk | +| `lora_r` (rank) | 8-128 | 32 | More capacity, more compute, more memory | +| `lora_alpha` | r to 4r | 2r = 64 | Stronger adapter, similar to higher LR | +| `lora_target_modules` | subset | all 7 + experts | Where adaptation can happen | +| `micro_batch_size` | 1-32 | 1 | More memory needed; smoother gradients | +| `gradient_accumulation_steps` | 1-32 | 4 | Larger effective batch, slower steps | +| `sequence_len` | 512-8192 | 2048 | Memory grows quadratically with attention | +| `warmup_ratio` | 0-0.2 | 0.1 | Slower early training, more stable | +| `lr_scheduler_type` | linear/cosine/constant | cosine | Decay shape | +| `weight_decay` | 0-0.1 | 0 (LoRA usually) | L2 regularization on adapters | + +### Effective batch size + +`effective_batch = micro_batch × gradient_accumulation_steps × num_GPUs` + +For us: `1 × 4 × 1 = 4`. Tiny. We get away with it because LoRA has few parameters and the SGD noise from small batches isn't catastrophic. Full fine-tuning typically needs batch 64-256. + +### Sequence length and quadratic memory + +Self-attention scales as `O(seq_len²)` in memory and compute. Doubling `seq_len` from 2048 to 4096 quadruples attention memory. For long-context training you need flash_attention (which our config has: `flash_attention: true`) which reduces this to `O(seq_len)` memory by recomputing attention in tiles. Even with flash, throughput halves. + +### Gradient checkpointing + +We have `gradient_checkpointing: true`. What it does: during the forward pass, instead of storing every intermediate activation, store only checkpoint activations and *recompute* others during backward pass. Memory savings: ~50%. Cost: ~25% throughput slowdown. + +Worth it when you'd otherwise OOM. We need it on this MoE model. + +### When to change rank + +- **Default `r=32, α=64`.** Start here. +- **`r=16` if** you have <1K examples and a small distribution shift (closely related dialects). Crystal v1 with r=16 was fine on Ruby-adjacent Crystal patterns. +- **`r=64` if** you're shipping at Q4_K_M and r=32 produced no behavior change. Or if you have a large adaptation surface (many anti-idioms, large vocabulary shift). +- **`r=128+` rarely justified.** Compute scales linearly; you're approaching full fine-tuning territory. At that point consider full fine-tune with QLoRA tricks. + +### When to change LR + +If train_loss is flat across many steps: LR too low. Multiply by 2-3. +If train_loss spikes or goes NaN: LR too high. Halve. +If grad_norm sustained >10: LR too high. +If eval_loss diverges from train_loss: epochs too long OR LR too high — try lowering LR before reducing epochs. + +--- + +## 8. MoE-specific gotchas + +Qwen3-Coder-30B is a **sparse mixture-of-experts** model. Each MLP block has multiple "expert" sub-networks; for each token, a gating network picks which 2-8 experts to route through. Total parameters: 30B. Active parameters per forward pass: ~3B. + +### Why this matters for LoRA + +The expert matrices contain most of the model's parametric knowledge. If you LoRA-target only attention + a couple MLP modules, you're applying adapters to the *router* and the routing logits, but not to the actual experts that produce the outputs. Result: the adapter has very little leverage over what the model says. + +### The fused expert layout + +In `Qwen3-Coder-30B-A3B-Instruct`, the experts are stored as *fused* tensors: + +- `experts.gate_up_proj` — one big tensor of shape `(num_experts, d_model, 2*intermediate)` (gate and up concatenated) +- `experts.down_proj` — one big tensor of shape `(num_experts, intermediate, d_model)` + +This is for efficiency: at inference, you do one big batched matmul over all experts rather than looping over experts one-by-one. But it means LoRA libraries that expect to find Linear modules with `.weight` and `.bias` *don't find them* — these are raw parameters, not modules. + +That's why our config has: + +```yaml +lora_target_parameters: + - experts.gate_up_proj + - experts.down_proj +``` + +`lora_target_parameters` (versus `lora_target_modules`) tells PEFT to apply LoRA directly to these tensor parameters, treating each as if it were a single Linear weight. + +### The Heretic v1.3.0 silent failure (jerboa-lora lesson) + +Earlier work in `~/mine/jerboa-lora` tried to use Heretic v1.3.0 for the LoRA injection. It had code like: + +```python +for expert in layer.mlp.experts: + inject_lora(expert.gate_up_proj) +``` + +But `layer.mlp.experts` is the fused parameter, not a list — it raises `TypeError`. The error was *swallowed* by a top-level `with suppress(Exception):` block. Training proceeded with experts completely *un-adapted*. The model looked like it was training (loss decreased, grad_norm normal) but the adapter was attached only to attention. Result: trained model behaved identically to a vanilla SFT-on-attention run. + +**The lesson:** verify your target modules are actually being adapted. Print `model.print_trainable_parameters()` before training starts. The count should be hundreds of millions for r=32 on a 30B MoE, not millions. + +### MoE forward-pass memory + +PEFT materializes the LoRA delta `BA` *at FP32* during the forward pass for numerical stability. For fused MoE experts where the underlying tensor is huge, this materialization can blow VRAM unexpectedly. The symptom: training runs for 5-10 steps then OOMs on a forward pass with longer sequences. + +Workarounds: +- Lower `sequence_len`. +- Gradient checkpointing (already on). +- Move to a larger GPU (H200 instead of A100). +- Accept it and rerun with smaller sequence_len. + +For our pod: 75.8 GB used with `sequence_len: 2048`. We're at the edge. + +### The AdaLoRA false positive + +Axolotl's default merge code does: + +```python +if config.lora_target_parameters: + raise "Detected AdaLoRA — refusing to merge" +``` + +This is a false positive — having non-empty `lora_target_parameters` is required for fused MoE experts, but axolotl thought it meant AdaLoRA (a different variant where rank adapts per-layer). The workaround: + +```yaml +merge_method: legacy +``` + +This tells axolotl to skip its own merge logic and call peft's `merge_and_unload()` directly, which works fine. + +--- + +## 9. Quantization and the LoRA-delta-survival problem + +### Quantization recap + +We train at BF16 (16-bit floats, 7-bit mantissa, 8-bit exponent — same dynamic range as FP32 but less precision). We deploy at lower precision to save memory and accelerate inference. Common targets: + +| Format | Bits/param | 30B model size | Quality | +|---|---|---|---| +| BF16 | 16 | 60 GB | Baseline (training format) | +| FP16 | 16 | 60 GB | Less dynamic range than BF16; legacy | +| Q8_0 (GGUF) | ~8 | 32 GB | ~Identical to BF16 in practice | +| Q6_K (GGUF) | ~6 | 24 GB | Mild degradation | +| Q5_K_M (GGUF) | ~5 | 20 GB | Noticeable but usable | +| Q4_K_M (GGUF) | ~4.5 | 18 GB | Strong compression; some degradation | +| Q4_0 (GGUF) | ~4 | 16 GB | Worse than Q4_K_M | +| MLX 6-bit | ~6 | 22 GB | Apple Silicon native | +| MLX 4-bit | ~4 | 16 GB | Apple Silicon | + +### The LoRA delta survival problem + +This is the central failure mode that bit jerboa-lora v1 and crystal-lora v1. + +Your LoRA training produces a delta `ΔW = (α/r)·BA`. This delta is *added* to the base weights. The merged checkpoint is `W₀ + ΔW`. Quantization rounds the merged weights to a coarse grid (4-bit = 16 possible values per group; 8-bit = 256). + +**If `||ΔW||` is smaller than the quantization step size, the delta is rounded away.** Your model behaves identically to the un-trained base. + +#### How to think about it + +BF16's effective precision near 1.0 is about `2^-7 ≈ 0.008`. Q4_K_M's effective precision (which uses per-block scales and minimums) is roughly `2^-5 ≈ 0.03` near typical weight magnitudes. So a LoRA delta with magnitude `0.001` per weight will: +- Train at BF16: visible (0.001 > 0.008? no, but accumulates over many forward passes and matters for the output) +- Quantize to Q4_K_M: rounded to zero. *Erased.* + +A delta of `0.05` survives Q4_K_M comfortably. A delta of `0.01` survives Q8_0 but is borderline at Q4. + +#### What controls delta magnitude + +`||ΔW|| ≈ (α/r) · ||B|| · ||A||`. After training: + +- `||B||` grows with LR and number of steps. +- Larger LR or more steps → larger `||B||` → larger delta. +- DPO at `5e-7` × 18 steps (jerboa v1): tiny `||B||`, delta < Q4 step size, *vanishes on quantization*. +- DPO at `5e-6` × ~750 steps (this repo): substantially larger `||B||`, delta survives Q4_K_M. + +The math is messy but the practical rule is: + +**If you must ship Q4_K_M, your effective adaptation needs to be substantial: r≥64 *or* multiple stages of training compounding their deltas *or* higher LRs. If you can ship Q8_0, r=32 with normal LRs survives fine.** + +This is *why* this repo's primary deployment is Q8_0 (`:latest` Ollama tag). Q4_K_M is offered as a fallback (`:q4_k_m`) but with the explicit note that fidelity is lower. + +### MLX 6-bit — the Apple Silicon path + +MLX 6-bit sits between Q4_K_M and Q8_0 in fidelity. For 30B models on 64GB unified-memory Macs, it's the right size — fits in memory with room for KV cache and long context, retains most LoRA delta, and inference is fast. + +The conversion is one command: `./convert_to_mlx.sh runpod-pipeline-final gerbil-mlx-6bit-v3`. The output is uploaded to HuggingFace as `jaimef21/gerbil-qwen3-coder-30b-mlx-6bit`. + +--- + +## 10. Deployment formats + +Once you have the merged BF16 checkpoint at `/workspace/dpo_merged`, you pull it down and convert to one or more of: + +### Ollama (GGUF) + +GGUF is the file format for `llama.cpp` and `ollama`. Built by: + +1. `python -m llama_cpp.convert_hf_to_gguf model_dir --outfile model.gguf --outtype f16` +2. `llama-quantize model.gguf model-Q8_0.gguf Q8_0` +3. `ollama create model-name -f Modelfile` where Modelfile points at the .gguf file and includes a SYSTEM prompt. + +This repo's `build_ollama_gguf.sh` does steps 1-3 for both Q8_0 and Q4_K_M. + +`push_ollama.sh` pushes to ollama.com (`jaimef/gerbil-qwen3-coder:latest` and `:q4_k_m`). + +### MLX (Apple Silicon) + +`./convert_to_mlx.sh runpod-pipeline-final gerbil-mlx-6bit-v3` calls `python -m mlx_lm.convert -q --q-bits 6`. Output is an MLX bundle, ~22 GB for 30B at 6-bit. + +Run via `mlx_lm.generate` (one-shot) or `mlx_lm.server` (OpenAI-compatible HTTP server). + +`upload_hf.sh` pushes the MLX bundle to HuggingFace. + +### vLLM / TGI / native HF + +For server-side deployment, you can serve the merged BF16 checkpoint directly through vLLM, Text-Generation-Inference, or transformers' `pipeline`. Higher memory requirements (60 GB for 30B BF16), but no quantization loss. Use this on a hosted GPU server, not a laptop. + +--- + +## 11. RAG fundamentals + +RAG = Retrieval-Augmented Generation. Pair a frozen language model with a *retriever* over your domain corpus. At query time: + +1. **Retrieve.** Search the corpus (vector similarity, BM25, hybrid) for relevant chunks. +2. **Augment.** Stuff the retrieved chunks into the prompt as context. +3. **Generate.** The LLM answers with the chunks in view. + +### Why bother with RAG when we just fine-tuned + +Different tools, different jobs. + +| Question | Use | +|---|---| +| "Teach the model a new dialect's syntax." | **Fine-tune (LoRA).** Syntax is implicit, hard to put in prompts every time. | +| "The model needs to look up the right SRFI number for `string-trim`." | **RAG.** Documentation is large, mostly memorizable but cheaper to retrieve. | +| "Behavior changes weekly as we update guidelines." | **RAG.** Cheaper than retraining. | +| "Domain has 100K facts." | **RAG.** Won't fit in fine-tune data. | +| "Domain has 100 anti-idioms we want to suppress." | **Fine-tune (DPO).** Behavioral, not factual. | +| "We need formal stylistic constraints (always use def, never define)." | **Fine-tune (CPT + DPO).** | +| "We need to cite sources verbatim." | **RAG.** Fine-tuning produces approximate recall. | + +### The retrieval side + +A RAG system has two main components: + +1. **Embedding model.** Maps text → dense vector (typically 384-1536 dimensions). Sentence-transformer models like `all-MiniLM-L6-v2` (384d) or modern code-aware models like `nomic-embed-code` (768d). +2. **Vector index.** Stores vectors with metadata. Common: FAISS (in-memory), Qdrant, Weaviate, pgvector (Postgres extension), Pinecone (hosted). + +### Chunking strategy + +You can't index entire documents — you need to slice them. Common strategies: + +- **Fixed-size chunks** (512 tokens). Simple. Often breaks semantic units. +- **Recursive splitting.** Try splitting on `\n\n`, fall back to `\n`, fall back to `. `, fall back to char count. LangChain's default. +- **Semantic chunking.** Use sentence embeddings to find topic boundaries. Heavier. +- **Document-structure chunking.** Split on markdown headings, code blocks, function definitions. Most code-RAG should do this. + +For Gerbil-style code corpora, semantically-aware chunking is non-trivial. A reasonable approach: chunk by top-level form (each `(def …)` and its preceding doc-comment block becomes one chunk). + +### Hybrid retrieval + +Pure vector search misses exact-match queries ("what does `string->json-object` do?"). Pure BM25 (keyword) misses paraphrases. **Hybrid search** combines both, typically with rank fusion (Reciprocal Rank Fusion). + +Vector for semantic, BM25 for exact-token, blend the rankings. Most production RAG systems do this. + +### Reranking + +After retrieval returns top-k (say, 20), pass them through a cross-encoder reranker (e.g., `bge-reranker-base`) which scores each (query, chunk) pair jointly. Slower than embeddings but much more accurate at picking the best 3-5 to actually put in context. + +### How RAG quality is evaluated + +- **Retrieval recall@k**: of the chunks that *should* be retrieved (gold-labeled), what fraction are in top-k? +- **Retrieval MRR (mean reciprocal rank)**: average of 1/rank of first relevant result. +- **Answer correctness**: does the LLM's final answer match the gold answer? Measured with LLM-as-judge or string/embedding similarity. +- **Faithfulness**: does the answer cite or stay within the retrieved context? Hallucination rate. + +### RAG + Fine-tune is the strongest pattern + +For Gerbil, the ideal production setup would be: + +1. **Fine-tuned model** (this repo) for syntax, idioms, anti-idioms. +2. **RAG over the Gerbil stdlib docs** for "what does this procedure return?" questions. +3. **Reranker** to pick relevant stdlib pages. + +The fine-tune handles surface form. RAG handles specific facts. Combined, you can answer "show me how to use `string-trim` to strip whitespace from both ends" with both correct syntax and correct citation. + +This repo's pipeline doesn't include RAG today, but the data corpora (stdlib doc-comments) are exactly what you'd index for a RAG retriever later. + +--- + +## 12. Fine-tune vs RAG: when to pick what + +A decision framework. + +### Pick fine-tuning when + +- You want **behavioral** changes: tone, format, style, idiom preference. +- Domain has **implicit knowledge** that's hard to retrieve (syntax, conventions). +- Inference latency matters and you can't afford retrieval overhead. +- You have **labeled preference pairs** (DPO is uniquely good here). +- You want the model to internalize a **prior** that survives novel queries. + +### Pick RAG when + +- Domain is **large** (>1M tokens of relevant info). +- Domain **updates frequently** (weekly+) — retraining is expensive. +- You need **citations** or factual grounding. +- Answer recall must be **verbatim** (definitions, API signatures). +- You have **engineering** to invest in a retrieval stack but **not** training infrastructure. + +### Combine when you can + +The strongest pattern: fine-tune for *what to say*, RAG for *what facts to cite*. Build them in that order. Fine-tune produces a working model first; add RAG when you discover the model is fluent but unreliable on specific facts. + +--- + +## 13. Lessons from three projects + +A consolidated table of what worked, what failed, and what we learned in each `~/mine/*-lora` project. + +### From `~/mine/jerboa-lora` + +| Lesson | What happened | Takeaway | +|---|---|---| +| **DPO LR floor** | v1 used DPO LR `5e-7` × 18 steps. Cumulative `||ΔW||` was below Q4_K_M step size. Quantized model = base model behavior. | DPO LR must produce a delta that survives target quantization. Use `5e-6` minimum for Q4 targets. | +| **Together AI attention-only ceiling** | v1 hosted training only adapted attention modules. Val loss plateaued at 2.08. No improvement possible. | Attention-only LoRA is a hard ceiling for content adaptation. Target MLPs (and MoE experts when present). | +| **Heretic v1.3.0 silent expert skip** | Heretic's iteration over fused MoE experts raised TypeError, caught by `suppress(Exception)`. Expert params never adapted. Training looked normal, model behaved like base. | Always verify trainable parameter counts. `print_trainable_parameters()` must show hundreds of millions, not millions. | +| **MoE forward memory** | PEFT materialized fp32 LoRA deltas during forward, blew VRAM on A100 80GB even with batch=1. | MoE LoRA needs H100 NVL or H200. Or aggressive gradient_checkpointing + reduced seq_len. | +| **`merge_method: legacy` workaround** | Axolotl's default merge refused to proceed when `lora_target_parameters` was non-empty (false AdaLoRA detection). | Always set `merge_method: legacy` for fused-expert configs. | +| **Similarity eval is volume-invariant** | Holdout eval (14 questions) is sensitive to which 14 you pick. Per-pair similarity vs `chosen`/`rejected` is stable as you grow data. | Trust per-pair similarity over fixed-set holdout for tracking real improvement. | +| **Staged training works** | CPT → SFT → DPO with inter-stage merge produced a noticeably better model than SFT-only or SFT+DPO without CPT. | The three-stage pipeline is worth its complexity for niche-dialect adaptation. | + +### From `~/mine/crystal-lora` + +| Lesson | What happened | Takeaway | +|---|---|---| +| **v1 → v3 quantization-aware rank scaling** | v1 at `r=16` Q4_K_M produced no behavior change. v3 at `r=64` Q8_0 produced clear behavior change. | Match LoRA rank to target quantization. r≥64 for Q4, r≥32 for Q8. | +| **CPT data scale matters** | v1 used 1.7K records (~2 MB). v3 used 31.6M tokens. v3 was qualitatively better — more grounded in Crystal syntax. | CPT benefits from volume more than SFT/DPO. Mine the corpus aggressively. | +| **DPO LR correction** | v1 at `5e-7` lost the delta. v3 at `5e-6` survived Q8_0 quantization with margin. | Match Crystal's lesson: DPO LR floor is `5e-6` for Q4-or-better targets. | +| **`paged_adamw_8bit` at r=64 on A100** | OOM during forward pass with r=64 + MoE-equivalent attention block widths. Crystal isn't MoE but the wider model surfaces the same issue. | At higher ranks, optimizer state grows. Use `paged_adamw_8bit` (already set) and consider `lora_dropout: 0` to free memory. | +| **Compile-gating eliminates 36% LLM-augmented bugs** | An LLM was used to augment Ruby-Crystal pairs. ~36% of LLM-generated pairs had compile errors. Filtering them out was the difference between learning preferences and learning to prefer broken code. | DPO data must be validated against a real compiler/interpreter. (Done here with `gxi`.) | +| **H200 viable, ~$57 for 14 GPU-hours** | Crystal v3 ran on H200 at higher seq_len and r=64. Total cost lower per quality unit than constrained A100 runs. | If MoE OOM is a problem, H200 is worth the cost. | + +### From this repo (`~/mine/gerbil-lora`, in progress) + +| Lesson | What happened | Takeaway | +|---|---|---| +| **gxi validation is fast enough to gate every DPO pair** | All 66 pairs run through gxi in <30s. Catches missing modules (SRFI-69 not in Gerbil), wrong sig (SRFI-128 `hash` is keyword), invalid match targets. | DPO build should require interpreter pass before pair is accepted. Build the gate at the start. | +| **Stdlib doc-comment mining is high-value SFT data** | `;;` blocks immediately preceding `(def …)` produced 855 high-quality examples — far more than human-written cookbooks could supply. | Walk the stdlib of your target language. Treat existing doc-comments as ground truth Q/A pairs. | +| **`type: completion` for CPT, `type: chat_template` for SFT, `rl: dpo` for DPO** | All three modes are first-class in axolotl. No custom training code needed. | Stay on axolotl unless you have a specific reason to leave. | +| **Idempotent `.done` markers** | Each stage writes a marker file. Re-running orchestrator skips done stages. Survives SSH drops. | Idempotency is non-negotiable for multi-stage training. | +| **Tmux-per-stage** | `stage_cpt`, `stage_sft`, `stage_dpo` each run in their own tmux session on the pod. Orchestrator polls. Killing orchestrator doesn't kill training. | Always decouple orchestrator from training process. | +| **Memory at the edge** | 75.8 / 80 GB. No headroom. Cannot raise batch size. | Plan capacity at config time. Bigger card if you need bigger batches. | + +--- + +## 14. Reading list and next steps + +### Original papers — in order to read + +1. **"LoRA: Low-Rank Adaptation of Large Language Models"** (Hu et al., 2021). The original. Pages 1-6 are accessible; the rest is empirical. **Required.** +2. **"QLoRA: Efficient Finetuning of Quantized LLMs"** (Dettmers et al., 2023). Extends LoRA to quantized backbones. Important for understanding 4-bit base + LoRA training (which we don't do here but is common elsewhere). +3. **"Direct Preference Optimization: Your Language Model is Secretly a Reward Model"** (Rafailov et al., 2023). The DPO paper. Skip the appendix proofs unless you want them; the main text is readable. +4. **"Training language models to follow instructions with human feedback"** (OpenAI, 2022 — InstructGPT). The RLHF/PPO baseline that DPO replaced. Context for why DPO is simpler. +5. **"Mixture-of-Experts Meets Instruction Tuning"** (various papers, 2023-2024). MoE-specific fine-tuning considerations. + +### Practical resources + +- **PEFT documentation** (huggingface.co/docs/peft). The library you're using under the hood. Source of truth on what LoRA targets are supported. +- **Axolotl docs and configs** (github.com/OpenAccess-AI-Collective/axolotl). Tons of working configs in `examples/`. +- **`llama.cpp` quantization docs** (github.com/ggerganov/llama.cpp/discussions). Explains GGUF formats and what each Q* level does. +- **The TRL library** (github.com/huggingface/trl). DPO, PPO, ORPO, SimPO implementations. + +### Newer techniques to know exist + +- **ORPO** (Odds Ratio Preference Optimization): combines SFT + preference learning into one stage. Sometimes outperforms DPO; sometimes doesn't. Worth trying on your next project. +- **SimPO** (Simple Preference Optimization): like DPO but doesn't need a reference model. Lower memory. +- **KTO** (Kahneman-Tversky Optimization): preference learning from unpaired feedback (just thumbs-up / thumbs-down, no need for `chosen`/`rejected` pairs). +- **rsLoRA, VeRA, DoRA**: rank-stabilized variants. Generally small improvements; not transformative. +- **GaLore**: low-rank gradients instead of low-rank weights. Allows full-parameter training in memory. Future direction. + +### RAG resources + +- **LangChain / LlamaIndex documentation**. Standard frameworks. Use sparingly; their abstractions sometimes hide more than they help. +- **`sentence-transformers` docs**. Embedding models. +- **FAISS tutorial** (github.com/facebookresearch/faiss/wiki). The reference vector index. +- **"Lost in the Middle"** (Liu et al., 2023). On where in the context window models actually pay attention. Critical for chunk-ordering decisions. +- **BM25 + dense retrieval blending** — the Pinecone / Vespa engineering blogs cover this well. + +### What to study next, in priority order + +1. **Re-derive the DPO loss** on paper. You will not internalize it without doing this. +2. **Run a 1B-parameter SFT yourself** on a small dataset, watching every metric. Pythia-1B + a few hundred Q/A pairs. Smallest pipeline that exhibits all the mechanics. +3. **Implement a toy RAG over a small corpus** (e.g., Python stdlib docs) with sentence-transformers + FAISS, no framework. ~200 lines. +4. **Read the QLoRA paper carefully.** It's the highest-density education on why quantization + LoRA interact the way they do. +5. **Build a hybrid retriever** (BM25 + dense + reranker). Once you understand both, you understand modern RAG. + +--- + +## Quick-reference cards + +### LoRA hyperparameter starting points + +```yaml +# Conservative — wide-domain adaptation, Q4_K_M target +lora_r: 64 +lora_alpha: 128 +lora_dropout: 0.0 +lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj] + +# Balanced (this repo) — niche dialect, Q8_0 primary deploy +lora_r: 32 +lora_alpha: 64 +lora_dropout: 0.0 + +# Light — closely-related dialect, BF16 deploy +lora_r: 16 +lora_alpha: 32 +lora_dropout: 0.05 +``` + +### Stage LR table (validated) + +| Stage | LR | Epochs | Notes | +|---|---|---|---| +| CPT | 2e-5 | 2 | Lower if base is highly specialized already | +| SFT | 1e-4 | 2 | Standard. Don't go above 2e-4. | +| DPO | 5e-6 | 3 | **Hard floor for Q4_K_M survival.** | + +### Quantization survival floor + +| Target | Min LoRA rank | Min DPO LR | Min `||ΔW||` | +|---|---|---|---| +| BF16 deploy | 16 | 1e-7 | any | +| Q8_0 deploy | 32 | 1e-6 | ~0.01 | +| Q4_K_M deploy | 64 | 5e-6 | ~0.03 | + +### When to switch to a bigger GPU + +- A100 80GB OOMs on forward pass with `sequence_len > 2048` and r > 64 on 30B MoE. +- H100 NVL (94GB) handles seq_len 4096 with r=64. +- H200 (141GB) handles seq_len 4096 with r=128 and batch>1. + +--- + +*This document is meant to be updated as the pipeline evolves. When you discover a new failure mode or land a new technique, add it here.*