Fine-tuning,
and the low-rank shortcut.

Before reaching for fine-tuning, know your options. There are three ways to make an LLM do what you want, in increasing order of cost and commitment.

Fine-tuning continues training a pretrained model on your own examples, updating its weights so the behavior is baked in — no prompt or retrieval needed at inference. It's how you get a model to reliably adopt a style or persona, always produce a specific format (strict JSON, a house tone), follow a task pattern consistently, or specialize in a narrow domain's language.

The common mistake is fine-tuning to add knowledge. That works poorly: facts crammed into weights are lossy, hard to update, and easy to hallucinate around — exactly what RAG does better and more cheaply. Fine-tuning teaches how to respond, not what is true. If your problem is "the model doesn't know X," use RAG; if it's "the model doesn't respond the way I need," consider fine-tuning.

Full fine-tuning updates all the model's weights. It's powerful but brutally expensive: you need enough GPU memory to hold the model, its gradients, and optimizer state — for a large model, many high-end GPUs — and you end up with a full copy of the model per task. For most teams that's a non-starter.

Parameter-efficient fine-tuning (PEFT) solves this by freezing the base model and training only a tiny number of new parameters. You get most of the quality for a fraction of the memory, compute, and storage — and you can keep the base model shared and swap small task-specific pieces. LoRA is the dominant PEFT method.

LoRA (Low-Rank Adaptation) freezes the original weight matrices and, next to each, injects two small matrices whose product is a low-rank update. During training, only these little adapter matrices learn; the huge base weights never move. At inference, the adapter's contribution is added to the frozen weight, so the effective weight is W + BA — the original plus a small learned correction.

The insight is that the change a task requires has low intrinsic rank — it can be captured by matrices far smaller than the full weight. So instead of training millions of parameters, you train thousands, using a fraction of the memory. QLoRA pushes this further: load the frozen base model in 4-bit quantized precision to slash its memory footprint, then train LoRA adapters on top — letting you fine-tune a very large model on a single GPU that couldn't otherwise hold it, with little quality loss.

Just how small is "low-rank"? Count it.

Take one attention projection — a 4096 × 4096 weight, about 16.8M numbers. LoRA replaces its update with two skinny matrices, B (4096 × r) and A (r × 4096). At rank r = 8 that's:

One matrix. The adapter trains 256× fewer numbers than the weight it corrects.

The adapter is applied with a fixed scale α/r, so the effective weight is W + (α/r)·BA. That's why α and r are tuned together — raise r for more capacity, and α/r keeps the update's magnitude sane. But the parameter count isn't even the memory that matters. Optimizers do. Adam keeps two fp32 moments per trainable parameter (8 bytes), so full fine-tuning a 7B model needs ~52 GB just for Adam state — before activations. LoRA trains ~4M params instead of 7B, cutting that to ~32 MB. That — not the disk size of the adapter — is what lets a fine-tune fit on one GPU.

The knobs that matter (LoRA/QLoRA)

Here's the part teams underestimate: fine-tuning is 90% data work. The method (LoRA and friends) is nearly commoditized; the differentiator is a clean, well-formatted, high-quality dataset of the exact behavior you want. Garbage in, garbage baked into the weights.

Format your data as the model expects (prompt–response pairs, chat turns, an instruction template). Prize quality over quantity — a few hundred to a few thousand excellent, consistent examples usually beat tens of thousands of noisy ones, because fine-tuning imitates whatever you show it, mistakes included. Ensure examples are consistent (contradictory examples teach nothing) and representative of real inputs. Hold out a clean test set from the start.

Fine-tuning has a dark side. Catastrophic forgetting: training hard on a narrow task can overwrite the model's general abilities — your specialized model gets great at the new thing and worse at everything else. And overfitting: with a small dataset, the model can memorize your examples rather than learn the pattern, producing brittle, parroting behavior.

Mitigations: PEFT methods like LoRA reduce forgetting by leaving the base weights frozen; keep learning rates modest and don't over-train (few epochs); optionally mix in general-purpose data so the model retains breadth; and — crucially — evaluate for regressions, not just improvements. A fine-tune that nails your task but tanks the model's reasoning or safety is a net loss you'll only catch if you measure the general abilities too.

Judge a fine-tune on a held-out set the model never trained on — otherwise you're measuring memorization. Compare against the honest baselines: the base model with good prompting, and a RAG setup. Fine-tuning only wins if it clearly beats those on your real metric, and doesn't regress general ability. Measure both the target behavior and a broad capability check.

Then weigh the ongoing costs: fine-tuning adds a training and data pipeline, model/adapter versioning, and re-tuning as needs change. It's worth it when you need consistent behavior/format/style at scale, want to shrink prompts (bake the instructions in) for latency and cost, or must specialize a smaller cheaper model to match a bigger one on your task. It's the wrong tool for knowledge (use RAG), for one-offs (use prompting), or when you can't produce quality data.