Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
LoRA vs full fine-tuning: how much should change?
Full fine-tuning updates every one of a model's billions of parameters. LoRA freezes almost all of them and trains a pair of tiny matrices instead — often under 1% of the parameter count. The surprising part isn't that LoRA is cheaper; it's that it usually works nearly as well.
The one distinction that decides everything
Full fine-tuning continues training and updates every weight in the model, exactly like pretraining, just on your data. LoRA (Low-Rank Adaptation) freezes the entire base model and instead injects small trainable matrices A (r×d) and B (d×r) alongside each targeted weight matrix, where rank r is tiny (often 4–64) compared to the full dimension d. Only A and B ever get gradients.
Need the biggest possible capability or knowledge shift and have the GPU budget? Full fine-tuning. Need to adapt behavior narrowly, cheaply, and want to preserve the base model's general skills? LoRA.
Head to head
| Dimension | LoRA | Full Fine-Tuning |
|---|---|---|
| Trainable params | Often 0.1%–1% of total | 100% |
| GPU memory | Fits on one consumer GPU (QLoRA) | Needs multi-GPU / high-end for 7B+ |
| Training speed | Faster per step, less memory bandwidth | Slower, full optimizer state for every weight |
| Storage per task | A few MB per adapter | A full model copy, tens of GB |
| Catastrophic forgetting | Lower — base weights untouched | Higher — can erode general capability |
| Capability ceiling | Good for narrow adaptation | Higher ceiling for large behavior/knowledge shifts |
| Multi-task serving | Hot-swap adapters on one loaded base | Separate full model per task |
| Iteration cost | Cheap — retrain in minutes on modest hardware | Expensive — full run every iteration |
When to use each
Reach for LoRA
- Limited GPU budget, or a single consumer card
- Many narrow, distinct tasks to support
- Want to preserve the base model’s general capability
- Need fast, cheap iteration cycles
- Serving many customized variants from one loaded base
Reach for full fine-tuning
- Need a large capability or knowledge shift, not a narrow tweak
- Have real training compute budget
- Training a foundation-scale change, not adapting an existing skill
- Quality ceiling matters more than training cost
- Domain is far enough from the base model’s training distribution
The decision rule: start cheap, escalate only if needed
LoRA (or its quantized variant QLoRA, which quantizes the frozen base to 4-bit and trains LoRA adapters on top) is almost always the right first move, because the downside of trying it is small — a few GPU-hours — while full fine-tuning commits real compute before you know if the adaptation even needs that much capacity. Reach for full fine-tuning specifically when you've measured that LoRA's ceiling is the actual bottleneck, not by default.
Prompt engineering, then LoRA, then full fine-tuning — in that order of committed cost, escalating only when the cheaper option demonstrably can’t reach the behavior you need.
Why a tiny low-rank update can work at all
It sounds implausible that training 0.5% of a model's parameters could get close to full fine-tuning's results — the intuition-defying finding behind LoRA (Hu et al., 2021) is that the WEIGHT UPDATE a fine-tuning run actually needs, ΔW, tends to have a low "intrinsic rank": most of what changes during adaptation can be captured by a much lower-dimensional transformation than the full weight matrix's dimensions would suggest. Concretely, instead of learning a full d×d update matrix ΔW, LoRA constrains it to the product of two much smaller matrices, B·A, where B is d×r and A is r×d for a small r — meaning ΔW is forced to be low-rank by construction, and empirically that constraint barely hurts quality for most adaptation tasks because the "true" needed update was low-rank anyway.
This is also why LoRA specifically helps with catastrophic forgetting: because the base weights W are literally frozen and never receive a gradient, the model's original capabilities are structurally protected — the adapter can only ADD a small, low-rank correction on top of what the base model already knows, it can't overwrite it. Full fine-tuning has no such protection: every weight is free to move, which is exactly what gives it a higher ceiling for large shifts AND a higher risk of quietly degrading skills nobody was testing for.
Constraining the update to be low-rank is a real capacity limit, not a free lunch — it's just a limit that, empirically, most narrow adaptation tasks don't actually need to exceed.
A worked scenario: 50 brand-voice fine-tunes for one product
A startup wants 50 different customer-specific variants of the same base model, each speaking in a different brand's voice and following a different output template. Full fine-tuning here means storing and serving 50 separate multi-gigabyte model copies — real infrastructure cost multiplied by 50, and a full training run's compute cost multiplied by 50, for what is fundamentally a narrow, similar-shaped adaptation repeated many times.
LoRA fits this shape exactly: train 50 small adapters (each a few megabytes) against one shared, loaded base model, and hot-swap the active adapter per incoming request at serving time. The base model's weights, and the GPU memory they occupy, are shared across all 50 customers; only the tiny A/B matrices differ. This is precisely the scenario LoRA was built for — many related, narrow adaptations of one shared foundation — versus the case in section 03 where full fine-tuning stays the right call: a single large behavioral or knowledge shift too big for a low-rank update to reach.
LoRA scales with the NUMBER of narrow variants you need; full fine-tuning scales with the SIZE of the single shift you need. Fifty small differences favors LoRA; one big difference favors full fine-tuning.
The flip point: how many parameters you actually train
For a d×d weight matrix, full fine-tuning updates all d² parameters; LoRA learns two small matrices (d×r and r×d) for a chosen rank r, so it trains 2·r·d parameters — a fraction 2r/d of full.
For a 4096×4096 layer at rank r=16, LoRA trains 2·16·4096 ≈ 131K parameters versus 16.8M for full — about 2·16/4096 = 0.78%, roughly 1/128th. And the memory gap is even larger, because full fine-tuning must also hold optimizer state (gradients plus Adam’s two moments) for every weight — several times the model — while LoRA holds it only for the tiny adapter. The flip: reach for full fine-tuning only when you need to change the model deeply (a genuinely new capability or domain shift); for adapting task, format, or style, LoRA captures most of the benefit at well under 1% of the trainable parameters and a fraction of the memory.
The picture: a frozen path and a low-rank side path
h = Wx + (α/r)·BAx. W never gets a gradient; only A and B do. A note on shapes, following Hu et al.: B is d × r and A is r × d, so B·A comes out d × d, the same shape as W. Together they hold 2·r·d parameters. B starts at zero, so at step 0 the update is zero and the model behaves exactly like the base. The α/r factor means changing r also changes how big the update is unless you adjust α to match. At inference there are two options. You can keep the side path, which lets you swap adapters per request. Or you can fold B·A into W once, after which the side path is gone and the model is an ordinary dense model again.Cost and trade-offs: memory, storage and speed for a 7B model
Worked on a model shaped like Llama 2 7B: 6.74B parameters, 32 layers, width 4,096. Here Time is training or serving cost, and Space is GPU memory or disk. Training memory uses the standard estimate for mixed-precision Adam of about 16 bytes per trained parameter (16-bit weight and gradient, plus a 32-bit master copy and Adam’s two moments). Frozen weights cost only their storage. Activations come on top of every row.
| Case | Time | Space | Why |
|---|---|---|---|
| Full fine-tuning | Gradients and an optimizer step for all 6.74B parameters | 6.74B × 16 bytes ≈ 108 GB before activations | Too big for one 80 GB GPU, which is why it needs multi-GPU sharding. |
| LoRA, r = 16 on Wq and Wv | Forward and backward still run through the whole frozen model | 13.5 GB frozen base (16-bit) + 8.4M × 16 bytes ≈ 134 MB | The original paper’s setup: 32 layers × 2 matrices × 2 × 16 × 4,096 = 8,388,608 parameters, 0.12% of the model. |
| LoRA, r = 16 on all linear layers | About the same step time | 13.5 GB + 40M × 16 bytes ≈ 640 MB | For a d_in × d_out matrix, LoRA trains r·(d_in + d_out); the MLP matrices are 4,096 × 11,008. 39,976,960 parameters, 0.59%. |
| QLoRA, same adapters | Slower per step: the 4-bit weights are dequantized on every pass | 6.74B × 0.5 bytes ≈ 3.4 GB base + adapters | Fits a consumer GPU. The adapter trains against the quantized base. |
| Storing 50 customer variants | Nothing to train again once stored | Full: 50 × 13.5 GB = 674 GB. LoRA on Wq, Wv: 50 × 16.8 MB ≈ 0.84 GB | The page’s brand-voice scenario in numbers: about 800× less disk. |
| Serving one adapter, merged | No extra latency: W + (α/r)·BA is one ordinary matrix | One full-size model per merged variant | Fastest, but you lose the ability to swap adapters. |
| Serving many adapters, unmerged | A small extra matrix multiply per adapted layer | One shared base + a few MB per adapter | How one loaded base serves many customers. |
| Doubling the rank | Barely changes step time | Adapter parameters, optimizer state and file size all double | Capacity grows linearly. It does not fix bad or too little data. |
The headline “under 1% of parameters” is about memory and storage, not compute. Every LoRA step still runs the full model forwards and backwards. What disappears is the weight gradient for W and, above all, the optimizer state, which is where 108 GB comes down to about 14 GB. Picking the rank: the LoRA paper found very small ranks, even 1 or 2 on Wq and Wv, competitive in its GPT-3 experiments. That is the low-intrinsic-rank argument from section 05 showing up in practice. So start small, at 8 or 16, and raise the rank only when evals show the adapter is out of capacity. Each doubling doubles the adapter and its optimizer state and leaves the 13.5 GB base untouched.
Common mistakes
- Reading 1% of parameters as 1% of the compute
The backward pass still has to carry gradients through every frozen layer to reach the adapters below them. A LoRA step costs a large share of a full step. The big savings are memory, optimizer state and disk. Budget GPU-hours on that basis.
- Changing the rank without touching alpha
The update is scaled by α/r. Double r with α fixed and you halve the scale each adapter direction gets, so a rank sweep also quietly sweeps the effective learning rate. Change one thing at a time. And do not copy the learning rate from a full fine-tuning recipe; LoRA usually needs a different one, so sweep it.
- Adapting too few layers, then blaming LoRA
The original paper adapted only Wq and Wv. The QLoRA paper found that adapters on all linear layers were needed to match full fine-tuning. If LoRA falls short, try more target modules before you jump to full fine-tuning; above, that is still only 0.59% of the parameters.
- Loading an adapter onto a different base
An adapter is a correction to one exact set of weights. Put it on a newer checkpoint of the “same” model, or a different quantization of it, and it is correcting weights it never saw. The result can look fine and score worse. Record the base model and its precision with every adapter, and re-run evals after merging.
- Checking forgetting only on the target task
Frozen base weights mean LoRA forgets less, not nothing. The adapter changes every output it touches. Keep a small general-skills eval next to the task eval and run both, or a gain on brand voice can hide a loss on reasoning.
Which would you pick?
Three situations. Pick the side you'd actually build — the explanation follows.
One product needs 50 variants of the same model, each matching a different customer's brand voice.
Brand voice is narrow behavioural adaptation, which is exactly what a low-rank update handles — and 50 adapters over one base is a fraction of the storage and GPU cost of 50 full checkpoints. Training one model on all 50 voices at once gives you a model that averages them, which is not what any single customer asked for.
A model must learn a specialised medical domain it has essentially no knowledge of, and you have the GPU budget.
The page's rule is explicit: the biggest possible capability or knowledge shift, with budget available, is the full fine-tuning case. Constraining the update to be low-rank is a real capacity limit — usually one narrow adaptation does not need to exceed, but acquiring a domain the base model lacks is precisely where it bites.
A team's first move on a new behaviour request is to schedule a full fine-tuning run.
The page's cheap default is to escalate in order of committed cost, moving up only when the cheaper option demonstrably cannot reach the behaviour you need. Starting at the top spends the most expensive option on a problem that may not require it, and having data ready lowers one cost while leaving the compute, the storage and the loss of the base model's general skills untouched.
Quick answers
LoRA or full fine-tuning — which should I use?
Start with LoRA — it’s cheap to try and covers most narrow adaptation tasks (tone, format, a specific skill) while preserving the base model’s general capability. Reach for full fine-tuning only once you’ve confirmed the adaptation needs a bigger capability shift than a low-rank update can capture.
Does LoRA hurt quality compared to full fine-tuning?
For most narrow adaptation tasks, the gap is small because the weight update those tasks actually need tends to be low-rank anyway. For large capability or knowledge shifts, full fine-tuning’s unrestricted updates give it a real, measurable edge.
What is QLoRA?
LoRA applied on top of a 4-bit-quantized frozen base model. It cuts memory further, making it possible to fine-tune multi-billion-parameter models on a single consumer GPU, at a small additional cost in precision from the quantization.
Can you serve many LoRA adapters from one base model?
Yes — this is one of LoRA’s biggest practical advantages. Because the base model is shared and unchanged, you can hot-swap small adapters per request, serving many customized behaviors from a single loaded model instead of many full model copies.
LoRA vs Full Fine-Tuning: rewrite the whole book, or add sticky notes?
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- The LLM Serving HandbookHow to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.Read →
- The Transformers HandbookThe architecture behind every LLM, built up from scratch — tokens and embeddings, positional encoding, self-attention (query/key/value), multi-head attention, the transformer block (FFN, residuals, layer norm), and how a decoder-only model generates text autoregressively.Read →
- The Fine-Tuning HandbookWhen to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.Read →
- The On-Device AI HandbookSmall models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.Read →