Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Handbooks  /  LoRA vs Full Fine-Tuning
AI~9 min readComparison
Head to Head

LoRA vs full fine-tuning: how much should change?

LoRAvsFull Fine-Tuning

Full fine-tuning updates every one of a model's billions of parameters. LoRA freezes almost all of them and trains a pair of tiny matrices instead — often under 1% of the parameter count. The surprising part isn't that LoRA is cheaper; it's that it usually works nearly as well.

01

The one distinction that decides everything

Full fine-tuning continues training and updates every weight in the model, exactly like pretraining, just on your data. LoRA (Low-Rank Adaptation) freezes the entire base model and instead injects small trainable matrices A (r×d) and B (d×r) alongside each targeted weight matrix, where rank r is tiny (often 4–64) compared to the full dimension d. Only A and B ever get gradients.

→ The rule

Need the biggest possible capability or knowledge shift and have the GPU budget? Full fine-tuning. Need to adapt behavior narrowly, cheaply, and want to preserve the base model's general skills? LoRA.

02

Head to head

DimensionLoRAFull Fine-Tuning
Trainable paramsOften 0.1%–1% of total100%
GPU memoryFits on one consumer GPU (QLoRA)Needs multi-GPU / high-end for 7B+
Training speedFaster per step, less memory bandwidthSlower, full optimizer state for every weight
Storage per taskA few MB per adapterA full model copy, tens of GB
Catastrophic forgettingLower — base weights untouchedHigher — can erode general capability
Capability ceilingGood for narrow adaptationHigher ceiling for large behavior/knowledge shifts
Multi-task servingHot-swap adapters on one loaded baseSeparate full model per task
Iteration costCheap — retrain in minutes on modest hardwareExpensive — full run every iteration
03

When to use each

Reach for LoRA

  • Limited GPU budget, or a single consumer card
  • Many narrow, distinct tasks to support
  • Want to preserve the base model’s general capability
  • Need fast, cheap iteration cycles
  • Serving many customized variants from one loaded base

Reach for full fine-tuning

  • Need a large capability or knowledge shift, not a narrow tweak
  • Have real training compute budget
  • Training a foundation-scale change, not adapting an existing skill
  • Quality ceiling matters more than training cost
  • Domain is far enough from the base model’s training distribution
04

The decision rule: start cheap, escalate only if needed

LoRA (or its quantized variant QLoRA, which quantizes the frozen base to 4-bit and trains LoRA adapters on top) is almost always the right first move, because the downside of trying it is small — a few GPU-hours — while full fine-tuning commits real compute before you know if the adaptation even needs that much capacity. Reach for full fine-tuning specifically when you've measured that LoRA's ceiling is the actual bottleneck, not by default.

→ The cheap default

Prompt engineering, then LoRA, then full fine-tuning — in that order of committed cost, escalating only when the cheaper option demonstrably can’t reach the behavior you need.

05

Why a tiny low-rank update can work at all

It sounds implausible that training 0.5% of a model's parameters could get close to full fine-tuning's results — the intuition-defying finding behind LoRA (Hu et al., 2021) is that the WEIGHT UPDATE a fine-tuning run actually needs, ΔW, tends to have a low "intrinsic rank": most of what changes during adaptation can be captured by a much lower-dimensional transformation than the full weight matrix's dimensions would suggest. Concretely, instead of learning a full d×d update matrix ΔW, LoRA constrains it to the product of two much smaller matrices, B·A, where B is d×r and A is r×d for a small r — meaning ΔW is forced to be low-rank by construction, and empirically that constraint barely hurts quality for most adaptation tasks because the "true" needed update was low-rank anyway.

This is also why LoRA specifically helps with catastrophic forgetting: because the base weights W are literally frozen and never receive a gradient, the model's original capabilities are structurally protected — the adapter can only ADD a small, low-rank correction on top of what the base model already knows, it can't overwrite it. Full fine-tuning has no such protection: every weight is free to move, which is exactly what gives it a higher ceiling for large shifts AND a higher risk of quietly degrading skills nobody was testing for.

→ The trade you’re actually making

Constraining the update to be low-rank is a real capacity limit, not a free lunch — it's just a limit that, empirically, most narrow adaptation tasks don't actually need to exceed.

06

A worked scenario: 50 brand-voice fine-tunes for one product

A startup wants 50 different customer-specific variants of the same base model, each speaking in a different brand's voice and following a different output template. Full fine-tuning here means storing and serving 50 separate multi-gigabyte model copies — real infrastructure cost multiplied by 50, and a full training run's compute cost multiplied by 50, for what is fundamentally a narrow, similar-shaped adaptation repeated many times.

LoRA fits this shape exactly: train 50 small adapters (each a few megabytes) against one shared, loaded base model, and hot-swap the active adapter per incoming request at serving time. The base model's weights, and the GPU memory they occupy, are shared across all 50 customers; only the tiny A/B matrices differ. This is precisely the scenario LoRA was built for — many related, narrow adaptations of one shared foundation — versus the case in section 03 where full fine-tuning stays the right call: a single large behavioral or knowledge shift too big for a low-rank update to reach.

→ The pattern generalizes

LoRA scales with the NUMBER of narrow variants you need; full fine-tuning scales with the SIZE of the single shift you need. Fifty small differences favors LoRA; one big difference favors full fine-tuning.

★

The flip point: how many parameters you actually train

For a d×d weight matrix, full fine-tuning updates all d² parameters; LoRA learns two small matrices (d×r and r×d) for a chosen rank r, so it trains 2·r·d parameters — a fraction 2r/d of full.

→ The number

For a 4096×4096 layer at rank r=16, LoRA trains 2·16·4096 ≈ 131K parameters versus 16.8M for full — about 2·16/4096 = 0.78%, roughly 1/128th. And the memory gap is even larger, because full fine-tuning must also hold optimizer state (gradients plus Adam’s two moments) for every weight — several times the model — while LoRA holds it only for the tiny adapter. The flip: reach for full fine-tuning only when you need to change the model deeply (a genuinely new capability or domain shift); for adapting task, format, or style, LoRA captures most of the benefit at well under 1% of the trainable parameters and a fraction of the memory.

The picture

The picture: a frozen path and a low-rank side path

input xd = 4096 W frozen, d × d16,777,216 weights, no grads trainable path A: r × ddown to r = 16 B: d × rstarts at zero × α/r + h out trainable: 2 × 16 × 4,096 = 131,072 (0.78%) after training, merge W + (α/r)·BA: no extra latency
One 4,096 × 4,096 layer at rank 16, as in the page’s flip-point box. The output is h = Wx + (α/r)·BAx. W never gets a gradient; only A and B do. A note on shapes, following Hu et al.: B is d × r and A is r × d, so B·A comes out d × d, the same shape as W. Together they hold 2·r·d parameters. B starts at zero, so at step 0 the update is zero and the model behaves exactly like the base. The α/r factor means changing r also changes how big the update is unless you adjust α to match. At inference there are two options. You can keep the side path, which lets you swap adapters per request. Or you can fold B·A into W once, after which the side path is gone and the model is an ordinary dense model again.
Trade-offs

Cost and trade-offs: memory, storage and speed for a 7B model

Worked on a model shaped like Llama 2 7B: 6.74B parameters, 32 layers, width 4,096. Here Time is training or serving cost, and Space is GPU memory or disk. Training memory uses the standard estimate for mixed-precision Adam of about 16 bytes per trained parameter (16-bit weight and gradient, plus a 32-bit master copy and Adam’s two moments). Frozen weights cost only their storage. Activations come on top of every row.

CaseTimeSpaceWhy
Full fine-tuningGradients and an optimizer step for all 6.74B parameters6.74B × 16 bytes ≈ 108 GB before activationsToo big for one 80 GB GPU, which is why it needs multi-GPU sharding.
LoRA, r = 16 on Wq and WvForward and backward still run through the whole frozen model13.5 GB frozen base (16-bit) + 8.4M × 16 bytes ≈ 134 MBThe original paper’s setup: 32 layers × 2 matrices × 2 × 16 × 4,096 = 8,388,608 parameters, 0.12% of the model.
LoRA, r = 16 on all linear layersAbout the same step time13.5 GB + 40M × 16 bytes ≈ 640 MBFor a d_in × d_out matrix, LoRA trains r·(d_in + d_out); the MLP matrices are 4,096 × 11,008. 39,976,960 parameters, 0.59%.
QLoRA, same adaptersSlower per step: the 4-bit weights are dequantized on every pass6.74B × 0.5 bytes ≈ 3.4 GB base + adaptersFits a consumer GPU. The adapter trains against the quantized base.
Storing 50 customer variantsNothing to train again once storedFull: 50 × 13.5 GB = 674 GB. LoRA on Wq, Wv: 50 × 16.8 MB ≈ 0.84 GBThe page’s brand-voice scenario in numbers: about 800× less disk.
Serving one adapter, mergedNo extra latency: W + (α/r)·BA is one ordinary matrixOne full-size model per merged variantFastest, but you lose the ability to swap adapters.
Serving many adapters, unmergedA small extra matrix multiply per adapted layerOne shared base + a few MB per adapterHow one loaded base serves many customers.
Doubling the rankBarely changes step timeAdapter parameters, optimizer state and file size all doubleCapacity grows linearly. It does not fix bad or too little data.

The headline “under 1% of parameters” is about memory and storage, not compute. Every LoRA step still runs the full model forwards and backwards. What disappears is the weight gradient for W and, above all, the optimizer state, which is where 108 GB comes down to about 14 GB. Picking the rank: the LoRA paper found very small ranks, even 1 or 2 on Wq and Wv, competitive in its GPT-3 experiments. That is the low-intrinsic-rank argument from section 05 showing up in practice. So start small, at 8 or 16, and raise the rank only when evals show the adapter is out of capacity. Each doubling doubles the adapter and its optimizer state and leaves the 13.5 GB base untouched.

Watch for

Common mistakes

  • Reading 1% of parameters as 1% of the compute

    The backward pass still has to carry gradients through every frozen layer to reach the adapters below them. A LoRA step costs a large share of a full step. The big savings are memory, optimizer state and disk. Budget GPU-hours on that basis.

  • Changing the rank without touching alpha

    The update is scaled by α/r. Double r with α fixed and you halve the scale each adapter direction gets, so a rank sweep also quietly sweeps the effective learning rate. Change one thing at a time. And do not copy the learning rate from a full fine-tuning recipe; LoRA usually needs a different one, so sweep it.

  • Adapting too few layers, then blaming LoRA

    The original paper adapted only Wq and Wv. The QLoRA paper found that adapters on all linear layers were needed to match full fine-tuning. If LoRA falls short, try more target modules before you jump to full fine-tuning; above, that is still only 0.59% of the parameters.

  • Loading an adapter onto a different base

    An adapter is a correction to one exact set of weights. Put it on a newer checkpoint of the “same” model, or a different quantization of it, and it is correcting weights it never saw. The result can look fine and score worse. Record the base model and its precision with every adapter, and re-run evals after merging.

  • Checking forgetting only on the target task

    Frozen base weights mean LoRA forgets less, not nothing. The adapter changes every output it touches. Keep a small general-skills eval next to the task eval and run both, or a gain on brand voice can hide a loss on reasoning.

Your call

Which would you pick?

Three situations. Pick the side you'd actually build — the explanation follows.

One product needs 50 variants of the same model, each matching a different customer's brand voice.

A model must learn a specialised medical domain it has essentially no knowledge of, and you have the GPU budget.

A team's first move on a new behaviour request is to schedule a full fine-tuning run.

Frequently asked

Quick answers

LoRA or full fine-tuning — which should I use?

Start with LoRA — it’s cheap to try and covers most narrow adaptation tasks (tone, format, a specific skill) while preserving the base model’s general capability. Reach for full fine-tuning only once you’ve confirmed the adaptation needs a bigger capability shift than a low-rank update can capture.

Does LoRA hurt quality compared to full fine-tuning?

For most narrow adaptation tasks, the gap is small because the weight update those tasks actually need tends to be low-rank anyway. For large capability or knowledge shifts, full fine-tuning’s unrestricted updates give it a real, measurable edge.

What is QLoRA?

LoRA applied on top of a 4-bit-quantized frozen base model. It cuts memory further, making it possible to fine-tune multi-billion-parameter models on a single consumer GPU, at a small additional cost in precision from the quantization.

Can you serve many LoRA adapters from one base model?

Yes — this is one of LoRA’s biggest practical advantages. Because the base model is shared and unchanged, you can hot-swap small adapters per request, serving many customized behaviors from a single loaded model instead of many full model copies.

▶  Watch it explained

LoRA vs Full Fine-Tuning: rewrite the whole book, or add sticky notes?

LoRA vs Full Fine-Tuning · Vibe Engines · 2026
Finished this one? 0 / 209 Handbooks done
NextThe Distributed Training Handbook Next on the “Training and adaptation” ladder · rung 3 of 7

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks