LoRA vs full fine-tuning: how much should change?

Full fine-tuning continues training and updates every weight in the model, exactly like pretraining, just on your data. LoRA (Low-Rank Adaptation) freezes the entire base model and instead injects small trainable matrices A (d×r) and B (r×d) alongside each targeted weight matrix, where rank r is tiny (often 4–64) compared to the full dimension d. Only A and B ever get gradients.

LoRA (or its quantized variant QLoRA, which quantizes the frozen base to 4-bit and trains LoRA adapters on top) is almost always the right first move, because the downside of trying it is small — a few GPU-hours — while full fine-tuning commits real compute before you know if the adaptation even needs that much capacity. Reach for full fine-tuning specifically when you've measured that LoRA's ceiling is the actual bottleneck, not by default.

It sounds implausible that training 0.5% of a model's parameters could get close to full fine-tuning's results — the intuition-defying finding behind LoRA (Hu et al., 2021) is that the WEIGHT UPDATE a fine-tuning run actually needs, ΔW, tends to have a low "intrinsic rank": most of what changes during adaptation can be captured by a much lower-dimensional transformation than the full weight matrix's dimensions would suggest. Concretely, instead of learning a full d×d update matrix ΔW, LoRA constrains it to the product of two much smaller matrices, B·A, where A is d×r and B is r×d for a small r — meaning ΔW is forced to be low-rank by construction, and empirically that constraint barely hurts quality for most adaptation tasks because the "true" needed update was low-rank anyway.

This is also why LoRA specifically helps with catastrophic forgetting: because the base weights W are literally frozen and never receive a gradient, the model's original capabilities are structurally protected — the adapter can only ADD a small, low-rank correction on top of what the base model already knows, it can't overwrite it. Full fine-tuning has no such protection: every weight is free to move, which is exactly what gives it a higher ceiling for large shifts AND a higher risk of quietly degrading skills nobody was testing for.

A startup wants 50 different customer-specific variants of the same base model, each speaking in a different brand's voice and following a different output template. Full fine-tuning here means storing and serving 50 separate multi-gigabyte model copies — real infrastructure cost multiplied by 50, and a full training run's compute cost multiplied by 50, for what is fundamentally a narrow, similar-shaped adaptation repeated many times.

LoRA fits this shape exactly: train 50 small adapters (each a few megabytes) against one shared, loaded base model, and hot-swap the active adapter per incoming request at serving time. The base model's weights, and the GPU memory they occupy, are shared across all 50 customers; only the tiny A/B matrices differ. This is precisely the scenario LoRA was built for — many related, narrow adaptations of one shared foundation — versus the case in section 03 where full fine-tuning stays the right call: a single large behavioral or knowledge shift too big for a low-rank update to reach.