LoRA,
explained.

Rungs 1 to 3 all end in a gradient step on every parameter, which means the cost of adapting a model scales with the model. What actually makes that expensive is not the weights — it is the optimiser state, since Adam keeps two running averages per trainable parameter, so a full fine-tune needs roughly three times the model's memory before activations. Adapting a large model therefore needs the hardware that trained it.

The change rests on an observation about what a fine-tune actually does: the update to a weight matrix has much lower rank than the matrix itself — adaptation moves the model within a small subspace. So do not represent the update as a full matrix. Freeze the pretrained weights and learn the update as a product of two thin matrices, one projecting down to a small rank and one back up. Only those two are trained, so the optimiser state shrinks with the rank rather than the model.

Stated in units: trainable parameters fall by up to four orders of magnitude, and with them the optimiser memory and the size of what you have to store per task. Two consequences matter as much as the saving. First, because the learned update is a product that can be added into the frozen weights before serving, there is no extra inference latency at all — which is what distinguishes this from adapter layers that insert extra depth into the network and pay for it on every token forever. Second, many task adapters can share one copy of the base model in memory, so a fleet serving a hundred fine-tunes needs one model and a hundred small deltas.

Fine-tuning is how you adapt a pretrained model to your task — your tone, your domain, your format. The classic recipe updates every weight in the model. For a model with billions of parameters, that means holding a full copy in GPU memory, optimizing all of it, and saving a fresh multi-gigabyte model per task.

Want ten specialized variants? Ten full copies. It's like rebuilding an entire skyscraper from the foundation up every time you want to repaint one room. Prohibitive for anyone without a data center — and wasteful even for those who have one.

The authors started from a hunch that turned out to be the whole paper: the change a model needs to learn a new task is far simpler than the model itself. Formally, the weight update has low intrinsic rank — it lives in a tiny subspace, not the full billion-dimensional space.

If the update is that simple, you don't need to represent it with a giant dense matrix. You can capture it with something much smaller — and that's the opening LoRA walks through.

Take a weight matrix W in the pretrained model. LoRA freezes it — never touches it again — and adds a side path: two small matrices, A and B, whose product represents the update. The layer now computes W·x + B·A·x, and only A and B are trained.

The trick is the shape. If W is d×d, then A is r×d and B is d×r, where the rank r is tiny — often just 4 or 8. Their product is the same size as W, but they contain a few thousand times fewer numbers to train.

Watch a rank-1 update describe a whole matrix. Say training produced these two tiny factors for a 4-dimensional layer (B starts as zeros — so LoRA begins as a no-op — and learns; A starts random):

Look at the rows of ΔW: they're 1×, 2×, 0×, and 1× the same row A. Eight stored numbers just described a sixteen-number update — that's what rank 1 means. The matrix can only push inputs toward one direction, but the paper's bet is that fine-tuning barely needs more than a few such directions. The bet pays off brutally well at real widths (d = 4096, one layer):

The forward pass adds the side path, scaled by α/r so different ranks train with similar learning-rate behavior:

Applied to GPT-3 175B (rank 4, attention W_q and W_v only), this cut trainable parameters 10,000× and the per-task checkpoint from 350 GB to 35 MB.

Earlier efficient-tuning methods bolted extra layers into the network, which made every prediction a little slower forever. LoRA sidesteps this entirely. After training, you can merge the small update back into the original matrix — W' = W + B·A — producing a model that is byte-for-byte the same size and speed as the original.

The paper is clear that the rank is chosen empirically — there is no principle that tells you what it should be — and that surprisingly small ranks suffice, which is evidence for the low-rank premise and also a result nobody predicted the size of. It also reports adapting only the attention projections rather than every matrix, again on measurement rather than derivation, and does not claim parity with full fine-tuning in all cases.

The limitation the field settled on later is a sharpening of exactly that premise. Low-rank updates are excellent at teaching a model a style, format or task interface, and comparatively weak at installing substantial new knowledge — which follows from the hypothesis rather than contradicting it: if adaptation is genuinely confined to a small subspace, then anything requiring a large change is out of reach by construction. The practical rule that emerged — reach for this to change behaviour, reach for continued pretraining to change what the model knows — is a consequence of the paper being right, not wrong.

How it held up: it is the default way anyone fine-tunes anything. The multiple-adapters-one-base pattern became a serving architecture, the variants that followed mostly adjust how the rank is allocated across layers, and the next rung removes the one cost this one leaves standing — that the frozen base still has to fit in memory at full precision.

Because each task's LoRA is just a few megabytes of A and B, you can keep one frozen base model and a shelf of tiny adapters — one for legal writing, one for code, one for your brand voice — and swap them in and out on the fly. Ten specialized models become one big model plus ten postage-stamp files, instead of ten skyscrapers.

On GPT-3 175B, LoRA cut the number of trainable parameters by about 10,000x and GPU memory by roughly 3x, while matching or beating full fine-tuning on the benchmarks tested — with no added inference latency.

Same quality, four orders of magnitude fewer parameters to train and store.

LoRA turned fine-tuning from a big-lab luxury into something a solo developer can do on a rented GPU — and it became the default of the whole parameter-efficient fine-tuning (PEFT) family. Its descendant QLoRA quantizes the frozen base to 4-bit and trains adapters on top, letting you fine-tune enormous models on a single consumer card.

Every "custom fine-tune," every marketplace of swappable model adapters, every hobbyist tuning a model at home traces back to one observation in this paper: the update you need is small, so train something small.

LoRA — Low-Rank Adaptation — is the reason customizing large models became cheap and ubiquitous. Full fine-tuning updates every weight, which for a big model means enormous memory and a full-size copy of the weights per task. LoRA freezes the base model entirely and trains only small, low-rank adapter matrices injected alongside the original weights — typically well under 1% of the parameters.

The bet rests on a real observation: the change a task induces in the weights has low intrinsic rank, so it can be captured by the product of two skinny matrices instead of a full dense update. You get most of the quality of full fine-tuning while training a tiny fraction of the parameters, with far less memory and no gradient updates to the frozen base.

The downstream effects reshaped practice. Adapters are small (megabytes, not gigabytes), so you can store, share, and hot-swap many task-specific LoRAs over one shared base model — the pattern behind the explosion of community fine-tunes. Combined with 4-bit quantization of the base (QLoRA), it put fine-tuning a capable model within reach of a single consumer GPU.

LoRA has trade-offs — it can underperform full fine-tuning on tasks that need large representational shifts, and adapter rank is a knob to tune — but as the default parameter-efficient fine-tuning method, it is everywhere: serving stacks that swap adapters per request, on-device personalization, and the entire ecosystem of shareable model customizations. It turned fine-tuning from a big-lab activity into something anyone can do.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →