LoRA,
explained.

Rungs 1 to 3 all end in a gradient step on every parameter, which means the cost of adapting a model scales with the model. What actually makes that expensive is not the weights — it is the optimiser state, since Adam keeps two running averages per trainable parameter, so a full fine-tune needs roughly three times the model's memory before activations. Adapting a large model therefore needs the hardware that trained it.

The change rests on an observation about what a fine-tune actually does: the update to a weight matrix has much lower rank than the matrix itself — adaptation moves the model within a small subspace. So do not represent the update as a full matrix. Freeze the pretrained weights and learn the update as a product of two thin matrices, one projecting down to a small rank and one back up. Only those two are trained, so the optimiser state shrinks with the rank rather than the model.

Stated in units: trainable parameters fall by up to four orders of magnitude, and with them the optimiser memory and the size of what you have to store per task. Two consequences matter as much as the saving. First, because the learned update is a product that can be added into the frozen weights before serving, there is no extra inference latency at all — which is what distinguishes this from adapter layers that insert extra depth into the network and pay for it on every token forever. Second, many task adapters can share one copy of the base model in memory, so a fleet serving a hundred fine-tunes needs one model and a hundred small deltas.

Fine-tuning is how you adapt a pretrained model to your task — your tone, your domain, your format. The classic recipe updates every weight in the model. For a model with billions of parameters, that means holding a full copy in GPU memory, optimizing all of it, and saving a fresh multi-gigabyte model per task.

Want ten specialized variants? Ten full copies. It's like rebuilding an entire skyscraper from the foundation up every time you want to repaint one room. Prohibitive for anyone without a data center — and wasteful even for those who have one.

The authors started from a hunch that turned out to be the whole paper: the change a model needs to learn a new task is far simpler than the model itself. Formally, the weight update has low intrinsic rank — it lives in a tiny subspace, not the full billion-dimensional space.

If the update is that simple, you don't need to represent it with a giant dense matrix. You can capture it with something much smaller — and that's the opening LoRA walks through.

Take a weight matrix W in the pretrained model. LoRA freezes it — never touches it again — and adds a side path: two small matrices, A and B, whose product represents the update. The layer now computes W·x + B·A·x, and only A and B are trained.

The trick is the shape. If W is d×d, then A is r×d and B is d×r, where the rank r is tiny — often just 4 or 8. Their product is the same size as W, but they contain a few thousand times fewer numbers to train.

Watch a rank-1 update describe a whole matrix. Say training produced these two tiny factors for a 4-dimensional layer (B starts as zeros — so LoRA begins as a no-op — and learns; A starts random):

Look at the rows of ΔW: they're 1×, 2×, 0×, and 1× the same row A. Eight stored numbers just described a sixteen-number update — that's what rank 1 means. The matrix can only push inputs toward one direction, but the paper's bet is that fine-tuning barely needs more than a few such directions. The bet pays off brutally well at real widths (d = 4096, one layer):

The forward pass adds the side path, scaled by α/r so different ranks train with similar learning-rate behavior:

Applied to GPT-3 175B (rank 4, attention W_q and W_v only), this cut trainable parameters 10,000× and the per-task checkpoint from 350 GB to 35 MB.

Earlier efficient-tuning methods bolted extra layers into the network, which made every prediction a little slower forever. LoRA sidesteps this entirely. After training, you can merge the small update back into the original matrix — W' = W + B·A — producing a model that is byte-for-byte the same size and speed as the original.

LoRA changes which parameters are trained; once the update is merged, a request runs through exactly the original computation. The recipe below is the one the paper used on RoBERTa, DeBERTa, GPT-2 and, at the largest scale, GPT-3 175B.

1. Choose which matrices to adapt. Each Transformer layer has four attention projections, W_q, W_k, W_v and W_o, and two MLP matrices. The paper adapts attention weights only and freezes the MLP blocks, LayerNorm and biases, for simplicity and parameter efficiency, leaving the rest to future work. The headline GPT-3 runs adapt W_q and W_v.
2. Attach two thin factors. For a frozen W₀ of shape d × k, add B (d × r) and A (r × k), with r much smaller than d and k. A starts from a random Gaussian and B from zeros, so ΔW = BA is exactly zero at step 0 and training begins from the pretrained model's own behavior.
3. Scale the side path by α/r. The layer computes h = W₀x + (α/r)·BAx. With Adam, tuning α is roughly the same as tuning the learning rate, so the authors set α to the first r they tried and never tuned it. Dividing by r reduces the need to re-tune hyperparameters when you change the rank.
4. Train only A and B. Gradients still flow backward through W₀ to reach earlier layers, but no gradient is kept for W₀ itself and the optimizer stores no state for it. On GPT-3 175B this cut training memory from 1.2 TB to 350 GB and gave about a 25% training speedup over full fine-tuning, because most weight gradients are never computed.
5. Spend a fixed budget across matrices. At 18M trainable parameters on GPT-3, about 35 MB in fp16, the budget buys rank 8 on one matrix type, rank 4 on two or rank 2 on all four, in all 96 layers. Spreading it over W_q and W_v, or over all four, beat concentrating it in one (results below).
6. Deploy by merging, or keep adapters apart. After training, W = W₀ + (α/r)·BA is computed once and served like the original model, with no added latency. Switching tasks means subtracting one BA and adding another. The paper names the catch: once merged, one batch cannot easily mix requests for different tasks; keeping adapters unmerged allows per-request adapters at some cost in speed.

What the learned update looks like

The authors probed ΔW directly. Adapters trained at r = 8 and r = 64 on the same task share their top singular direction strongly, with normalized subspace similarity above 0.5 for both W_q and W_v, while the other directions of the rank-64 adapter look mostly like noise: evidence that the useful update is genuinely very low rank. ΔW also correlates with W₀ more than a random matrix would, but it amplifies directions that W₀ does not emphasize, and by a large factor, about 21.5 for r = 4 in the layer they examined.

Later work revisited two defaults. Rank-stabilized LoRA (rsLoRA) argued that α/r shrinks updates too much at large ranks and proposed α/√r instead, and DoRA splits each weight into a magnitude and a direction and applies the low-rank update to the direction. Serving systems such as S-LoRA and Punica keep thousands of adapters unmerged and batch requests across them with custom kernels, which works around the batching limit.

• W0 — the pretrained weight, frozen; it receives no gradient update and no optimizer state.
• A — the down-projection to r dimensions, Gaussian-initialized; B — the up-projection back to d, zero-initialized, so ΔW starts at exactly zero.
• r — the rank: the number of independent directions the update can use.
• α / r — a fixed scale. α was set equal to the first r tried, so at that rank the scale is exactly 1.

For GPT-3, with d = k = 12,288 and rank 4 on W_q and W_v in all 96 layers: 2 · 96 · 4 · 24,576 ≈ 18.9M trainable parameters, about 0.01% of 175B. At the extremes, r = min(d, k) lets BA represent any update, so LoRA becomes as expressive as fully fine-tuning that matrix, while r = 1 confines the update to a single direction. On W_q plus W_v, rank 1 was already within half a point of rank 8 on both WikiSQL and MultiNLI.

The paper compared LoRA with full fine-tuning, BitFit (bias-only tuning), prefix-embedding and prefix-layer tuning, and adapter layers, on RoBERTa and DeBERTa (GLUE), GPT-2 (E2E NLG and other table-to-text tasks) and GPT-3 175B (WikiSQL, MultiNLI-matched and SAMSum). The GPT-3 results carry the argument:

The which-matrices ablation fixed the budget at 18M parameters and moved it around:

Putting the whole budget into W_q or W_k was clearly worse; W_q and W_v at rank 4, or all four at rank 2, matched or beat every single matrix at rank 8. More rank did not help either: on W_q and W_v, rank 64 scored 73.5 on WikiSQL against 73.8 at rank 8 and 73.4 at rank 1. The weak results belong to the baselines. Prefix-embedding tuning reached only 63.1 on WikiSQL, and on GPT-2 medium at batch size 1, two adapter designs added 20.7% and 30.3% latency, the cost LoRA's merge avoids.

The paper is clear that the rank is chosen empirically — there is no principle that tells you what it should be — and that surprisingly small ranks suffice, which is evidence for the low-rank premise and also a result nobody predicted the size of. It also reports adapting only the attention projections rather than every matrix, again on measurement rather than derivation, and does not claim parity with full fine-tuning in all cases.

The limitation the field settled on later is a sharpening of exactly that premise. Low-rank updates are excellent at teaching a model a style, format or task interface, and comparatively weak at installing substantial new knowledge — which follows from the hypothesis rather than contradicting it: if adaptation is genuinely confined to a small subspace, then anything requiring a large change is out of reach by construction. The practical rule that emerged — reach for this to change behaviour, reach for continued pretraining to change what the model knows — is a consequence of the paper being right, not wrong.

How it held up: it is the default way anyone fine-tunes anything. The multiple-adapters-one-base pattern became a serving architecture, the variants that followed mostly adjust how the rank is allocated across layers, and the next rung removes the one cost this one leaves standing — that the frozen base still has to fit in memory at full precision.

Because each task's LoRA is just a few megabytes of A and B, you can keep one frozen base model and a shelf of tiny adapters — one for legal writing, one for code, one for your brand voice — and swap them in and out on the fly. Ten specialized models become one big model plus ten postage-stamp files, instead of ten skyscrapers.

LoRA turned fine-tuning from a big-lab luxury into something a solo developer can do on a rented GPU — and it became the default of the whole parameter-efficient fine-tuning (PEFT) family. Its descendant QLoRA quantizes the frozen base to 4-bit and trains adapters on top, letting you fine-tune enormous models on a single consumer card.

Every "custom fine-tune," every marketplace of swappable model adapters, every hobbyist tuning a model at home traces back to one observation in this paper: the update you need is small, so train something small.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →