LoRA,
explained.

Fine-tuning is how you adapt a pretrained model to your task — your tone, your domain, your format. The classic recipe updates every weight in the model. For a model with billions of parameters, that means holding a full copy in GPU memory, optimizing all of it, and saving a fresh multi-gigabyte model per task.

Want ten specialized variants? Ten full copies. It's like rebuilding an entire skyscraper from the foundation up every time you want to repaint one room. Prohibitive for anyone without a data center — and wasteful even for those who have one.

The authors started from a hunch that turned out to be the whole paper: the change a model needs to learn a new task is far simpler than the model itself. Formally, the weight update has low intrinsic rank — it lives in a tiny subspace, not the full billion-dimensional space.

If the update is that simple, you don't need to represent it with a giant dense matrix. You can capture it with something much smaller — and that's the opening LoRA walks through.

Take a weight matrix W in the pretrained model. LoRA freezes it — never touches it again — and adds a side path: two small matrices, A and B, whose product represents the update. The layer now computes W·x + B·A·x, and only A and B are trained.

The trick is the shape. If W is d×d, then A is r×d and B is d×r, where the rank r is tiny — often just 4 or 8. Their product is the same size as W, but they contain a few thousand times fewer numbers to train.

Earlier efficient-tuning methods bolted extra layers into the network, which made every prediction a little slower forever. LoRA sidesteps this entirely. After training, you can merge the small update back into the original matrix — W' = W + B·A — producing a model that is byte-for-byte the same size and speed as the original.

Because each task's LoRA is just a few megabytes of A and B, you can keep one frozen base model and a shelf of tiny adapters — one for legal writing, one for code, one for your brand voice — and swap them in and out on the fly. Ten specialized models become one big model plus ten postage-stamp files, instead of ten skyscrapers.

On GPT-3 175B, LoRA cut the number of trainable parameters by about 10,000x and GPU memory by roughly 3x, while matching or beating full fine-tuning on the benchmarks tested — with no added inference latency.

Same quality, four orders of magnitude fewer parameters to train and store.

LoRA turned fine-tuning from a big-lab luxury into something a solo developer can do on a rented GPU — and it became the default of the whole parameter-efficient fine-tuning (PEFT) family. Its descendant QLoRA quantizes the frozen base to 4-bit and trains adapters on top, letting you fine-tune enormous models on a single consumer card.

Every "custom fine-tune," every marketplace of swappable model adapters, every hobbyist tuning a model at home traces back to one observation in this paper: the update you need is small, so train something small.