LoRA freezes the pretrained weights and learns small low-rank update matrices for selected layers. You get most of full fine-tuning’s quality while training well under 1% of the parameters — cheap to train, tiny to store, and swappable per task at serve time. QLoRA pushes further by keeping the frozen base in 4-bit. This is the default way anyone fine-tunes open models today.
Worked example: for a d×d weight matrix, LoRA trains two skinny matrices d×r and r×d (rank r≈8–16) instead of the full d² weights — for d=4096, r=16 that is ~0.8% of the parameters. Gotcha: LoRA captures low-rank updates well but underperforms full fine-tuning when a task needs large representational shifts; raise the rank or fall back to full tuning then.