QLoRA,
explained.

LoRA removes the optimiser state and leaves one cost untouched: the frozen base model still has to sit in accelerator memory, at sixteen bits per parameter, for the forward and backward passes to happen at all. For a large model that is the whole budget, so "cheap fine-tuning" still meant renting a cluster.

The change is to quantise the frozen base to four bits and backpropagate straight through it into the adapters. The reason this is sound rather than reckless is worth stating precisely, because it is the conceptual core: the quantised weights are never updated. A quantised weight that you train would accumulate rounding error step after step; a quantised weight that is frozen is just a fixed distortion of the original function, and the adapters — which are trained in full precision — can learn to compensate for it. Gradients pass through the quantised weights to reach the adapters without ever being applied to them.

Three pieces make the four bits survivable, and each targets a specific leak. The data type is chosen so that its quantisation levels match the distribution weights actually have — approximately normal and centred on zero — rather than being spaced uniformly, which would waste most of sixteen available levels on ranges few weights occupy. The per-block scaling constants are themselves quantised a second time, because at four bits the metadata is no longer negligible against the data. And optimiser state is paged to host memory so a transient spike fails slowly instead of fatally. Stated in units: a 65B-parameter fine-tune on a single GPU.

Training memory isn't just weights. Full fine-tuning with AdamW holds the 16-bit weights, 16-bit gradients, and two 32-bit optimizer states — roughly 16 bytes per parameter before activations. For a 65B model that's north of a terabyte; even LoRA, which shrinks gradients and optimizer state to almost nothing, still needs the frozen base model resident in 16-bit: 130 GB for 65B. The adapters were cheap; the statue they wrapped around was not.

The obvious move — store the frozen base in 4-bit — had an obvious objection: everyone "knew" 4-bit was for inference, and training through quantized weights would degrade everything. QLoRA's contribution is showing that with the right data type and plumbing, the objection is wrong.

QLoRA freezes the base model in 4-bit NF4 and trains 16-bit LoRA adapters on top. During each forward/backward pass, the touched 4-bit weights are dequantized to 16-bit on the fly, used, and discarded — gradients flow through the frozen quantized weights but are only applied to the adapters.

Sixteen levels is all 4 bits buy you. Spend them evenly across the range (like INT4 does) and you waste levels where weights are rare. Neural weights are approximately normally distributed — bunched near zero, thin tails — so QLoRA's NormalFloat-4 places its 16 levels at the quantiles of a normal distribution: each level owns an equal share of the actual weights.

It's the information-theoretic answer to "where do I put 16 points to describe a bell curve?" — dense levels where weights crowd, sparse levels in the tails. In ablations, NF4 beat standard FP4 consistently — the difference between 4-bit-that-degrades and 4-bit-that-doesn't.

Block-wise quantization stores a scale constant per block of weights — and at QLoRA's small block size (64, chosen for accuracy), those 32-bit constants add up: ~0.5 bits per parameter of pure overhead, gigabytes at 65B scale. QLoRA's cheeky fix: quantize the quantization constants too — the per-block scales get their own 8-bit quantization with a second-level constant.

It saves ~0.37 bits/param — about 3 GB on a 65B model. Not glamorous, but it's the difference between fitting on the card and not. Systems papers are won in the decimals.

Watch the decimals add up. A 32-bit scale per 64-weight block is pure overhead on top of the 4 NF4 bits:

That 0.373-bit gap, multiplied across 65 billion parameters, is the 3 GB — and 3 GB is exactly the margin between "fits on a 48 GB card" and "doesn't." Put the whole frozen base on the scale next to a normal fine-tune:

The adapters and their optimizer state are ~1% of this and ride in the headroom — the frozen base was the whole memory problem.

Even with everything above, training memory spikes — a long sequence's activation checkpointing can momentarily demand gigabytes, and one spike OOMs an otherwise-fitting run. QLoRA uses paged optimizers: optimizer states live in unified memory that the NVIDIA driver pages between GPU and CPU on demand, like OS virtual memory. Spikes swap out; training hiccups instead of dying.

Together the three tricks turn "fits on average" into "fits, period" — the practical difference between a paper result and a tool people actually run overnight on one machine.

QLoRA is why "fine-tune it on your own data" became a weekend project instead of an infrastructure program. Shipped into bitsandbytes and Hugging Face PEFT within weeks, it became — and remains — the default recipe behind most open-model fine-tunes you've ever downloaded. The NF4-frozen-base pattern also normalized a bigger idea: quantization isn't just a deployment trick, it's a training-time tool.

Conceptually it completed LoRA's argument. LoRA said: the update is low-rank, so shrink the training. QLoRA added: the base is redundant, so shrink the storage. Between them, the two papers cut fine-tuning's cost by orders of magnitude in each dimension — and the open-source model ecosystem is the compound interest.

QLoRA stores the frozen base model in 4 bits, computes in 16 bits and trains LoRA adapters in 16 bits. Each piece below is specified exactly in the paper, and together they deliver the headline: fine-tuning a 65B model in under 48 GB of GPU memory, where regular 16-bit fine-tuning needs more than 780 GB.

1. Build the NF4 code book. The 16 values come from quantiles of a standard normal distribution: each value averages two neighboring quantiles, and the set is normalized into [−1, 1]. To represent zero exactly, the negative and positive halves are estimated separately, 2^^k−1 quantiles for the negative side and 2^^k−1 + 1 for the positive side, and one of the two duplicate zeros is dropped. The result is 7 negative values, an exact 0 and 8 positive values.
2. Quantize block by block. Each weight tensor is cut into blocks of 64. A block's absolute maximum rescales it into [−1, 1], and each weight is stored as the 4-bit index of the nearest NF4 value. Small blocks stop one outlier from crushing the resolution of many weights.
3. Double-quantize the block constants. The per-block scales, one 32-bit float per 64 weights, are themselves quantized to 8-bit floats in blocks of 256 after subtracting their mean, with one 32-bit constant per block of scales. Overhead falls from 0.5 to about 0.127 bits per parameter.
4. Dequantize on the fly and compute in BF16. In every forward and backward pass, the weights a layer needs are expanded from NF4 to BFloat16, used and discarded. Storage stays near 4.1 bits per parameter while the arithmetic is 16-bit.
5. Put LoRA adapters on every linear layer. The adapters are BF16 and are the only trainable parameters. Computing their gradients requires backpropagating through the dequantized base weights, but no update is ever applied to those weights. Adapters on the query and value projections alone, the original LoRA default, did not match 16-bit fine-tuning on large models; adapters on all linear layers of each Transformer block did, and once they were everywhere the rank barely mattered. The chatbot runs used r = 64 and α = 16.
6. Page the optimizer. Adam's states live in NVIDIA unified memory, which the driver moves to CPU RAM when the GPU runs short, so a long sequence's memory spike slows a step instead of ending the run.
7. Train and evaluate at scale. LLaMA models from 7B to 65B were fine-tuned on eight instruction datasets: OASST1, HH-RLHF, Alpaca, Self-Instruct, Unnatural Instructions, FLAN v2, Chip2 and Longform. More than 1,000 models were trained in all, and the best chatbots, called Guanaco, were trained on OASST1.

The same arithmetic scales down. The 33B model trains on a single 24 GB consumer GPU in under 12 hours, and the deployed Guanaco 7B, still in 4 bits, needs about 5 GB of memory.

• W^NF4 — the frozen base weights, stored as 4-bit indices into the NF4 code book.
• c2 — the per-block absmax scales, one per 64 weights, stored as 8-bit floats; c1 — one 32-bit constant per 256 of those scales.
• L1, L2 — the LoRA factors, the only trained tensors (the LoRA scale is left out here, as in the paper's own equation).
• QX — the quantile function of the standard normal distribution; k = 4 bits.

At the extremes of block size: one weight per block would make quantization almost lossless but spend 32 bits of scale on every 4-bit weight, while one block per tensor would let a single large weight set the scale for millions of others and push most of them onto the levels nearest zero. Blocks of 64 are the compromise, and double quantization is what makes a block that small affordable.

The paper first checked that 4-bit training loses nothing, by replicating 16-bit full fine-tuning and 16-bit LoRA on RoBERTa-large (GLUE) and T5 (Super-NaturalInstructions), then fine-tuned LLaMA 7B to 65B and measured MMLU and chatbot quality. Chatbots were judged on the 80-prompt Vicuna benchmark and on OASST1 prompts, by GPT-4 and by human raters, including an Elo tournament between systems.

NF4 is what makes 4 bits safe: FP4 trailed the 16-bit baseline by about a point on MMLU, while NF4 with double quantization matched it. The data findings were less expected. A 9k-example OASST1 set beat a 450k-example FLAN v2 subset on chatbot quality, so suitability mattered more than size, and MMLU scores did not predict chatbot scores in either direction.

The ablations cut against defaults too. LoRA on only the query and value projections fell short of full fine-tuning on large models; adapters on all linear layers were required. And the evaluation instrument had biases the authors measured themselves: GPT-4 favored whichever system appeared first in its prompt, and it rated its own answers higher than human raters did, an Elo of 1348 from GPT-4 against 1176 from humans.

The trade is acknowledged: dequantising weights on the fly makes each step slower than the same fine-tune in sixteen bits. This buys feasibility, not speed — the right comparison is against not being able to run at all.

The part that was contested is the evaluation rather than the method. The headline quality claims rested substantially on judgements by another language model and on preference benchmarks that were, at the time, new and not well understood. Later work documented systematic biases in model-as-judge evaluation — toward longer answers, toward a position in the comparison, toward outputs resembling the judge's own style. The technique's value did not depend on those numbers and has survived them, but "matches sixteen-bit fine-tuning" was measured with an instrument that was itself under review.

How it held up: four-bit base plus full-precision adapters is simply how local fine-tuning is done, and the distribution-matched data type and double quantisation were absorbed into the quantisation toolchain generally. The wider significance is who it moved: before this rung, adapting a large model required an organisation; after it, a person with one GPU. Most of what the open-weight ecosystem on the next ladder produced downstream of each release depended on this being possible.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →