LABS · 17  /  EFFICIENCY

The Rounding Room.

A model is billions of numbers, each stored in 16 bits. But do you need all 16? Round each weight to a coarser grid — 8 bits, 4, even 2 — and the model shrinks by that factor, while precision blurs. Watch the trade play out, and discover why NF4 beats plain integers by putting its levels where the weights actually live.

Act 1 · Rounding to a grid
Snap a weight to the nearest level
A quantized format has only a fixed set of representable values — a grid. Every real weight snaps to its nearest grid point, and the gap is the rounding error. Drag the weight, change the bit-width, and watch the grid get coarser as bits drop.
Bits:
Original weight0.37
Original value Nearest grid level Rounding error
16
Grid levels
2^bits
0.33
Stored value
0.04
Rounding error
Fewer bits = fewer levels = a coarser grid = bigger rounding error. INT8 has 256 levels (fine), INT4 has 16 (usable), INT2 has 4 (brutal). The art is losing bits without losing the model.
Act 2 · Levels where the weights live
INT4 vs NF4 on a real distribution
Neural weights aren't spread evenly — they bunch near zero in a bell curve. Plain INT4 spaces its 16 levels evenly, wasting most on the empty tails. NF4 puts its levels at the quantiles of the bell — dense where weights crowd. Toggle and watch the total error drop.
Format:
Weight distribution Grid levels (16)
Total rounding error
NF4 improvement
vs even INT4
16
Levels (both)
Same 16 levels, same 4 bits — but NF4 spends them where weights actually are. That's the whole idea behind QLoRA's frozen 4-bit base: no quality loss, a quarter of the memory.
Act 3 · The trade, at scale
Memory saved vs quality kept
This is the decision every deployment makes. Pick a model size and a precision and see what fits — and roughly what quality you keep. Below ~4 bits the curve turns: error accumulates faster than memory savings justify.
Model size70B
Precision:
— GB
Model memory
vs FP16
Smallest GPU that fits
Quality retained
The sweet spot for most deployments is INT4/NF4: ~4× smaller, quality essentially intact. INT8 is the zero-risk default; INT2 is research territory where cleverness fights collapse.

Why you can throw bits away

A model's memory is brutally simple: parameters × bytes-per-parameter. FP16 spends 2 bytes each; a 70B model is ~140 GB. Quantization attacks the second factor — store each weight in 1 byte (INT8), half a byte (INT4), a quarter (INT2) — and the model shrinks by exactly that ratio. A 70B model at INT4 is ~35 GB: suddenly it fits on one card.

The surprise is that this works. Rounding each weight introduces error, yet the model barely notices — because meaning is spread across billions of parameters, and no single rounded weight moves the output much. Down to about 4 bits, done carefully, quality loss is often within noise. Below that, errors compound and the model starts to slip.

The NF4 insight

Weights are approximately normally distributed — a bell curve bunched near zero. Evenly spaced levels (INT4) waste most of their 16 slots on the empty tails. NF4 places levels at the quantiles of a normal distribution, so each level covers an equal share of the actual weights — dense in the middle, sparse in the tails. Same bits, far less error. It's the format QLoRA froze its base model in.

It's also a speed win

Decode is memory-bandwidth-bound: generating a token means streaming every weight through the GPU. Fewer bytes per weight means less traffic, so quantization often makes generation faster too — the same reason it fits on smaller cards. Note one subtlety: quantizing weights shrinks the model, but the KV cache is a separate budget with its own dtype.

The catch

Not all quantization is free. Naive rounding of everything hurts; production methods (GPTQ, AWQ, NF4) protect the weights that matter, use per-block scales, and sometimes keep sensitive layers in higher precision. And activations are trickier than weights. But the headline holds: 4-bit is the practical floor where models stay sharp and memory drops 4× — which is why it powers on-device AI and cheap serving alike.

Check yourself

Four questions. Round to the nearest right answer.

Finished this one? 0 / 22 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs