Quantization vs distillation: shrink the numbers, or the model?

Quantization keeps the exact same model — same architecture, same parameter count — but stores each weight in fewer bits: dropping from 16-bit floats to 8-bit or 4-bit integers roughly halves or quarters the memory, with no change to what the model is. Distillation does the opposite: it trains a brand-new, smaller student model to imitate the outputs of a larger teacher, producing a genuinely smaller network with fewer parameters. Quantization compresses the representation; distillation compresses the model.

Because they act on different axes, the two compose cleanly and teams routinely chain them: distill a large teacher into a smaller student to cut the parameter count, then quantize that student to INT8 or INT4 to cut the memory footprint further. A third axis, pruning (zeroing out unimportant weights), stacks on top of both. Together they can hit a size-and-latency target that no single technique reaches — say, taking a 70B teacher down to a distilled 7B student and then quantizing it to 4-bit to run on a single modest GPU.

Each technique degrades quality in its own way. Quantization is remarkably forgiving down to 8-bit — the loss is usually small enough to ignore, which is why INT8 is a safe default. Push to 4-bit and below and accuracy slips more noticeably; quantization-aware training (simulating low precision during training, rather than just rounding a finished model) recovers much of that loss at aggressive bit-widths. Crucially, quantization keeps the model's full breadth — it still knows everything it knew, just a little less precisely.

Distillation trades differently. A well-trained student can match the teacher closely on the tasks it was distilled for, sometimes almost indistinguishably. But it's a smaller network with less capacity, so it tends to lose general breadth — the long tail of rare knowledge and edge behaviors the big teacher had. That's fine when you want a specialist and dangerous when you quietly needed a generalist. The honest framing: quantization keeps breadth and sacrifices a little precision everywhere; distillation keeps sharpness on-task and sacrifices coverage off-task.

You have a strong 70B model and a single GPU that can't hold it at 16-bit precision. The fastest path is quantization: drop it to INT4 and its memory footprint falls by roughly 4×, often fitting on that one card, with quality good enough for many workloads — and you did it in an afternoon with no training data. It's still the same 70B model, just stored more coarsely. This is the right first move almost every time.

Now suppose that quantized 70B is still too slow or too expensive per token at your request volume, and your actual task is narrow — say, classifying and routing support tickets. Here distillation earns its cost: train a 7B (or smaller) student on the big model's outputs for that task, and you get a model that's an order of magnitude cheaper and faster to run, purpose-built for the job. Then quantize that student too. The 70B was a great teacher; it doesn't have to be what you serve.

Both shrink a model, but differently: quantization stores the same weights in fewer bits (no training, minutes to apply); distillation trains a new, smaller model to imitate the teacher (a real training run). So the deciding number is how much smaller you need.