Small models, everywhere: the on-device AI handbook.

In 2022, "small" meant "toy." Today a modern 8B model beats the 175B giants of that era on most practical tasks. Two forces did it. Distillation: frontier models now generate and grade the training data, so a small model inherits capability instead of rediscovering it from raw internet text — every frontier release quietly raises the ceiling for the small models trained in its shadow. And data quality: the phi-family experiments proved that textbook-grade tokens beat web-scale sludge, parameter for parameter.

The result is a genuine tier system: sub-2B models for classification, extraction and autocomplete on phones; 3–8B for chat, summarization and RAG on laptops; 8–14B approaching last year's frontier for reasoning-lite work on a single consumer GPU.

Local inference speed obeys one equation. Generating a token means streaming essentially all active weights through the processor once. So:

Everything about the on-device world follows from this law. Quantization works because INT4 weights are 4× fewer bytes than FP16 — near-4× more tokens/sec, same bandwidth. Apple Silicon punches high because unified memory offers 100–800 GB/s to a big pool. NPUs matter for prefill and energy, but decode still queues at the bandwidth window. Size your expectations with the GPU VRAM calculator — the same math, one slider away.

The pattern across the stack: OpenAI-compatible local APIs everywhere, so the same application code targets a cloud frontier model in production and a local 8B in development — or routes between them at runtime.

The mistake is framing it as either/or. The pattern that ships in 2026 is hybrid: a local model handles the instant, private, frequent 80% — autocomplete, classification, summarizing the meeting you just had — and a router-style fallback escalates the hard 20% to the cloud, with user consent where privacy is the point.

That last pattern deserves emphasis: a LoRA/QLoRA fine-tune of an 8B model on a few thousand task-specific examples routinely beats a prompted frontier model on that one task — at zero marginal cost, forever. Narrow beats general when you own the narrow.

Day-to-day, on-device work is an evals discipline. Leaderboard rank tells you little — small models vary wildly by task, and quantization moves quality in task-specific ways. The working loop: shortlist three candidate models in your size budget, build a golden set of 50–200 real examples from your task, score each model × quantization level (Q4 vs Q8 can matter more than model choice), and watch the tails — small models fail weirder than big ones, so read the worst 10 outputs, not the average.