Small models, everywhere: the on-device AI handbook.

In 2022, "small" meant "toy." Today a modern 8B model beats the 175B giants of that era on most practical tasks. Two forces did it. Distillation: frontier models now generate and grade the training data, so a small model inherits capability instead of rediscovering it from raw internet text — every frontier release quietly raises the ceiling for the small models trained in its shadow. And data quality: the phi-family experiments proved that textbook-grade tokens beat web-scale sludge, parameter for parameter.

As a rule of thumb, not a standard, sizes sort roughly like this: sub-2B models for classification, extraction and autocomplete on phones; 3–8B for chat, summarization and RAG on laptops; 8–14B approaching last year's frontier for reasoning-lite work on a single consumer GPU.

Local inference speed obeys one equation. Generating a token means streaming essentially all active weights through the processor once. So:

Everything about the on-device world follows from this law. Quantization works because INT4 weights are 4× fewer bytes than FP16 — near-4× more tokens/sec, same bandwidth. Apple Silicon punches high because unified memory offers 100–800 GB/s to a big pool. NPUs matter for prefill and energy, but decode still queues at the bandwidth window. Size your expectations with the GPU VRAM calculator — the same math, one slider away.

Both halves are one multiplication. A model's size is params × bits/8, and quantization shrinks it enough to fit and speeds decode by the same factor:

Same bandwidth, a quarter of the bytes, so four times the tokens per second — and it finally fits in RAM. The runnable version below computes fit and speed for any model, precision, and device.

A request runs in two phases that stress different hardware. Prefill processes the whole prompt in parallel, so it is limited by compute, and its cost shows up as time to first token. Decode then emits one token at a time, so it is limited by the memory bandwidth described above, and its cost shows up as tokens per second. That split explains why an NPU can make a long prompt feel instant while chat speed barely changes, and why trimming a prompt helps latency more than a faster chip does.

Phones add a third constraint: heat. Sustained decode keeps the chip busy for as long as the answer runs, and the OS responds by throttling clocks. Apple's ProcessInfo.thermalState documentation says only that at higher thermal states "your app should reduce usage of system resources." So treat the speed from a short benchmark as a ceiling. Measure a five-minute run, and shorten outputs or fall back to the cloud when the thermal state climbs.

Weights are not the only thing that has to fit. Every token in the context adds a key and a value vector for each layer, and that KV cache grows with context length. For a model with L layers, H_kv key-value heads and head size d, FP16 costs 2 × L × H_kv × d × 2 bytes per token. Take Llama 3.2 3B, whose published config has 28 layers, 8 key-value heads and a head size of 128:

Grouped-query attention (8 KV heads instead of 24) already cut this by 3×. The KV Cache Calculator runs the same arithmetic for any config.

Then comes the cap you don't control. A phone app doesn't get "the device's RAM"; it gets a per-app limit, and the OS may terminate it above that. On iOS, Apple's com.apple.developer.kernel.increased-memory-limit entitlement tells the system that core features "may perform better by exceeding the default app memory limit on supported devices", and Apple adds that your app must still behave correctly if the extra memory isn't available. os_proc_available_memory() returns the bytes your app may still allocate, which Apple documents as advisory and warns "don't cache it." Budget from that number at load time, not from the spec sheet.

The pattern across the stack: OpenAI-compatible local APIs everywhere, so the same application code targets a cloud frontier model in production and a local 8B in development — or routes between them at runtime.

Quantization is where small models earn their fit. AWQ (Activation-aware Weight Quantization, MLSys 2024 Best Paper) starts from the observation that "protecting only 1% salient weights can greatly reduce quantization error," and that those channels are found from the activation distribution, not the weights. It scales them up before rounding, and needs no backpropagation or reconstruction. Quantization-aware training (QAT) goes further by simulating low precision during training. Google's Gemma 3 QAT release (April 2025) cut the 27B model's weight memory from 54 GB (BF16) to 14.1 GB (int4), which fits a single RTX 3090, and reduced the perplexity drop from quantizing by 54%. The same post lists the 4B model at 2.6 GB and the 1B at 0.5 GB, weights only.

For runtimes, ExecuTorch 1.0 (October 2025) deploys PyTorch models to phones, embedded boards and laptops without converting formats. It promoted XNNPACK, Core ML, Qualcomm's Hexagon NPU path and Arm Ethos-U to production-ready, and Meta says it powers on-device features in Instagram, WhatsApp, Messenger and Facebook. Next to llama.cpp, Ollama and MLX from the stack table, that's the "export from PyTorch" option: pick it when you want the model you trained and quantized in PyTorch to be the model that ships.

The mistake is framing it as either/or. The pattern that ships in 2026 is hybrid: a local model handles the instant, private, frequent 80% — autocomplete, classification, summarizing the meeting you just had — and a router-style fallback escalates the hard 20% to the cloud, with user consent where privacy is the point.

That last pattern deserves emphasis: a LoRA/QLoRA fine-tune of an 8B model on a few thousand task-specific examples routinely beats a prompted frontier model on that one task — at zero marginal cost, forever. Narrow beats general when you own the narrow.

Day-to-day, on-device work is an evals discipline. Leaderboard rank tells you little — small models vary wildly by task, and quantization moves quality in task-specific ways. The working loop: shortlist three candidate models in your size budget, build a golden set of 50–200 real examples from your task, score each model × quantization level (Q4 vs Q8 can matter more than model choice), and watch the tails — small models fail weirder than big ones, so read the worst 10 outputs, not the average.