LLM vs SLM: which one fits where it has to run?

An LLM is a frontier-scale model — tens to hundreds of billions of parameters or more — typically served from a datacenter over an API, with broad general reasoning and deep long-tail knowledge. An SLM (small language model, roughly under ~10B parameters, often 1–3B) is compact enough to run on-device — a phone, a laptop, an edge box — with no network round-trip.

Model routing (cascade) is the pattern that gets used at scale: send requests to a cheap SLM first, and escalate only the ambiguous or hard ones to a bigger LLM. Because most real traffic is easy, this cuts cost and latency dramatically without giving up quality on the requests that actually need it. An on-device SLM can also do a first pass and call out to a cloud LLM only when its own confidence is low.

The "small = weak" assumption is increasingly outdated for a specific reason: the recent gains in small models came primarily from better DATA CURATION and training recipes, not just from adding parameters — the "textbooks are all you need" line of research showed that a smaller model trained on carefully curated, high-quality data can outperform a much larger model trained on noisier, less curated web data. Modern ~3B-class SLMs now often beat 2023-era 70B-class models on many benchmarks, which means "small" increasingly describes efficient-for-its-capability, not simply weak.

This reframes the whole comparison: the LLM-vs-SLM line is less and less about a fixed capability gap that's permanently true, and more about DEPLOYMENT constraints — where does this need to run, what's the latency budget, does privacy require the data to never leave the device. Those constraints don't change as models get better; they're determined by the product, not the model generation.

A mobile keyboard app's autocomplete and autocorrect need sub-50ms suggestions with zero network dependency — sending every keystroke to a cloud API would be both too slow and a real privacy and cost problem at that volume. An SLM (or smaller) running on-device is the only fit: the constraint isn't "what's the best possible suggestion," it's "what can respond fast enough, offline, without leaking every keystroke."

The same app's "help me rewrite this paragraph" feature is invoked rarely, by explicit user action, and tolerates a second or two of latency — a much better fit for calling out to a cloud LLM, where the higher-quality rewrite is worth the round-trip because the constraint here (occasional use, quality matters, latency is forgivable) is the opposite of autocomplete's. Same app, same company, two different deployment constraints, two different right answers.