The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Millions of devices you don't fully control
Datacenter inference serving assumes reliable connectivity, homogeneous hardware, and servers you can SSH into. None of that holds for an edge fleet: devices are tiny, intermittently connected, wildly heterogeneous, and you can't just log in and check what's wrong.
We'll build for that reality: model compression to fit real constraints, offline-capable inference, staged rollout that accounts for devices you can't instantly reach, bandwidth-conscious updates, and devices that verify their own state rather than trusting a push arrived.
What the new pieces do
- Edge Deviceclient
- Tiny memory and compute compared to a datacenter GPU, intermittently connected, and one of potentially millions running the same model.
Step 1 · The baseline
Every request goes to the cloud
Simplest version: every edge device sends every inference request to a central cloud server and waits for a response. What breaks at real fleet scale?
Every inference request from every device goes to a central cloud server. What breaks at fleet scale?
Server scaling doesn't solve the deeper issues here: real-time use cases need latency a round trip to the cloud can't guarantee, bandwidth/cost scale directly with fleet size and request volume, and the device becomes totally unusable the instant it loses connectivity — none of which more server capacity fixes.
Round-tripping every inference to the cloud stacks network latency onto every request (bad for anything real-time), costs scale linearly with a potentially enormous fleet's total request volume, and — critically — the device simply can't function at all the moment its connection drops, which for genuinely mobile or remote edge devices is a common, not rare, occurrence.
Even devices with generally good connectivity experience real, if infrequent, connectivity gaps — and the latency and bandwidth cost problems exist regardless of connectivity quality, affecting every device in the fleet, not just poorly-connected ones.
Cloud-only inference stacks network latency onto every request, scales cost directly with fleet size, and makes every device totally dependent on connectivity that edge devices often don't reliably have. Real-time, cost-efficient, resilient operation requires the device to do at least some inference itself.
Why this piece earns its place
The cloud path is worth keeping on the diagram even after inference moves on-device, and deleting it is the version of this answer that loses the argument. Some work still belongs in the datacenter — a model too large to compress usefully, a request the device's hardware genuinely cannot serve — so the real decision is not cloud or edge but which slice crosses the network, and that should be a runtime choice rather than one frozen at build time. The second-order consequence is the one that bites. Putting inference on hardware you do not own turns a deploy into a release: you can no longer fix a bad model by restarting a process, you inherit whatever update channel the platform gives you, and your inference code is now pinned against OS versions and platform runtimes you do not control, which change underneath you without asking. Everything after this step is the price of that. Skip it while the fleet is small and the requests are occasional: the round trip is cheaper than the release machinery, and shipping the boring version stays the right call until either the device count or a latency requirement makes it the wrong one.
What the new pieces do
- Fleet Control Planegateway
- The central point that produces, versions, and rolls out models to the fleet — but never assumes it has a live connection to every device.
Step 2 · Compress the model to fit
A datacenter model doesn't fit on a phone
Move inference onto the device itself. A model sized for a datacenter GPU is often far too large for an edge device's memory, compute, and battery budget. What has to happen first?
A model built for datacenter serving is far too large for edge hardware. What needs to happen before it can run on-device?
Even capable modern edge devices have memory, compute, and battery budgets orders of magnitude smaller than a datacenter GPU — most full-size models simply won't fit or run acceptably without real compression.
Model compression techniques shrink a model's footprint substantially, at some accuracy cost, in exchange for something that can genuinely run within an edge device's real, severe resource constraints — an explicit, deliberate trade rather than assuming the full model just fits.
Restricting to only the most powerful edge devices would exclude the vast majority of a realistic fleet — the whole point is making inference work across genuinely resource-constrained, typical edge hardware.
Run models through a Compression Pipeline — quantization, distillation, pruning — before they're deployable to edge hardware, producing a genuinely smaller, edge-appropriate model that trades some accuracy for actually fitting within real device resource budgets.
Why this piece earns its place
Compression does not produce a model, it produces one artifact per model-and-target pair, and that multiplication is what really sets the cost of this box. A quantized graph leans on kernels the silicon actually implements: the integer operation one accelerator runs natively may be missing on another, so the same weights get built, packed and shipped several times over. What follows from that is a versioning problem rather than a size one. The recipe — bit widths, which layers stay in float, the calibration sample used to pick the ranges — is an input to the artifact, and so is the converter that applied it. Pin both. A recipe on its own will not rebuild an old release once the toolchain has moved a version or two, and the first time you need that rebuild is the worst possible moment: a device is misbehaving on a build you shipped and can no longer reproduce, so you cannot even tell whether the fault is in the weights or in the packing. Skip the pipeline entirely when there is exactly one target: hand-tuning a single build beats automating it. The moment a second class of hardware appears, that stops scaling — and building the matrix before you have shipped into it is far cheaper than reconstructing it afterwards.
What the new pieces do
- Compression Pipelineservice
- Shrinks a model to fit severe on-device memory, compute, and battery budgets — trading some accuracy for the ability to run at all on the target hardware.
- Compressed Model Versionsdata
- Versioned, compressed model artifacts specifically built for edge deployment — distinct from full-size server-serving models.
Step 3 · Run inference locally, offline-capable
The device shouldn't need a live connection to function
With a compressed model deployed, should the device still call the cloud for every inference, or run entirely on its own?
The device runs inference locally, entirely offline-capable once it has the compressed model — no live connection needed for normal operation. This directly solves Step 1's core problems: no per-request network latency, no per-request bandwidth cost, and full functionality even when connectivity genuinely drops.
Why this piece earns its place
Two things move onto the device with the model and only one of them is the weights. The other is everything around them — the tokenizer, the image resize, the normalization constants, the thresholds applied to the output. Reimplemented in the app's language against the platform's own text and imaging libraries, those drift from what the training pipeline did, and the result is the nastiest failure on this page: inference completes, latency looks normal, nothing is logged, and the answers are quietly a little wrong. Ship the preprocessing as part of the artifact, not as app code that happens to travel alongside it. The other thing worth saying out loud is that the per-request cost did not vanish, it moved onto the user's battery and thermal headroom. A model that benchmarks well on a cold device throttles some minutes into sustained use, so the number that matters is the one measured hot. Step 2's per-target builds also have a consequence that lands here: two devices can return different answers for the same input without either of them being broken, so a complaint cannot be reproduced by replaying it centrally. The only reproduction worth anything is on the same hardware, which is an argument for keeping a physical unit of every target you build for rather than an emulator and a hope.
Step 4 · Staged rollout — harder than a server rollback
You can't instantly reach every device
A new model version needs to roll out to potentially millions of devices. If it turns out to be bad, how quickly can you actually undo that, compared to rolling back a server-side model?
A bad model version needs to be rolled back across a fleet of edge devices. How does this compare to server-side rollback?
A server rollback works because every server instance is continuously connected and can be reached immediately. Edge devices are frequently offline, so a "rollback" is really just an updated instruction waiting for each device to next check in — which could be minutes or days later depending on that specific device's connectivity pattern.
Because devices aren't continuously connected the way servers are, "roll back" doesn't mean "instantly fixed everywhere" — it means "the corrected version is now available for any device that checks in," which can take an unpredictable amount of time per device, and the fleet is genuinely inconsistent in the meantime.
The blast radius being per-device doesn't make the ROLLBACK MECHANISM easier — reaching every device to actually deliver the correction is exactly the hard part, regardless of each device's individual blast radius.
Use a Staged Rollout Manager that gradually expands a new version's availability, monitoring for problems before wider release — but design explicitly for the reality that a rollback isn't instantaneous across the fleet: offline devices remain on whatever version they last received until they reconnect, so the fleet can be inconsistent for an unpredictable window.
Why this piece earns its place
Two things make staging different here, and neither of them is the percentages. First, cohort membership is computed on the device rather than handed out. A server could perfectly well assign one when a device checks in — but then it owns a row per device, has to keep those rows correct across reinstalls and hardware swaps, and still cannot tell you weeks later which stage a given device was actually in when you go back to analyse one. A stable hash of the device identifier gives you the same buckets with no central state, unchanged across reconnects and reconstructable from the identifier alone. Second, and worse, the population is biased in exactly the direction that hides the problem: the devices that take a new version first are the ones online, on WiFi and charging — the healthiest, best-connected slice of the fleet — so an early stage reading clean is not evidence of anything. A stage has to stay open long enough for the slow-checking-in devices to be represented, and that, not the clock, is what sets its length. Since the correction itself travels slowly, the lever you actually want is a kill switch the device evaluates locally: a flag that says fall back to the previous version is bytes against a model's megabytes, so it propagates on a different timescale. Build that channel separately, before you need it.
- server rollbacknear-instant, continuous connectivity
- edge rollbackeventually consistent, per-device reconnect
- fleet statecan be inconsistent during the window
What the new pieces do
- Staged Rollout Managerservice
- Rolls a new version out to the fleet gradually — but "rollback" here can't instantly reach a device that isn't currently online.
Step 5 · Bandwidth-conscious delivery
Don't push the full model every time
Delivering a model update to millions of devices over real-world connections (often cellular, often metered or slow). Should every update ship the full model?
Shipping a model update to millions of devices, often over cellular. Ship the full model every time?
Shipping the full model on every update, at fleet scale, means enormous aggregate bandwidth cost and a slow, expensive download for every device — especially painful over cellular connections many edge devices rely on.
Delta updates dramatically reduce the data transferred when only part of a model actually changed, and opportunistic timing (waiting for WiFi/charging rather than forcing an immediate cellular download) respects the device's real resource constraints — both meaningfully reduce cost and user impact at fleet scale.
Requiring a wired connection would exclude the vast majority of genuinely mobile edge devices from ever receiving updates — this doesn't match how most real edge devices actually connect.
Use a Delta Updater: ship only the changed portions of the model where possible, and prefer opportunistic delivery — WiFi, charging — over immediately forcing a large download over a device's metered cellular connection and battery.
Why this piece earns its place
What makes a delta small is settled before the diff runs, and in two places rather than one. Training is the obvious one: fine-tune from the previous checkpoint and the weights barely move; retrain from scratch with a new seed and the delta is a whole model wearing a smaller name. The other is the layer Step 2 added. Everything shipped here is quantized, so a weight that moved barely at all can still cross a bucket boundary — a light fine-tune can change a large share of the packed values while the model itself hardly moved, and container compression then scatters those changed bytes through the file. So diff the unpacked artifact, keeping tensor layout stable between builds, rather than the container you ship. Then there is the base-version problem: a patch is written against one specific predecessor, so a device several versions behind either applies a chain in order or falls back to a full download, and how many predecessors you keep patches for is a storage-against-bandwidth call you make once and live with for years. Finally, apply patches atomically and verify the reconstructed artifact against a hash before switching to it — a half-applied patch does not fail, it produces a file that loads and infers nonsense. Which means the device holds both models at once, and that doubled footprint belongs in the budget from Step 2, where nobody ever counts it.
What the new pieces do
- Delta Updaterservice
- Ships only the changed weights, not the full model, and prefers WiFi/charging windows over expensive, battery-draining cellular pushes.
Step 6 · Telemetry without full device control
You can't SSH into a customer's phone
You need to know how the fleet is actually performing — error rates, latency, resource usage — but can't directly access or continuously monitor devices you don't fully control.
Collect lightweight, aggregate, privacy-conscious telemetry reported opportunistically when connectivity allows — performance and error signals, not real-time streaming or invasive device access. This respects both the device's constrained resources (telemetry shouldn't itself be a meaningful battery/bandwidth cost) and user privacy, while still giving the fleet operator real visibility into how deployed models are actually performing.
Why this piece earns its place
The dangerous property of opportunistic telemetry is that it only arrives from devices healthy enough to send it. A device the last update pushed into a crash loop, or one whose model no longer loads, reports nothing at all — so the fleet's worst outcomes are precisely the ones missing from the data, and the dashboard gets quieter as things get worse. The defence is to alert on absence rather than on error rate, which means holding a denominator the devices do not supply: expected check-ins, activations, the known size of each rollout cohort. Without one, a rate computed over whoever reported is a rate over survivors. The second property is that every event is buffered and uploaded late, stamped with a clock you do not control. Events arrive out of order, from the future, and occasionally from 1970, and a given day's numbers keep revising for as long as the slowest devices take to check in. And the schema is itself a release. A field nobody thought to collect cannot be backfilled, and adding one means waiting out the same rollout a model waits out — so the question you urgently need answered during an incident is answerable only if you were already collecting it. Emit a small, stable set of coarse counters from day one, stamped with the build that emitted them, rather than a rich schema you expect to keep iterating on.
What the new pieces do
- Telemetry Aggregatordata
- Lightweight, privacy-conscious performance and error signals reported when connectivity allows — not real-time streaming from devices you don't control.
Step 7 · Graceful degradation for weaker devices
Not every device can run the latest model
The fleet includes a wide range of device capability — some can run the newest, best model version; older or weaker devices genuinely can't.
Maintain a fallback path for resource-constrained or older devices: a smaller, older, or simpler model variant (or heuristic) for devices that can't run the latest version, rather than either forcing an incompatible deployment or leaving weaker devices with no working inference at all.
Why this piece earns its place
The hard part of a tiered fleet is not maintaining the variants, it is deciding which tier a device belongs to — and you have to do that for hardware that did not exist when you shipped. An allowlist of device names means every new phone launch becomes a release you are forced to make, and anything you have not heard of gets nothing at all. So tier by measured capability instead: available memory, whether an accelerator is present, a short benchmark run once at install, with a conservative default for anything that fails to classify. Two consequences. Fleet-wide quality numbers stop meaning much, because they become a weighted average over models that behave differently, and a real regression in the smallest tier hides comfortably inside an improving total — so every metric here needs a tier dimension from the start rather than bolted on during an incident. And each tier is a standing obligation: a training lane, a compression target, a slot in every rollout, for as long as those devices exist. Add one when the population justifies it, not because the spectrum looks incomplete.
Step 8 · The sharp edges
Devices that verify, and versions that eventually sunset
Two real long-term risks: a device that stays on a bad version far longer than expected because it happened to be offline through the whole rollback window, and version fragmentation as the fleet accumulates many old model versions over years.
Rather than relying purely on a server push reaching every device, have each device self-check its current version against the Version Manifest — the known-good current version — whenever it reconnects, actively pulling a correction if it's out of date rather than assuming a push already arrived while it was offline. For version fragmentation, track how many devices remain on each old version over time and deliberately sunset very old versions on a defined timeline, rather than supporting an ever-growing number of historical versions indefinitely.
Why this piece earns its place
Making the manifest authoritative makes it the most load-bearing thing you serve, so design it that way. It is tiny and read by everything, which means a CDN in front of it and jitter on each device before it checks — when a region's connectivity comes back, every device there reconnects inside the same few minutes and asks the same question at once. Sign it, and verify the signature on the device before acting on it. The manifest names a version rather than carrying one, so forging it does not by itself run an attacker's code — it pins the fleet to an old or known-bad build, a fleet-wide outage delivered down your own healing path. It becomes code execution only if artifact fetch is unauthenticated too, so sign the artifacts as well as the file naming them. The sharper trap is inside self-healing itself: a device that heals toward a version it cannot load will heal, crash and heal again indefinitely, and it is offline-shaped enough that you may never see it happen. So the device keeps the last version that actually ran, counts consecutive load failures against a candidate, and pins itself back rather than obeying the manifest a third time. Sunsetting has a matching floor — keep one full artifact per tier alive, not just the tail of a patch chain, or a device dark for two years has nothing legal to start from.
What the new pieces do
- Version Manifestdata
- The known-good current version — devices check against this themselves on reconnect, rather than trusting that a push reached them.
You did it
You just designed an edge inference fleet.
Everything you assembled, in order
- Cloud-only inference fails at fleet scale — latency, cost, and total dependency on connectivity edge devices often don't reliably have.
- Model compression (quantization, distillation, pruning) is a deployment requirement for fitting severe on-device resource budgets, not an optional optimization.
- On-device, offline-capable inference removes per-request latency and bandwidth cost, and keeps devices functional without a live connection.
- Fleet-wide rollback is eventually consistent, bounded by each device's own reconnect timing — fundamentally different from near-instant server rollback.
- Delta updates and opportunistic delivery timing respect real bandwidth and battery constraints at fleet scale.
- Telemetry is lightweight, aggregate, and opportunistic, respecting both device constraints and privacy, within the limits of what you actually control.
- Graceful degradation via fallback model variants serves fleet hardware heterogeneity rather than assuming one uniform target.
- Devices actively self-check their version against a known-good manifest on reconnect, rather than passively trusting a push arrived while offline.