The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Privacy decides what leaves, not just confidence
An on-device agent can run on the NPU — fast, offline, private. But the small on-device model isn't as capable as a cloud model, so for hard queries you want to escalate. The usual framing is "escalate when the local model is unsure." That's only half the story: some data — health, private messages, keys — must never leave the device, no matter how uncertain the local model is. How do you get cloud capability without breaking that boundary?
We'll build the hybrid where privacy is the routing constraint that comes first: a privacy classifier gates escalation before confidence is even considered, so must-stay-local data is answered on-device even when that means a worse answer. Then confidence decides escalation for what's allowed to leave, redaction minimizes what's sent, and an audit records every decision. The confidence-cascade math itself lives in the LLM Router design; the fully-local alternative is the Edge Inference Fleet — this is the privacy-gated middle.
What the new pieces do
- Device / Userclient
- A phone or laptop running an on-device agent. The user wants both capability (which sometimes needs the cloud) and privacy (some data must never leave the device) — and those two goals are in tension.
Step 1 · The baseline
Send everything to the cloud
The simplest capable design: forward every query to a big cloud model. Answers are great. For an agent handling personal data on a personal device, what's the problem?
An on-device agent forwards every query to a cloud model for the best answer. What's the core problem?
Best answers at the cost of shipping every piece of personal data off the device. A personal agent sees health info, private messages, credentials — sending all of it to the cloud is a privacy breach the user never agreed to, and often a regulatory one, regardless of answer quality.
Exactly: cloud-everything means all personal data leaves the device by default, which is a privacy and compliance failure for a personal agent, on top of being offline-fragile and costly per call. The user wanted a capable agent that respects a privacy boundary; this respects no boundary at all.
Latency is secondary. The core problem is that a personal agent sends all of a user's private data off-device by default — a privacy breach independent of how fast the cloud responds.
Cloud-everything ships all of the user's personal data off the device by default — a privacy and often regulatory breach for a personal agent — while also failing offline and costing a cloud call per query. Capability came at the price of respecting no privacy boundary at all.
Why this piece earns its place
The reason to sit with a design you are about to throw away is that it is the one that actually ships first, and walking it back later is not a refactor. Data that has left cannot be recalled. It sits in the provider's request logs and outside any delete path you control — so a privacy fix only ever ships forward, and every query the old build sent is still gone. The retrofit is harder than it looks for a second reason: cloud-everything has no notion of sensitivity in its data model, because nothing ever needed one. Adding the gate later means going back and labelling categories of data the app never tracked, on a device where that context has usually already been flattened into a prompt string. The honest case for keeping the baseline is narrower than it sounds. "Those documents already sit in that vendor's cloud" is not on its own a reason to forward the query: storage and inference are separate endpoints with their own retention and their own subprocessors, often from different vendors, so the query path can cross a boundary the document store never did. What to check is whether the inference endpoint sits inside the same agreement as the store; where it genuinely does, a work-account agent over those documents gains little from the hybrid. On a personal device there is no such agreement to point at.
What the new pieces do
- Cloud Modelcontrol
- A large model in the cloud — more capable than the on-device model, but every query sent to it leaves the device. Escalating to it is a capability gain and a privacy cost.
Step 2 · The default
Answer on-device with the NPU model
Flip the default: a small model runs on the device's NPU and answers queries locally. Why is on-device-by-default the right foundation for a private, capable agent?
Why make the on-device NPU model the default responder instead of the cloud?
They're usually less capable, not more — that's exactly why escalation exists. The reason to default to on-device isn't accuracy; it's that a local answer is private, fast, and offline-capable by construction.
For the hardest queries the local model genuinely isn't enough, which is why this is a hybrid, not a fully-local design. On-device-by-default reduces cloud use dramatically but doesn't remove the need for escalation.
Right: making on-device the default means most queries are answered with zero data leaving the device. Privacy, latency, offline capability, and cost all improve for the common case — and the cloud is reserved for the minority of queries that actually need it, which is where the privacy question gets interesting.
The on-device NPU model answers by default: private by construction, fast, and offline-capable. Most queries the small model can handle incur no data leaving the device, no latency, and no cloud cost — so escalation becomes the rare exception, and the privacy question narrows to just those exceptions.
Why this piece earns its place
Making the device the default responder quietly costs you your view of the system. Most traffic now leaves no server-side trace at all — that is the point — so the common case is invisible in production telemetry by construction, and every question you will want answered later has to be designed for on-device measurement up front: how often the local model is wrong, which query classes it handles badly, whether an update regressed anything. Retrofitting that means asking for data you just promised not to collect. The second consequence is that the on-device model is a shipped artifact, not a deployment. It lives on a fleet you cannot update atomically, across hardware tiers with different NPUs, so more than one model version is always answering and a behaviour change arrives as a migration rather than at a flag flip. That makes the escalation rate a property of the fleet's version mix rather than a constant — and the escalation rate is also your cloud bill and your egress volume, both forecast from a distribution you do not control.
What the new pieces do
- On-Device Modelgateway
- A small model running on the device's NPU. It answers most queries entirely locally — fast, offline-capable, and private by construction because nothing leaves the device.
Step 3 · The constraint that comes first
Gate escalation on privacy, before confidence
A query might be a candidate for escalation. Before deciding whether the local model is confident enough, there's a more important question. What is it, and why must it come first?
Before deciding whether to escalate an uncertain query, what must the agent check first?
Availability matters for the mechanics of escalation, but it's not the gating question. The thing that must be checked before anything else is whether this query's data is even ALLOWED to leave the device — a constraint that availability can't override.
Right: privacy is the routing constraint, and it comes before confidence. The classifier decides whether this query touches data that must never leave (health, credentials, private messages). If it does, escalation is off the table regardless of the confidence score — the agent answers locally with whatever it can. A better answer is never worth violating the privacy boundary.
This is the trap. Confidence-first means an uncertain model escalates sensitive data to get a better answer — exactly the privacy violation the design exists to prevent. Confidence can only decide escalation for data that privacy has ALREADY cleared to leave.
The Privacy Classifier runs first: it decides whether the query's data may leave the device at all. Must-stay-local data (health, credentials, private messages) is answered on-device even when the model is uncertain — a better answer is never worth breaching the boundary. Only privacy-cleared queries reach the confidence decision.
Why this piece earns its place
Two operational details decide whether this ordering survives contact with production. The first is what the gate does when it is the thing that breaks — the classifier times out, crashes, or its policy fails to load on a cold start. The only safe behaviour is to answer locally and refuse escalation: a gate that fails open fails directly into the leak it exists to stop, and that default gets chosen quietly, by whoever is fixing a latency regression at three in the morning. The second is that the classifier sits on the critical path of every escalation candidate, ahead of any network call. If it is heavy enough to feel slow, someone will eventually propose running it after the confidence gate to save the work on queries that were never going to escalate — and that reordering is precisely the bug this step exists to prevent. Keep it cheap enough that the argument never starts. One scoping rule sits underneath both: what gets classified is the payload the request would actually carry, not the sentence the user typed. By the time a query reaches this gate the agent has usually gathered local context of its own — a calendar entry, the file that was open, a message thread — and classifying the typed text alone clears a harmless-looking question whose assembled context is nothing of the kind.
What the new pieces do
- Privacy Classifierservice
- The FIRST gate: classifies whether the query and its data are allowed to leave the device at all. Data marked must-stay-local is never escalated, regardless of how the confidence gate feels about it.
Step 4 · When to escalate
Confidence decides — for what may leave
For a query privacy has cleared to leave, now the familiar question: answer locally or escalate to the cloud? What decides, and how much of that decision does this design need to build?
For a privacy-cleared query, what decides whether to answer on-device or escalate to the cloud?
Correct: confidence is the escalation signal, but it operates only on the subset privacy already allowed to leave. A confident local answer stays local; an uncertain one escalates for a better answer. The detailed cascade — how to threshold confidence, calibrate it, trade cost against quality — is the LLM Router design's territory; here the key point is that this gate runs AFTER privacy, never before.
Escalating every cleared query throws away the whole benefit of the on-device default — cost, latency, and unnecessary data movement all spike. Escalation should be reserved for queries the local model can't handle well, which is what confidence measures.
Load-balancing is an operational concern, not the routing decision. What determines escalation is whether the local model can answer well (confidence), applied only to queries privacy has cleared.
For privacy-cleared queries, the Confidence Gate applies the standard cascade: a confident on-device answer stays local; an uncertain one escalates to the cloud for a better answer. The depth of that decision — thresholding, calibration, cost-quality trade-offs — is the LLM Router design's subject; the essential point here is that confidence runs only after privacy has already permitted the data to leave.
Why this piece earns its place
Two things about this gate are specific to running it here rather than inside a pure router. First, the threshold is not only a cost dial. Lowering it to get better answers raises the volume of data crossing the boundary, so a quality-tuning session is silently a privacy change and deserves the same review as one — which argues for the threshold living in device configuration rather than being set per request. The one exception to that — to the threshold being configuration rather than a request field — is an explicit user action: someone who taps "ask the cloud" has consented to that single escalation, which is a different event from the agent deciding by itself, and the audit has to be able to tell the two apart. It is not an exception to the ordering. That tap still arrives behind the privacy classifier, and a must-stay-local query is refused there exactly as it would be if the agent had asked. Second, this gate only ever sees the privacy-cleared subset, so everything you know about its calibration comes from a censored sample. You cannot measure how it would have behaved on the queries privacy refused, because they never reached it — and they are not a random slice, they are often the hardest ones. Say that out loud rather than reporting the number as if it covered everything.
What the new pieces do
- Confidence Gateservice
- Only for privacy-cleared queries: if the on-device model is confident, answer locally; if it's uncertain, consider escalating. This is the when-to-escalate decision — see LLM Router for the cascade math in depth.
Step 5 · Minimize what leaves
Redact before sending to the cloud
A query is cleared and the model is unsure, so it will escalate. Should the agent send the full context to the cloud? What's the right principle for the payload?
When a query does escalate to the cloud, what should be sent?
Sending everything maximizes both answer quality and leakage — including identifiers and unrelated private context the cloud doesn't need. "Allowed to leave" isn't "send it all"; data minimization still applies to permitted escalation.
Right: data minimization. Even for data cleared to leave, you strip it to what the cloud actually needs — remove names, IDs, and unrelated private context that don't affect the answer. The cloud gets enough to help, and no more, so the privacy cost of a necessary escalation is as small as possible.
A hash the cloud can't read is a context the cloud can't use to answer — that defeats the point of escalating. The goal isn't to hide the query from the model that must process it; it's to send the minimum necessary, redacting what the model doesn't need.
The Redactor strips an escalating query to the minimum the cloud model needs — removing identifiers and unrelated private context — so even permitted escalation leaks as little as possible. "Allowed to leave" is not "send everything": data minimization applies to every byte that crosses the boundary.
Why this piece earns its place
Where the redactor runs is most of the point: on the device, before the request is built. A scrubber in a cloud-side proxy has already received the data it is scrubbing, which puts the boundary on the wrong side of the network. Once it is local, two things are harder than they look. Redaction has to be judged across a session rather than per field — stripping the name out of a query whose remaining details still describe exactly one person is a pattern that matched and a minimization that failed, and the cloud sees the sequence of queries, not one message in isolation. And redaction is lossy, so its cost lands on answer quality: an over-aggressive pass returns advice that does not fit the user's situation, and what they experience is a bad agent, not a privacy win. Telling those two apart needs the before and the after side by side, and the only place that can happen is on the device, in the moment, while both still exist — what outlives it is a measurement, not the pair. So the redactor scores its own work as it runs and emits that: which categories it stripped, how much of the query survived. That verdict is the only evidence of this component you will ever hold, because the material that would let you re-judge a call later is the material you just promised nobody keeps.
What the new pieces do
- Redactorservice
- For queries that will escalate, strips the payload to the minimum the cloud needs — removing identifiers and unrelated private context — so even permitted escalation leaks as little as possible.
Step 6 · Verifiability
Audit every escalation decision
The agent is now making privacy-critical routing decisions continuously. How does the user (or a regulator) ever trust that the boundary is actually being respected?
How do you make the agent's privacy boundary verifiable?
"Trust the code" is not something a user or regulator can verify, and it gives you nothing to investigate if a leak is suspected. A privacy-critical system needs an auditable record of what it actually did, not just an assurance that it should behave.
Escalations matter, but so do the decisions to keep data local — the record has to show what was kept on-device and why, so you can verify the boundary held for sensitive queries, not just observe what left.
Right: an on-device record of every decision (escalated or kept local, and the reason) makes the privacy boundary auditable. The user can confirm sensitive queries never left, a regulator can verify compliance, and a suspected leak can be investigated — and keeping the audit on-device means the record itself doesn't become a new leak.
An on-device Escalation Audit records every routing decision — what was sent, what was kept local, and why. That makes the privacy boundary verifiable rather than merely asserted: the user can confirm sensitive queries stayed local, a regulator can check compliance, and a suspected leak is investigable — and keeping the log on-device means it doesn't become a leak of its own.
Why this piece earns its place
Build the record out of decisions, not payloads — the classification, the reason, the policy version that produced it, a digest of what was sent rather than its text — and keep it in the same encrypted store as the data it describes. The policy version earns its place: the rules will tighten, and a call made under last month's rules only reads as correct if the record says which rules were live when it was made. The log also grows forever on hardware with no spare room, so it needs a cap and a rotation rule, and the moment you choose that number you have decided how far back a dispute can be investigated: a product decision wearing storage clothing. The consequence people do not plan for arrives when you actually show the log to a user. An audit nobody reads is theatre, but a readable one constrains the classifier upstream — every decision now has to be explainable in one printable sentence, which quietly rules out any reason you cannot put into words.
What the new pieces do
- Escalation Auditdata
- An on-device record of every escalation decision — what was sent, what was kept local, and why — so the privacy boundary is verifiable and any leak is investigable.
The payoff
Capable when it can be, private when it must be
Put it together: an agent that answers most things instantly on-device, escalates the genuinely hard queries for a better answer — but never lets must-stay-local data leave, minimizes what does, and keeps a verifiable record of every choice.
That's the hybrid edge-cloud agent. On-device by default, privacy as the first routing gate, confidence as the second, redaction on what leaves, and an on-device audit — capability and privacy reconciled by making the boundary a hard constraint the confidence score can never override.
Everything you assembled, in order
- On-device NPU model answers by default — private, fast, offline — so escalation is the rare exception, not the norm.
- A privacy classifier is the FIRST gate: must-stay-local data is answered on-device even when the model is unsure — privacy is a hard constraint, not a tiebreaker confidence can override.
- Confidence decides escalation only for privacy-cleared queries (the cascade depth lives in the LLM Router design — reused, not re-derived here).
- A redactor minimizes what leaves — "allowed to leave" is never "send everything"; even permitted escalation sends the least necessary.
- An on-device audit records every decision (sent vs kept local, and why), so the privacy boundary is verifiable and any leak investigable.