The big idea

Privacy decides what leaves, not just confidence

An on-device agent can run on the NPU — fast, offline, private. But the small on-device model isn't as capable as a cloud model, so for hard queries you want to escalate. The usual framing is "escalate when the local model is unsure." That's only half the story: some data — health, private messages, keys — must never leave the device, no matter how uncertain the local model is. How do you get cloud capability without breaking that boundary?

We'll build the hybrid where privacy is the routing constraint that comes first: a privacy classifier gates escalation before confidence is even considered, so must-stay-local data is answered on-device even when that means a worse answer. Then confidence decides escalation for what's allowed to leave, redaction minimizes what's sent, and an audit records every decision. The confidence-cascade math itself lives in the LLM Router design; the fully-local alternative is the Edge Inference Fleet — this is the privacy-gated middle.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end ask a sensitive question the edge model is unsure about and watch it refuse to escalate. Hit Begin.

Step 1 · The baseline

Send everything to the cloud

The simplest capable design: forward every query to a big cloud model. Answers are great. For an agent handling personal data on a personal device, what's the problem?

Design decision: An on-device agent forwards every query to a cloud model for the best answer. What's the core problem?

The call: Every query — including deeply private data the user expects to stay on the device — leaves the device, so the agent is a continuous privacy leak, and it also fails when offline and costs a cloud call every time. — Exactly: cloud-everything means all personal data leaves the device by default, which is a privacy and compliance failure for a personal agent, on top of being offline-fragile and costly per call. The user wanted a capable agent that respects a privacy boundary; this respects no boundary at all.

Cloud-everything ships all of the user's personal data off the device by default — a privacy and often regulatory breach for a personal agent — while also failing offline and costing a cloud call per query. Capability came at the price of respecting no privacy boundary at all.

Default-to-cloud is default-to-leak: For a personal-device agent, sending data off-device is the exceptional, costly act — not the default. The whole design inverts the baseline: answer on-device by default, and treat leaving the device as something that must be justified and constrained.

Step 2 · The default

Answer on-device with the NPU model

Flip the default: a small model runs on the device's NPU and answers queries locally. Why is on-device-by-default the right foundation for a private, capable agent?

Design decision: Why make the on-device NPU model the default responder instead of the cloud?

The call: A local answer is private by construction (nothing leaves the device), fast, and works offline — so the majority of queries the small model can handle incur no privacy cost, no latency, and no cloud bill. — Right: making on-device the default means most queries are answered with zero data leaving the device. Privacy, latency, offline capability, and cost all improve for the common case — and the cloud is reserved for the minority of queries that actually need it, which is where the privacy question gets interesting.

The on-device NPU model answers by default: private by construction, fast, and offline-capable. Most queries the small model can handle incur no data leaving the device, no latency, and no cloud cost — so escalation becomes the rare exception, and the privacy question narrows to just those exceptions.

Local by default, cloud by exception: Inverting the baseline is the whole move. When on-device is the default, the only queries that raise a privacy question are the ones considered for escalation — and that's exactly where the next gate lives.

Step 3 · The constraint that comes first

Gate escalation on privacy, before confidence

A query might be a candidate for escalation. Before deciding whether the local model is confident enough, there's a more important question. What is it, and why must it come first?

Design decision: Before deciding whether to escalate an uncertain query, what must the agent check first?

The call: Whether the query's data is allowed to leave the device at all — a privacy classifier that runs first, so must-stay-local data is answered on-device even if the model is unsure. — Right: privacy is the routing constraint, and it comes before confidence. The classifier decides whether this query touches data that must never leave (health, credentials, private messages). If it does, escalation is off the table regardless of the confidence score — the agent answers locally with whatever it can. A better answer is never worth violating the privacy boundary.

The Privacy Classifier runs first: it decides whether the query's data may leave the device at all. Must-stay-local data (health, credentials, private messages) is answered on-device even when the model is uncertain — a better answer is never worth breaching the boundary. Only privacy-cleared queries reach the confidence decision.

Privacy is a hard constraint, not a tiebreaker: This is what distinguishes a hybrid privacy-aware agent from a plain confidence cascade: the privacy gate is not a factor the confidence score can outvote. It comes first, and it can veto escalation outright — capability yields to the boundary, never the reverse.

Step 4 · When to escalate

Confidence decides — for what may leave

For a query privacy has cleared to leave, now the familiar question: answer locally or escalate to the cloud? What decides, and how much of that decision does this design need to build?

Design decision: For a privacy-cleared query, what decides whether to answer on-device or escalate to the cloud?

The call: The on-device model's confidence: answer locally if confident, escalate to the cloud only if uncertain — the standard escalation cascade, applied only to already-privacy-cleared queries. — Correct: confidence is the escalation signal, but it operates only on the subset privacy already allowed to leave. A confident local answer stays local; an uncertain one escalates for a better answer. The detailed cascade — how to threshold confidence, calibrate it, trade cost against quality — is the LLM Router design's territory; here the key point is that this gate runs AFTER privacy, never before.

For privacy-cleared queries, the Confidence Gate applies the standard cascade: a confident on-device answer stays local; an uncertain one escalates to the cloud for a better answer. The depth of that decision — thresholding, calibration, cost-quality trade-offs — is the LLM Router design's subject; the essential point here is that confidence runs only after privacy has already permitted the data to leave.

Confidence is downstream of privacy: The cascade is real and important, but in this design it's the second gate, not the first. Reusing the router's escalation logic on a privacy-pre-filtered stream is what makes the agent both capable and boundary-respecting.

Step 5 · Minimize what leaves

Redact before sending to the cloud

A query is cleared and the model is unsure, so it will escalate. Should the agent send the full context to the cloud? What's the right principle for the payload?

Design decision: When a query does escalate to the cloud, what should be sent?

The call: Only the minimum the cloud model needs to help — redact identifiers and unrelated private context first, so even permitted escalation leaks as little as possible. — Right: data minimization. Even for data cleared to leave, you strip it to what the cloud actually needs — remove names, IDs, and unrelated private context that don't affect the answer. The cloud gets enough to help, and no more, so the privacy cost of a necessary escalation is as small as possible.

The Redactor strips an escalating query to the minimum the cloud model needs — removing identifiers and unrelated private context — so even permitted escalation leaks as little as possible. "Allowed to leave" is not "send everything": data minimization applies to every byte that crosses the boundary.

Send the least that still works: The privacy classifier decides whether data may leave; the redactor decides how much. Minimizing the payload means a necessary escalation costs the smallest possible slice of privacy, not the whole context.

Step 6 · Verifiability

Audit every escalation decision

The agent is now making privacy-critical routing decisions continuously. How does the user (or a regulator) ever trust that the boundary is actually being respected?

Design decision: How do you make the agent's privacy boundary verifiable?

The call: Keep an on-device audit trail of every routing decision — what was sent, what was kept local, and why — so the boundary is verifiable and any suspected leak is investigable. — Right: an on-device record of every decision (escalated or kept local, and the reason) makes the privacy boundary auditable. The user can confirm sensitive queries never left, a regulator can verify compliance, and a suspected leak can be investigated — and keeping the audit on-device means the record itself doesn't become a new leak.

An on-device Escalation Audit records every routing decision — what was sent, what was kept local, and why. That makes the privacy boundary verifiable rather than merely asserted: the user can confirm sensitive queries stayed local, a regulator can check compliance, and a suspected leak is investigable — and keeping the log on-device means it doesn't become a leak of its own.

A boundary you can't verify isn't one: Privacy claims are only as good as your ability to check them. The audit trail turns "we respect your data" into "here is exactly what left and what didn't" — kept on-device so the accountability record isn't itself a privacy cost.

The payoff

Capable when it can be, private when it must be

Put it together: an agent that answers most things instantly on-device, escalates the genuinely hard queries for a better answer — but never lets must-stay-local data leave, minimizes what does, and keeps a verifiable record of every choice.

That's the hybrid edge-cloud agent. On-device by default, privacy as the first routing gate, confidence as the second, redaction on what leaves, and an on-device audit — capability and privacy reconciled by making the boundary a hard constraint the confidence score can never override.

• On-device NPU model answers by default — private, fast, offline — so escalation is the rare exception, not the norm.
• A privacy classifier is the FIRST gate: must-stay-local data is answered on-device even when the model is unsure — privacy is a hard constraint, not a tiebreaker confidence can override.
• Confidence decides escalation only for privacy-cleared queries (the cascade depth lives in the LLM Router design — reused, not re-derived here).
• A redactor minimizes what leaves — "allowed to leave" is never "send everything"; even permitted escalation sends the least necessary.
• An on-device audit records every decision (sent vs kept local, and why), so the privacy boundary is verifiable and any leak investigable.