The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Knowing when NOT to answer matters more than answering
A support agent that confidently answers every ticket, including ones where it genuinely doesn't have the right information, is worse than one that admits uncertainty and hands off — a wrong answer about someone's actual order or account erodes trust far more than a quick, honest escalation.
We'll build around that principle: grounding in real account state, confidence-aware escalation, tiered resolution by risk, seamless handoff, and measuring success by whether the issue actually STAYED resolved.
What the new pieces do
- Customerclient
- Wants their issue actually resolved, not a confidently-worded non-answer — and never wants to repeat themselves after being escalated.
Step 1 · The baseline
Answer from general knowledge alone
Simplest version: the agent answers every ticket using general product knowledge — documentation, FAQs, common patterns. What breaks when the real answer depends on something specific to THIS customer?
The agent answers from general product knowledge alone. What breaks for account-specific questions?
A large share of real support tickets ("where's my order," "why was I charged this amount," "what plan am I on") depend entirely on facts specific to THIS customer's account — general product documentation simply doesn't contain that information, correct or otherwise.
General knowledge is fundamentally the wrong source for questions like "where is MY order" or "why was I charged this" — these require actual, current, customer-specific data, and answering from general knowledge alone means either a vague non-answer or a confident guess that could easily be factually wrong.
Account-specific questions (order status, billing, plan details) are typically a LARGE share of real support volume, not a small edge case — this is a central problem to solve, not a minor one.
General product knowledge can't answer questions that depend on a specific customer's actual account or order state — attempting to anyway produces vague non-answers or confident guesses that may be factually wrong, which is worse than honestly not knowing.
Why this piece earns its place
The hard part of this baseline is not that general knowledge is thin — it is that the split between a general question and an account-specific one is itself a classification, and it fails quietly in one direction. Do you offer refunds is general. Do you offer refunds on the order I placed in March is not, and it arrives worded almost identically. An agent reading the surface form answers the general version of a specific question, and the customer gets a fluent paragraph that has nothing to do with their order. Worse, your deflection number rewards exactly that: the ticket closed, nobody escalated, the dashboard improved. Ship this baseline anyway, and run it for a couple of weeks before building anything else. It gives you the real distribution of what people actually ask, which is the only honest input into deciding which system to integrate against first — orders before billing, or the reverse, is a question your ticket log answers and your intuition does not. You would skip the baseline only for an internal tool where nothing is per-customer.
What the new pieces do
- Support APIgateway
- Accepts every incoming ticket and orchestrates triage, grounding, resolution, and escalation.
Step 2 · Ground in real account data
Retrieve the actual facts, not just documentation
Fix Step 1: connect the agent to the customer's real account and order state.
The agent needs to answer account-specific questions. What should it retrieve to do so accurately?
A general summary of COMMON issues still doesn't contain THIS customer's specific facts — it might improve the vagueness of a guess but doesn't solve the underlying problem of not having the real, current data.
Retrieving the real, live account/order data means the agent's answer is actually correct for THIS customer, right now — not a plausible-sounding guess based on what's typical, but the genuine current state of their account.
Account and order state changes constantly (new orders, plan changes, billing events) — a one-time cached snapshot from signup would be stale for almost every real support scenario.
Ground responses in the customer's actual, current Account/Order State, retrieved live from the real systems of record — so answers reflect genuine, verifiable facts about this specific customer, not general patterns or stale snapshots.
Why this piece earns its place
Grounding turns the agent into a client of billing, orders, shipping and identity all at once, and that fan-out is where the real work sits. Each of those systems has its own auth, its own rate limit and its own bad day, so the retrieval layer needs a per-system timeout and a partial-result contract — I have orders, I do not have billing, here is what I can safely say. The distinction that bites is between an empty result and a failed read. No rows matched, and the query never ran, arrive at the agent looking almost identical, and collapsing them produces the worst sentence in support: we have no record of that order, said to someone whose order exists. Absence and failure have to leave the retrieval layer as different things, and only one of them is ever allowed to reach a customer. Two more consequences. A read and the action taken on it are separate moments, so anything consequential re-reads at the point of execution rather than trusting the number the turn opened with. And whatever you retrieve now lives in your logs and in every request you send a model provider, so what gets redacted before it leaves is decided here, not after the first audit.
What the new pieces do
- Account/Order Statedata
- The actual, current facts about THIS customer's account and orders — what most real support answers genuinely depend on, not general product knowledge.
Step 3 · Recognize uncertainty
Confidence has to be a first-class signal
Even with real account data, some tickets are genuinely ambiguous, require judgment, or touch high-stakes situations (a billing dispute, an angry customer, a legal threat). Should the agent always attempt a full resolution?
A ticket is ambiguous or high-stakes. Should the agent always attempt to fully resolve it itself?
Attempting resolution on genuinely ambiguous or high-stakes tickets risks producing a wrong or inappropriate response BEFORE a human ever sees it — for the categories where confidence should be low, guessing first and correcting later is the wrong order of operations.
Recognizing low confidence and escalating BEFORE attempting a risky guess avoids ever producing (and potentially acting on) a wrong response for exactly the tickets where being wrong matters most — confidence has to gate action, not just be assessed after the fact.
Relying on customers to know to ask for escalation misses cases where the CUSTOMER doesn't realize their issue is actually ambiguous or high-stakes from the agent's perspective — the agent's own confidence assessment needs to drive this, not customer awareness.
Build Triage + Confidence assessment as a first-class step: the agent estimates how confident it genuinely is, and low-confidence, ambiguous, or high-stakes tickets escalate before any resolution attempt — rather than guessing first and hoping it's right.
Why this piece earns its place
Confidence is only useful if it is calibrated, and a model asked how sure it is will not give you that on its own — it gives you a fluent number that barely moves when it is wrong. In practice the signal is assembled: did the account lookup return one unambiguous match or three plausible ones, how often has this category actually stayed resolved before, and only then the model's own claim. Keep the threshold in configuration, per category, not in the prompt. You will move it more than once, and a prompt edit is a deploy while a config value is a decision the people reading the queue can make for themselves. The consequence nobody plans for is that this threshold is a staffing dial. Every notch you lower it sends more tickets to humans, so a model upgrade that shifts calibration arrives as a queue backing up on a Tuesday rather than as a quality alert — which is why a threshold change and a model change should never ship together. And log the score beside what eventually happened to the ticket, or you are setting the most consequential number in the system by feel.
What the new pieces do
- Triage + Confidenceservice
- Classifies the ticket and — critically — estimates how confident it should be, recognizing uncertainty rather than guessing on ambiguous or high-stakes issues.
Step 4 · Tiered resolution by category and risk
Not every ticket needs the same treatment
Some tickets are simple and low-risk (a password reset, an order-status lookup); others are complex or consequential (a refund dispute, an account closure). Should every ticket go through the same resolution path?
Ticket complexity and risk vary enormously. Should every ticket follow the same resolution path?
Treating a simple password-reset request the same as a high-stakes billing dispute either over-automates the risky case or over-escalates the trivial one — neither serves customers or support efficiency well.
Tiering matches the level of automation to the actual risk and complexity of the category — trivial, well-understood issues get fast full automation, moderately complex ones get an assisted draft a human reviews, and genuinely complex or high-stakes issues go straight to a human, unautomated.
A blanket assisted-review tier for even the most trivial, well-understood cases (a password reset) adds unnecessary human review overhead for something that could be safely and fully automated — tiering by actual risk avoids this waste.
Tier resolution by category and risk: fully auto-resolve well-understood, low-risk tickets; partially automate (draft for human review) moderately complex ones; fully escalate genuinely ambiguous or high-stakes tickets — matching the level of automation to actual risk rather than one uniform process for everything.
Why this piece earns its place
The tier table is the part of this design that support owns, not engineering. It is a list of categories and what the system may do with each, it changes as the business changes, and if it exists only in code then every change is a ticket filed against a team that does not take the calls. Give it an audit trail too — who moved refunds to assisted, and when — because that is the first question asked after anything goes wrong. The discipline worth keeping is that nothing starts in the auto tier. A category enters as assisted and graduates only when its measured results earn it, which turns an unwinnable argument about what feels safe into a promotion you can point at. The demotion path matters just as much: a category whose results degrade has to be able to fall back by itself, or your only lever mid-incident is a deploy. And the assisted tier quietly pays for itself twice — every time a human edits a draft before sending it, the difference between what was written and what went out is the most direct signal you will ever get about where the agent is wrong. Capture that diff, not just the sent text.
What the new pieces do
- Auto-Resolverservice
- Fully resolves well-understood, low-risk categories — but never a consequential action without actual identity/authorization verification.
Step 5 · Seamless handoff
The customer should never have to repeat themselves
A ticket escalates to a human agent. Should the human start from scratch, or receive what the AI agent already gathered?
A ticket escalates to a human. Should the human start the conversation over from scratch?
Starting over means the customer has to repeat everything they already explained — one of the most common and frustrating pain points in real support experiences, and one this design has every ability to avoid.
A complete context transfer lets the human agent pick up exactly where the AI left off — reading what's already been established rather than making the customer re-explain their issue from the beginning, which is exactly the frustration escalation should avoid causing.
Requiring the human to manually reconstruct context that was already captured in the conversation wastes time and still risks the customer needing to re-explain details that were already discussed with the AI agent.
Transfer the full conversation and any partial diagnosis via Handoff Context to the receiving human agent — so escalation feels like a continuation, not a restart, and the customer is never asked to repeat what they've already explained.
Why this piece earns its place
The failure here is not losing the context, it is dumping it. A human picks this up with a queue behind them and about a glance to spend, so a raw transcript is technically a complete transfer and practically the same as none. What they need first is a short lead: what the customer is asking for, what has actually been verified, and — the part that gets left out — what the agent was unsure about. A partial diagnosis handed over without its uncertainty is worse than no diagnosis, because a human who trusts it inherits the mistake and stops checking. Two operational edges. Ownership has to be explicit at the moment of transfer: while a handoff sits in a queue the customer keeps typing, and if nobody decided who holds the conversation you get the agent and the human replying to the same person in the same thread. And the seam has to run both ways — if the human's resolution never comes back into the system, every escalated ticket is a permanent hole in what you know about your hardest cases, which are precisely the ones worth learning from.
What the new pieces do
- Human Agentclient
- Handles ambiguous, high-stakes, or low-confidence tickets — and needs the full prior context to avoid making the customer repeat themselves.
- Handoff Contextdata
- The complete conversation and any partial diagnosis, transferred whole to whoever picks up an escalated ticket.
Step 6 · Measure success by outcome, not activity
"A response was sent" isn't the goal
How should resolution quality actually be measured — by whether the agent sent a reply, or by something else?
Track outcomes via the Outcome Tracker: whether the customer actually stopped contacting support about the SAME issue within a reasonable window, not just whether a response was sent. A ticket "closed" by the agent that the customer reopens or re-contacts about days later wasn't actually resolved — it just looked resolved in the moment.
Why this piece earns its place
This metric is lagging by construction: you only know a resolution held once the window has passed, so it can never gate a live request. That means running two numbers at once — a fast proxy on the request path, something like how often the agent declines or escalates, and the slow outcome number whose only job is to tell you whether that proxy is lying. Watch the denominator. Measure only the tickets the agent chose to close and the figure improves every time it escalates more, which is the most comfortable way to build a number that rises while the system gets less useful; count everything that entered the category instead. The honest limitation is that silence is ambiguous — a satisfied customer and one who gave up and left look identical from here, so this needs something beside it that can tell them apart. And treat the window length as versioned configuration, not a tuning knob: lengthen it and every historical figure silently re-scores itself, leaving you a trend whose shape belongs to the change rather than to anything the system did.
What the new pieces do
- Outcome Trackerdata
- Tracks whether a customer actually stopped contacting support about the same issue — the real measure of resolution quality, not just "a reply was sent."
Step 7 · Tone and de-escalation
A frustrated customer needs a different response, not just a correct one
A technically correct response delivered in a flat, generic tone can make an already-frustrated customer feel worse, not better.
Detect frustration or anger signals in the conversation and adjust accordingly — a more empathetic tone, and, for genuinely escalated emotional situations, prioritizing faster handoff to a human agent even for issues that might otherwise be auto-resolvable. Being technically correct isn't sufficient when the delivery itself affects whether the interaction actually resolves the customer's underlying frustration.
Why this piece earns its place
Tone is worth building only if it changes behavior rather than vocabulary. Rewriting the same reply with warmer adjectives is cosmetic; what actually helps a frustrated customer is the agent asking fewer diagnostic questions, not asking for information it could look up itself, and getting to the point faster. Order matters: adjustment belongs after the confidence gate, never instead of it. A warm, apologetic wrong answer reads as insincere in a way a flat wrong answer does not, so empathy applied to an ungrounded response makes the interaction worse than the plain version would have been. Score the conversation, not the message. Frustration is a trajectory — someone patient three turns ago now writing in short sentences is the case you want to catch, and a classifier reading only the last line will miss it while over-firing on people who are simply blunt. That last part is why this needs checking per language and per channel before you trust it: directness that reads as anger in one locale is ordinary phrasing in another, and a false positive here spends a human agent on someone who was never upset.
Step 8 · The sharp edges
Social engineering and gaming attempts
Two adversarial patterns: a customer confidently claiming authority they may not actually have to push through an unauthorized consequential action, and a customer repeatedly rephrasing the same request hoping to get a more favorable automated answer by chance.
Require actual identity/authorization verification before any consequential action (a refund, an account change) — a confident claim asserted in a chat message is never treated as proof of authorization, no matter how it's phrased. For gaming attempts, detect repeated near-identical requests and route them to human review rather than letting persistence alone eventually produce a favorable automated outcome by chance.
Why this piece earns its place
Verification has to be enforced where the action happens, not where the text is written. A rule living in the prompt is a suggestion made to a system whose input is controlled by the person you are defending against; the refund endpoint itself should refuse any call that does not carry proof verification actually ran. Then a conversation that talks its way past every instruction still cannot move money, because the thing holding the line was never reading the conversation. Everything consequential also needs an idempotency key and an audit record naming what was verified and by which method. After an incident the question is what the agent did and on whose authority, and a transcript does not answer that — it shows what was said, not what was checked. Scope the repetition signal carefully too: counting near-identical requests per session is beaten by opening a new session, which is the first thing anyone tries, so it has to be counted against the customer. And this is the step you would skip outright for a read-only agent — verification, audit, idempotency and rate limits all arrive with the first tool that writes.
You did it
You just designed a support resolution agent.
Everything you assembled, in order
- General product knowledge can't answer account-specific questions — a confident guess is worse than an honest escalation.
- Grounding in real, live account/order state is what makes answers actually correct for a specific customer, not just plausible-sounding.
- Confidence assessment has to gate action BEFORE a resolution attempt, not just be reported after the fact.
- Resolution is tiered by category and risk — full automation, assisted drafting, or full escalation — rather than one uniform process.
- Escalation transfers full context so customers never have to repeat what they've already explained.
- Resolution quality is measured by whether the issue actually stayed resolved, not just whether a response was sent.
- Tone-aware de-escalation matters — correctness alone doesn't address an already-frustrated customer's experience.
- Consequential actions require actual identity verification, never just a confidently-asserted claim, and repeated gaming attempts are their own signal for review.