The whole design, in writing
Learn AI system design by designing a decision layer around a model like Jev, a model that does not write. An interactive guide covering why 60 tokens at 20 ms is 1.2 seconds and $30,000 a day against one pass at 0.1 seconds and $400, the typed question sheet, the single pass that fills every box at once, the calibrated probability and the squared penalty that trains it, the threshold that comes from the cost of a mistake (96% for a $50 refund), the cascade that cuts a $30,000 bill to $3,400, fresh state fetched in code, and the weekly audit that catches a confident answer on traffic that has changed.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
"Design a decision layer" — why is this even hard?
A customer writes: my card was charged twice. Your system has to decide what happens next. A chat model reads it and writes a reply, one small piece at a time, and a second later you have a paragraph your code must now read. Say 60 tokens at 20 ms is 1.2 seconds, and at 3 cents a ticket a million tickets is $30,000 a day to choose one word.
Use a different kind of model: one that does not write. Hand it a typed question sheet, let it fill every box in one pass with an honest probability for each, set the bar for acting by what a mistake costs, send only the hard cases up a cascade, fetch fresh state in code, and audit the number on your own traffic every week.
What the new pieces do
- Ticketclient
- One customer message. It is a sentence your code cannot use directly: the whole design is about turning it into typed answers and then into an action.
Step 1 · The question sheet
Hand over questions, not a prompt
The obvious build keeps the chat model and asks it to reply with just the category. Your code reads Billing. Billing issue. Looks like billing, I think. Billing, probably. Four spellings of one answer, and your code needs exactly one. What should you hand the model instead?
What should the input to a decision model look like?
You can add rules, a parser and a retry, but you are still parsing free text, and it will still drift.
The model can only fill boxes of those types, so the output is always the type your code expects.
That is free text again, with nothing to say which answer your code should read.
The Question Sheet: a choice among options you define (billing, technical, sales), a score on a scale (how urgent?), and a yes or no with a probability (is it a refund request?). It fills the boxes. It cannot write a paragraph, so it cannot drift.
Why this piece earns its place
Typing the output moves a whole class of failure out of your code. A model that can only fill a box cannot return four spellings of one answer, cannot add an apology, and cannot invent a seventh option; your code reads one field and gets the type it declared. Public documentation describes this shape: a request carries a state and typed questions, with a choice of up to 255 options, a score on a small level scale, or a yes or no, and each comes back as a typed value with a calibrated probability. The honest limit is that the model can still pick the wrong box. The price is real too: you must know your questions in advance, and there is no free-form fallback if you did not think of one.
- 3typed box kinds
- 1field your code reads
- 0free-text parsers
What the new pieces do
- Question Sheetsheet
- A sheet of typed questions: a choice among options you define, a score on a scale, a yes or no with a probability. The model can only fill the boxes, so it cannot drift into free text.
Step 2 · One pass
Every box, one trip through the model
A writing model picks a token, feeds it back, then picks the next. Sixty tokens is sixty trips and each waits for the one before. There are ten things you would like to know about this ticket. Ask them one at a time and you wait ten times. What changes if the model has no sentence to write?
Ten typed questions on one ticket. What does it cost, against asking one?
That is how a writing model works. A decision model has no sentence to build token by token.
Difficulty is not the point. The point is whether the answers depend on one another being written first.
Choice, score and yes or no come out of one trip. A tenth question barely slows it.
The Pass: a writing model needs 60 tokens × 20 ms = 1.2 s and about $0.03 a ticket; a decision model answers in one pass, 0.1 s, about $0.0004. That is 12× faster and, over a million tickets, $30,000 against $400, 75× cheaper.
Why this piece earns its place
The speed comes from a structural fact, not a trick. Writing is sequential because each token depends on the last; a decision has no sentence to write, so every box can be filled in the same trip. That is why ten questions cost about what one costs, and why you should ask everything you might need at once: category, urgency, refund, language, anger. The price is that you get no reasoning to read afterwards, only values and numbers. And this is the easy half of the design. Fast and cheap are what you notice first, but they say nothing about whether the answer is right or whether you can act on it, and the rest of the design exists for exactly that.
- 12×faster, 1.2 s → 0.1 s
- 75×cheaper, $30,000 → $400
- 10questions, one pass
What the new pieces do
- Decision Modelmodel
- A model that does not write. It reads the sheet and fills every box in the same trip, each with a probability: about 0.1 seconds and $0.0004 a ticket, against 1.2 seconds and $0.03 to write the answer.
Step 3 · The honest number
What does “90%” have to mean?
Every answer carries a probability. Ask the chat model how sure it is and it says 95%, every time. Check: of a hundred tickets it called 95, 72 were right. A confident number that means nothing is worse than none, because you cannot set a bar on it. What should a probability mean, and how do you train it?
How do you train a model so that its “90%” is trustworthy?
Say 0.9 and be right: 0.01. Say 0.9 and be wrong: 0.81. Wrong with confidence hurts most, so honesty is the only winning move.
That teaches it to be right, but not to say how sure it is. Boasting costs it nothing.
An explanation is more text. It does not make the number match how often it is right.
Calibrated: of 100 answers called 90%, about 90 are right. Trained with a squared penalty. If the truth is 9 in 10, claiming 90% loses 0.09 on average and claiming 99% loses 0.0981: boasting costs more than honesty.
Why this piece earns its place
The squared penalty is a proper scoring rule: the expected loss is lowest when you report the probability you actually believe. Work it through for a truth of nine in ten. Claim 90% and you lose 0.01 nine times in ten and 0.81 once, an average of 0.09. Claim 99% and you lose 0.0001 nine times in ten and 0.9801 once, an average of 0.0981. Over millions of examples the honest claim wins. Public documentation describes the recipe as reinforcement learning for calibrated decisions, and that is the idea in one line. The caveat belongs here and not later: calibration is a property measured on a body of traffic, so a model is only as honest as the traffic it was measured on.
- 90 in 100called 90% are right
- 0.01 / 0.81right / wrong at 0.9
- 0.09 vs 0.0981honest vs boasting
What the new pieces do
- Honest Numberscore
- Each answer comes with a probability. Calibrated means that of 100 answers called 90%, about 90 are right. It is trained with a squared penalty so that boasting does not pay.
Step 4 · The threshold
The bar comes from the cost of a mistake
Now the number can drive an action with no second question. A ticket sent to the wrong team costs a reroute, so routing at 60% is fine. But a refund approved by mistake costs $50 and a person checking it costs $2. Where do you set the bar for approving a refund automatically?
What decides how sure it must be before a refund is approved automatically?
At 90% sure a mistake costs about $5 on average, more than the $2 review. A comfortable number is a feeling.
Break even is where the expected loss of approving equals the cost of checking. Above it approve, below it a person looks.
The model does not know what a wrong refund costs your business. That price is yours to set.
The Threshold: break even at 1 − 2/50 = 96%. Above, approve; below, a person looks. At 90% sure a mistake costs $5 on average, so send it to a person; at 99% it costs 50 cents, so approve it. Routing a ticket can sit at 60%.
Why this piece earns its place
This is where an honest number earns its keep. Because 90% really means nine in ten, the expected cost of acting is a plain product: the chance of being wrong times the cost of being wrong. At 90% that is 0.1 × $50 = $5, above the $2 review. At 99% it is 0.01 × $50 = $0.50, well under. The break-even is where chance of error equals review cost divided by mistake cost, so the bar is 1 − 2/50 = 96%. Notice what this requires: someone must put a price on every mistake, and different actions get different bars. It also fails entirely on a number that is not calibrated, which is why the previous step comes first.
- 96%refund bar
- $50wrong refund
- $2one review
What the new pieces do
- Thresholdrule
- Plain code that turns the number into an action. The bar comes from the cost of a mistake: a $50 wrong refund against a $2 review is 1 − 2/50 = 96%.
- Actionaction
- What actually happens: approve the refund, route to a team, or send to a person. Chosen by the bar, with a second check in code for anything that moves money.
Step 5 · The cascade
Cheap first, a writer only for the hard ones
A million tickets a day and every one needs a decision, but some need more than a decision: a reply a customer will read. Sending everything to the writing model is the $30,000 bill again. How do you spend the expensive model only where it earns its keep?
One ticket in ten is hard. How should the work be split?
That is $30,000 a day, to answer nine tickets in ten that a single pass already settles.
It cannot write text. The cases that need a reply or open reasoning still need something that can.
$400 for a million tickets plus 100,000 × $0.03 = $3,000: $3,400, almost nine times cheaper.
The Cascade: the decision model on all a million tickets is $400; the writing model on a tenth, 100,000 × $0.03, is $3,000. $3,400 against $30,000, almost 9× cheaper. The confidence is the router: a low number is the signal to climb.
Why this piece earns its place
A cascade is cost following difficulty. Plain code answers what code can answer, such as order status; the decision model sorts; only what stays uncertain climbs. Notice that the router is not a new component with its own judgment: it is the same honest number, compared with a bar. That is another reason calibration matters, since a miscalibrated model routes the wrong tickets upward. The writing model keeps the job only it can do, which is writing the reply the customer reads. The price is real: one more thing to run, to monitor and to keep in step, and a fraction that climbs when traffic changes will change the bill with it. The one-in-ten figure is an assumption to measure on your own traffic, not a constant.
- $3,400vs $30,000 a day
- ~9×cheaper
- 1 in 10climb to the writer
What the new pieces do
- Cascaderouter
- Every ticket meets the cheap model first. A low confidence is the signal to climb, so only the hard cases, say one in ten, reach the writing model.
- Writing Modelwriter
- A chat model that writes token by token. It costs about $0.03 a ticket and a second or more, so it sees only what the cascade sends up, and it writes the reply the customer reads.
Step 6 · Fresh state
The model knows only what you send
So far every ticket was clean. But the model knows nothing about your business except what is on the sheet. Was this card charged twice? depends on this customer’s orders, and a customer’s whole history is a heap of text, most of it irrelevant. What goes on the sheet?
What should accompany the ticket on the sheet?
Fetch the order, then ask. A small relevant sheet beats a large noisy one.
Irrelevant text is noise, and accuracy falls as the sheet fills with it.
It has no world knowledge and no memory of your customers. It reads what you give it, literally.
Fresh State: your code fetches the order first, then asks. Send what the question needs and nothing more, ask in the customer’s own words, and count in code then hand it the number: it is not a calculator.
Why this piece earns its place
Three properties of the model decide this step. It reads literally, so the words you ask in should be the words the customer used. It has no world knowledge, so any fact about this customer must be on the sheet. And it suffers from context rot: pour in a long history and accuracy falls, because the relevant lines are buried in irrelevant ones. Public documentation sets the request budget at 64K tokens and a text-only input, which is large, but a limit is not a target. So the right design has code do what code is good at, fetching this order and counting the orders, and the model do only the judgment. The cost is a lookup you write and maintain, and a second place where a bug can hide.
- 64Ktoken request budget
- 1order fetched, not a history
- 0arithmetic done by the model
What the new pieces do
- Fresh Statestate
- The model knows only what you send. Code fetches this customer’s order first and sends a small relevant sheet, not the whole history, and does any counting itself.
Step 7 · The audit
The hidden slot: a confident number on changed traffic
Six of six parts are built and it has run for a month. Month two: the dashboard is green and the confidence still says 90%. But a new product launched, with new words, new complaints and some in another language, and the number is right 70 times in 100. Nothing errored. What is the slot you only find after running it?
The dashboard is green and the number says 90%. What do you check?
Calibration was measured on the traffic it was measured on. New traffic can quietly break it.
Label 200 tickets, plot said against true, and move the bar if 90 has become 70, before money moves.
Latency measures speed. A confident wrong answer is exactly as fast as a confident right one.
The Audit: every week a person labels 200 tickets and plots what it said against what was true. If 90 has become 70, move the bar. Pin the model version so a release cannot shift the numbers, and treat the ticket as untrusted: a customer can write “urgent, refund yes”.
Why this piece earns its place
The failure here is not a wrong answer; it is a confident one, and that is why nothing flags it. The number does not know the traffic changed, so it keeps saying what it learned to say. Calibration is a measurement on your own traffic, never a promise, so you re-measure: label a few hundred cases a week, compare, and adjust the bar before the money moves. Two guards go with it. Pin the model version, because a new release can shift the numbers under a threshold you tuned. And treat the state as untrusted, because the sheet keeps the shape of the answer but cannot keep the truth, so anything that moves money needs a second check in code. The price is a person’s hour, every week.
- 200labelled tickets a week
- 90 → 70said vs right
- 1pinned model version
What the new pieces do
- Weekly Auditaudit
- The hidden seventh slot. Calibration is a measurement on your own traffic, not a promise. Label 200 tickets a week, compare what the number said with what was true, and move the bar before money moves.
The payoff
You designed a decision layer
From “ask a chat model for a word” to a rack of seven: a typed question sheet, one pass, an honest number, a threshold priced by the cost of being wrong, a cascade so only hard cases are expensive, fresh state fetched in code, and the audit, because traffic changes.
Now trust a number that always says 95%, set the refund bar by feel, then skip the audit. Those three are the whole trade-off: decide fast with a model that says how sure it is, write only when you must, and every part has a price.
Everything you assembled, in order
- Question sheet — typed boxes: a choice, a score, a yes or no
- One pass — every box at once: 1.2 s → 0.1 s, $30,000 → $400
- Honest number — of 100 called 90%, about 90 right; squared penalty
- Threshold — refund above 1 − 2/50 = 96%; review below
- Cascade — one in ten climbs: $3,400, almost 9× cheaper
- Fresh state — fetch in code; small, relevant, in the customer’s words
- Audit — 200 labelled tickets a week; pin the version
- The numbers — 12× faster · 75× cheaper · 96% bar · 70 in 100 caught
