Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
A demo isn't
a product.
Anyone can get an impressive LLM demo in an afternoon — the model does the task once, on a hand-picked input, and everyone's amazed. The distance from that to a thing you can charge for at scale is enormous, and it's where most AI features quietly die. Because a product has to be right on the messy inputs, affordable at a million requests, and reliable under load — none of which a demo tests. AI product engineering is the discipline of crossing that gap, and it rests on two numbers a demo never shows you: your unit economics and your eval score. This handbook makes both runnable.
The last mile
AI features have a famous shape: the first mile is trivial and the last mile is brutal. Getting to a demo — the model summarizing a document, answering a question, writing some code — is fast and thrilling. But a demo is a single curated success, while a product is a distribution of real inputs at scale. The demo hides the failure rate on the ugly 20% of inputs, the bill when a million users arrive, the latency under load, and the day the model returns malformed JSON or gets prompt-injected.
AI product engineering is the work of crossing that gap, and it's a genuinely different job from prompting. It layers three concerns a playground ignores: unit economics (every request costs real money in tokens, so you must know your cost and margin), quality measurement (LLMs are non-deterministic, so "it looks good" must become a number via evals), and reliability engineering (retries, fallbacks, caching, guardrails, latency budgets around a model that will sometimes fail). A demo proves the model can do the task; product engineering makes it do the task correctly, affordably, and dependably a million times. The two numbers that gate that transition — cost and eval score — are the spine of this handbook.
Shipping an LLM product is unit economics plus a quality gate: know your cost-per-request and margin, measure quality with evals, and ship only when quality and margin both clear a bar — because a demo tests none of that.
Unit economics
The first number a demo hides is cost. LLM APIs charge by the token — a price per million input tokens and a (usually higher) price per million output tokens — so every request has a real, computable cost: (in_tokens × in_price + out_tokens × out_price), per million. At your price to the customer, that cost determines your gross margin — the fraction of revenue left after the model bill — and whether the feature makes money at all.
This turns a lot of "engineering" choices into economic ones. A long system prompt, a big retrieved context, a chatty output, or a multi-step agent loop that calls the model ten times all multiply the token count, and therefore the cost, of every single request. A feature that's charming in a demo can be a money-loser at scale if its per-request cost exceeds what customers will pay — and you won't discover that from the demo, only from the arithmetic. So the discipline is to compute cost-per-request early, check the margin at your intended price, and if it's thin, attack it deliberately: a cheaper or smaller model for easy cases, shorter prompts, output limits, caching of repeated calls, or batching. Unit economics is what makes an AI feature a business rather than an expensive tech demo, and it's invisible until you do the multiplication.
The eval gate
The second number is quality, and it's slippery because LLM outputs are non-deterministic and "looks good" isn't measurable. The answer is an eval: a test set of representative inputs with expected properties, and a scorer that grades a candidate — a new prompt, a different model, a changed pipeline — against it. Now quality is a number, and you can gate on it.
An eval gate is the rule that a change ships only if it passes the quality bar and the economic bar. This is the single practice that most separates a real AI product from a prototype. It catches the prompt tweak that looks better on the three examples you eyeballed but is worse across a hundred — the classic way LLM changes silently regress. And by folding in the margin bar, it also blocks the higher-quality change that would push you underwater on cost. Shipping on a demo means shipping on a curated anecdote; shipping on an eval gate means shipping on measured quality within a cost budget. The whole point is to make "should we ship this?" a computation, not a vibe — which is exactly what the next section formalizes.
The product math
Cost-per-request is tokens times per-token price; margin is what's left of the price after that cost; profit needs price above cost:
1k in @ $2/M + 1k out @ $10/M = $0.012/request. At a $0.05 price, margin = (0.05−0.012)/0.05 = 76%. At $0.005 you'd lose money.
You ship on the eval gate — both the quality score and the margin must clear their bars:
quality 0.9 ≥ 0.8 and margin 0.76 ≥ 0.5 → ship; quality 0.7 < 0.8 → blocked even if cheap. The runnable version below computes cost, margin, profitability, and the ship decision.
Cost, margin, and the ship decision
An LLM product lives or dies on two numbers a demo never shows. Cost-per-request is tokens times per-token price (input + output), which sets your gross margin at whatever you charge and whether you're profitable per request at all. And you ship on an eval gate: a change goes out only if its quality score clears a bar AND its margin clears a bar — so a cheap-but-low-quality release is blocked, and so is a great-but-unprofitable one. Change the token counts, prices, your price point, quality score, and volume, and watch the cost, margin, profitability, and the ship decision.
The reliability stack
Cost and quality are the two headline numbers, but an AI product also needs the plumbing that keeps a probabilistic component dependable in production. This is the layer between "the model works in the demo" and "the feature works for everyone, always":
| Layer | What it does |
|---|---|
| Evals | Turn quality into a number on a representative test set; gate every change. The core loop. |
| Structured outputs | Force machine-readable output (JSON schema / tool calls) so downstream code can rely on the shape. |
| Retries & fallbacks | The model or API will fail sometimes — retry, and fall back to a simpler model or a safe default. |
| Caching | Cache repeated or similar requests to cut cost and latency — often the biggest economic lever. |
| Guardrails | Validate inputs and outputs, block unsafe content, defend against prompt injection. |
| Observability | Log prompts, costs, latencies, and quality so you can see regressions and runaway spend in production. |
None of these appear in a demo, and all of them are load-bearing in a product. The theme is treating the LLM as an unreliable, expensive, non-deterministic component and engineering around that honestly — the same way you'd wrap any flaky external dependency, plus the cost and safety concerns that are unique to models. The teams that win at AI products aren't the ones with the cleverest prompt; they're the ones who built the boring stack — evals, structured outputs, retries, caching, guardrails, observability — that makes the clever prompt dependable and affordable at scale. That stack, wrapped around a measured cost number and a measured quality number, is what "AI product engineering" actually means, and it's the whole difference between a viral demo and a durable product.
Pitfalls
The first and most common is shipping on the demo — mistaking an impressive curated example for evidence the feature works. Without evals you have no idea what your real failure rate is, and you'll ship a change that improved your three test cases while regressing the long tail. Build the eval set before you build the feature; it's the instrument that makes everything else measurable. The second is ignoring cost until the bill arrives. Token costs are invisible in development and brutal at scale, and a feature can be quietly unprofitable for months — compute cost-per-request and margin up front, and monitor spend in production.
Two more. Over-engineering the model, under-engineering the system: teams pour effort into prompt cleverness while skipping the retries, caching, structured outputs, and observability that actually determine whether the product is reliable and affordable — the boring stack is where the value is. And no cheaper path for easy cases: sending every request to your most expensive model wastes money when a smaller model, a cache hit, or a rule would handle the majority — route by difficulty. The unifying idea is that an AI product is not a prompt; it's a system with an economic model and a quality bar wrapped around a probabilistic component. Measure the two numbers that a demo hides — cost and eval score — gate every change on both, and build the reliability stack that keeps a flaky, pricey model dependable at scale. Do that and you cross the last mile that most AI features never finish; skip it and you have a great demo and no business.
Build the eval set first and gate every change on quality and margin. Compute cost-per-request early and monitor spend. Invest in the boring reliability stack (structured outputs, retries, caching, guardrails, observability) over prompt cleverness, and route easy cases to cheaper models. A demo tests none of this — the product is the system around the model.
Worked example: one feature through the eval gate, step by step
A document-summary feature, priced to customers at $0.025 per request, on the page’s model at $2/M input and $10/M output, with the page’s bars: quality ≥ 0.8 and margin ≥ 50%. The eval scores are illustrative; the costs and margins follow from the formula.
| Candidate | Tokens in / out | Cost | Margin | Eval | Gate |
|---|---|---|---|---|---|
| 1. Baseline prompt | 3,000 / 400 | $0.0100 | 60% | 0.78 | Blocked: quality |
| 2. Add retrieved context, all of it | 8,000 / 400 | $0.0200 | 20% | 0.90 | Blocked: margin |
| 3. Retrieve only the top 3 chunks | 4,500 / 400 | $0.0130 | 48% | 0.87 | Blocked: margin, just |
| 4. Plus cap the summary at 250 tokens | 4,500 / 250 | $0.0115 | 54% | 0.86 | Ship |
Each row is cost = in × $2/M + out × $10/M and margin = (0.025 − cost) / 0.025. Candidate 2 is the best summary and would lose most of its revenue to the model bill, which is the “great but unprofitable” case the gate exists to block. Candidate 3 fails by two points, and the fix is on the output side: cutting 150 output tokens saves $0.0015, as much as cutting 750 input tokens, because output costs 5× input here. At 200,000 requests a month, candidate 4 earns (0.025 − 0.0115) × 200,000 = $2,700 of gross margin. Note what the gate did not do: it never asked which prompt looked best in a demo.
Where this shows up in production and interviews
- Pricing and usage limits
Credits, request caps and “fair use” limits on AI plans exist because cost-per-request is real. They are the margin bar turned into a product rule: a heavy user cannot push a request’s cost above what the plan earns.
- Model upgrades and price changes
When a provider ships a new model or changes a rate card, both gate inputs move at once. Re-run the eval and recompute the margin before switching, the same way you would test any dependency upgrade.
- Evals in CI
The gate is most useful as an automated check on every prompt or pipeline change, next to the unit tests. A merge that drops the eval score or raises the token count fails before it reaches users.
- System design interviews
“Design an AI feature” questions reward naming both numbers: estimate tokens per request and cost at the stated volume, then say how you would measure quality and what bar would block a launch.
Check yourself
Use the page’s prices ($2/M in, $10/M out) unless a question says otherwise.
An agent loop calls the model 10 times per user request, each call with 2,000 input tokens and 300 output tokens. What does one user request cost?
One call is 2,000 × $2/M + 300 × $10/M = $0.004 + $0.003 = $0.007. Ten calls is $0.07. A loop multiplies cost by its step count, so a $0.05 price that works for a single call loses money here.
A new prompt scores 0.92 on the eval (bar 0.8) but raises cost from $0.012 to $0.030 at a $0.05 price. Does it ship with a 50% margin bar?
Margin becomes (0.05 − 0.030) / 0.05 = 40%, under the 50% bar, so the gate blocks it. The options are to cut its tokens, route easy cases to a cheaper model, or raise the price, then run it through the gate again.
You can either cut 1,000 input tokens or 200 output tokens from every request. Which saves more?
Input: 1,000 × $2/M = $0.002. Output: 200 × $10/M = $0.002. They save the same. At a 5:1 price ratio, one output token is worth five input tokens, so a small output cut often matches a large prompt trim.
A change improves the three examples your team eyeballed but drops the eval from 0.85 to 0.79. The team says the eval is too strict. What do you do?
Block it, then look at the failing eval cases. Three hand-picked examples are a demo; the eval is the distribution. If the failures show the eval’s expected answers are genuinely wrong, fix the eval and re-score, but do not ship on the anecdote.
Quick answers
What is AI product engineering?
Turning an LLM's capability into a reliable, affordable product — unit economics, evals, and the reliability stack around a non-deterministic model, not just prompting.
How do you calculate LLM cost?
(input tokens × input price + output tokens × output price) per million; times volume for the monthly bill. Long prompts and agent loops multiply it.
What is an eval gate?
A rule that a change ships only if it clears a quality bar (measured on a test set) and a margin bar — blocking regressions and unprofitable changes.
Why isn't a demo enough?
A demo is one curated success; a product is a distribution of real inputs at scale — with failure rates, cost, latency, and safety a demo never tests.
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- The LLM Serving HandbookHow to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.Read →
- The Transformers HandbookThe architecture behind every LLM, built up from scratch — tokens and embeddings, positional encoding, self-attention (query/key/value), multi-head attention, the transformer block (FFN, residuals, layer norm), and how a decoder-only model generates text autoregressively.Read →
- The Fine-Tuning HandbookWhen to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.Read →
- The On-Device AI HandbookSmall models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.Read →