A demo isn't
a product.

AI features have a famous shape: the first mile is trivial and the last mile is brutal. Getting to a demo — the model summarizing a document, answering a question, writing some code — is fast and thrilling. But a demo is a single curated success, while a product is a distribution of real inputs at scale. The demo hides the failure rate on the ugly 20% of inputs, the bill when a million users arrive, the latency under load, and the day the model returns malformed JSON or gets prompt-injected.

AI product engineering is the work of crossing that gap, and it's a genuinely different job from prompting. It layers three concerns a playground ignores: unit economics (every request costs real money in tokens, so you must know your cost and margin), quality measurement (LLMs are non-deterministic, so "it looks good" must become a number via evals), and reliability engineering (retries, fallbacks, caching, guardrails, latency budgets around a model that will sometimes fail). A demo proves the model can do the task; product engineering makes it do the task correctly, affordably, and dependably a million times. The two numbers that gate that transition — cost and eval score — are the spine of this handbook.

The first number a demo hides is cost. LLM APIs charge by the token — a price per million input tokens and a (usually higher) price per million output tokens — so every request has a real, computable cost: (in_tokens × in_price + out_tokens × out_price), per million. At your price to the customer, that cost determines your gross margin — the fraction of revenue left after the model bill — and whether the feature makes money at all.

This turns a lot of "engineering" choices into economic ones. A long system prompt, a big retrieved context, a chatty output, or a multi-step agent loop that calls the model ten times all multiply the token count, and therefore the cost, of every single request. A feature that's charming in a demo can be a money-loser at scale if its per-request cost exceeds what customers will pay — and you won't discover that from the demo, only from the arithmetic. So the discipline is to compute cost-per-request early, check the margin at your intended price, and if it's thin, attack it deliberately: a cheaper or smaller model for easy cases, shorter prompts, output limits, caching of repeated calls, or batching. Unit economics is what makes an AI feature a business rather than an expensive tech demo, and it's invisible until you do the multiplication.

The second number is quality, and it's slippery because LLM outputs are non-deterministic and "looks good" isn't measurable. The answer is an eval: a test set of representative inputs with expected properties, and a scorer that grades a candidate — a new prompt, a different model, a changed pipeline — against it. Now quality is a number, and you can gate on it.

An eval gate is the rule that a change ships only if it passes the quality bar and the economic bar. This is the single practice that most separates a real AI product from a prototype. It catches the prompt tweak that looks better on the three examples you eyeballed but is worse across a hundred — the classic way LLM changes silently regress. And by folding in the margin bar, it also blocks the higher-quality change that would push you underwater on cost. Shipping on a demo means shipping on a curated anecdote; shipping on an eval gate means shipping on measured quality within a cost budget. The whole point is to make "should we ship this?" a computation, not a vibe — which is exactly what the next section formalizes.

Cost-per-request is tokens times per-token price; margin is what's left of the price after that cost; profit needs price above cost:

1k in @ $2/M + 1k out @ $10/M = $0.012/request. At a $0.05 price, margin = (0.05−0.012)/0.05 = 76%. At $0.005 you'd lose money.

You ship on the eval gate — both the quality score and the margin must clear their bars:

quality 0.9 ≥ 0.8 and margin 0.76 ≥ 0.5 → ship; quality 0.7 < 0.8 → blocked even if cheap. The runnable version below computes cost, margin, profitability, and the ship decision.

Cost and quality are the two headline numbers, but an AI product also needs the plumbing that keeps a probabilistic component dependable in production. This is the layer between "the model works in the demo" and "the feature works for everyone, always":

None of these appear in a demo, and all of them are load-bearing in a product. The theme is treating the LLM as an unreliable, expensive, non-deterministic component and engineering around that honestly — the same way you'd wrap any flaky external dependency, plus the cost and safety concerns that are unique to models. The teams that win at AI products aren't the ones with the cleverest prompt; they're the ones who built the boring stack — evals, structured outputs, retries, caching, guardrails, observability — that makes the clever prompt dependable and affordable at scale. That stack, wrapped around a measured cost number and a measured quality number, is what "AI product engineering" actually means, and it's the whole difference between a viral demo and a durable product.

The first and most common is shipping on the demo — mistaking an impressive curated example for evidence the feature works. Without evals you have no idea what your real failure rate is, and you'll ship a change that improved your three test cases while regressing the long tail. Build the eval set before you build the feature; it's the instrument that makes everything else measurable. The second is ignoring cost until the bill arrives. Token costs are invisible in development and brutal at scale, and a feature can be quietly unprofitable for months — compute cost-per-request and margin up front, and monitor spend in production.

Two more. Over-engineering the model, under-engineering the system: teams pour effort into prompt cleverness while skipping the retries, caching, structured outputs, and observability that actually determine whether the product is reliable and affordable — the boring stack is where the value is. And no cheaper path for easy cases: sending every request to your most expensive model wastes money when a smaller model, a cache hit, or a rule would handle the majority — route by difficulty. The unifying idea is that an AI product is not a prompt; it's a system with an economic model and a quality bar wrapped around a probabilistic component. Measure the two numbers that a demo hides — cost and eval score — gate every change on both, and build the reliability stack that keeps a flaky, pricey model dependable at scale. Do that and you cross the last mile that most AI features never finish; skip it and you have a great demo and no business.