Proving it worked,
before renewal
MIT’s NANDA study found roughly 95% of enterprise AI pilots produced no measurable profit-and-loss impact. Read that carefully: no measurable. A large share of those systems did something useful and could not demonstrate it, because nobody instrumented anything until the budget conversation. This is how not to be in that 95%.
The gap that kills deployments
“It works” is your opinion. “Adjudication time went from 41 minutes to 9, on 2,300 cases, over eight weeks” is a fact your champion can take into a meeting you are not in.
At renewal, your champion has to defend this spend against three other things competing for the same budget, in a room where you are not present. Everything you build toward this is really about arming that one conversation.
Two things make this specifically an FDE problem rather than a product-analytics problem. First, you are usually measuring a workflow change inside someone else’s organisation, where you do not own the systems the baseline lives in. Second, the measurement has to be agreed before the result is known — a metric proposed after the fact is always suspected of being chosen to flatter.
The commercial logic behind the whole role rests on this. a16z’s services-led-growth thesis is that companies deliberately accept lower-margin deployment work because it produces defensible customer outcomes.8 That bargain only holds if outcomes are demonstrable. An FDE who cannot prove impact is a cost centre wearing an engineer’s badge.
Instrument on day one, not at renewal
The single highest-return hour of an engagement is capturing the baseline before you change anything.
Capture the baseline while it still exists
Time the manual workflow with a stopwatch if that is what it takes. Count the cases. Count the errors. Get last quarter’s numbers from whoever has them. Once your system is live the baseline is gone forever, and reconstructing it after the fact is both painful and unconvincing.
Write it down and send it to your champion the same week: “here is what today looks like, agreed?” That email is your most valuable artefact at renewal.
Agree the one number
Not a dashboard of twelve. One primary metric that everyone signs off on, with two supporting ones. If the room cannot agree on one, you have discovered a scoping problem, not a measurement problem — and better now than in month four.
Build the telemetry into the thing you ship
Who used it, when, on what, what they did with the output, and whether they overrode it. Adding this later is an awkward retrofit; adding it in week two costs almost nothing and is the difference between evidence and anecdote.
Log every time a human changes the system’s output. It is simultaneously a quality signal, a trust signal, and a free stream of labels for your golden set. A falling override rate over eight weeks is the most persuasive chart you can put in a business review, because it shows the humans deciding to trust it.
The metrics that survive scrutiny
Four tiers. A business review needs at least one from each of the first three.
| Tier | Metric | What it proves | How it is attacked |
|---|---|---|---|
| Adoption | Weekly active users of the workflow (not the app) | People chose to use it without being told to. | “They were mandated to.” Counter with voluntary-use segments. |
| Adoption | Share of eligible cases going through the system | It handles real volume, not a demo slice. | “Only the easy ones.” Counter by segmenting by difficulty. |
| Workflow | Time per case, before and after | The work actually got faster. | “The baseline was inflated.” Counter with the week-one signed-off email. |
| Workflow | Deflection rate — cases resolved without a human | Capacity was genuinely created. | “Quality dropped.” Counter with eval results on the same period. |
| Quality | Eval score on the frozen golden set, over time | Speed did not come from cutting corners. | “Your test set is easy.” Counter with stratification and the human baseline. |
| Quality | Override rate, trending | Trust is increasing among the actual users. | Hard to attack. This is why it is the strongest chart you have. |
| Business | Hours returned × loaded cost, or revenue/loss moved | Money. | “Those hours were not redeployed.” See the next section. |
API calls, tokens processed, documents ingested, “queries served”. These measure your system’s activity, not the customer’s outcome, and a sophisticated buyer knows the difference. Including them dilutes the metrics that do matter and signals that you could not find a real one.
Making an ROI claim a CFO will accept
Most engineer-authored ROI numbers fall over on the same three objections. Pre-empt all three and the number survives.
hours_saved = cases_processed × (minutes_before − minutes_after) / 60
gross_value = hours_saved × fully_loaded_hourly_cost
net_value = gross_value − licence − infra − their_internal_effort
# then subtract the three objections below, in writing, before they are raised.
“Saved hours are not saved money”
Correct, and it is the objection that sinks most AI business cases. Nobody was made redundant, so the salary line did not move. The fix is to say what the hours became: a backlog cleared, a queue that no longer needs contractors at quarter-end, a headcount request withdrawn, cases handled without the seasonal temp hires. Get your champion to name the redeployment before you write the number.
“How much of that was you?”
Other things changed in the same period. Never claim the whole delta. Segment: compare teams that adopted against teams that did not, or compare case types the system covers against ones it does not. A defensible partial attribution beats an indefensible total one, and offering the caveat yourself buys enormous credibility.
“Will it hold?”
Eight weeks of improvement can be a novelty effect. Answer with the trend line, the override rate falling rather than flat, and the drift monitoring that will catch degradation after handoff. This is where your eval work pays a commercial dividend.
Report a range with the assumptions listed, and include the case that did not work. A single confident number reads as marketing; a range with stated assumptions and one honest failure reads as engineering — and engineering is what they are buying from you.
The business review
Forty-five minutes, quarterly, and it decides the renewal. Build it once and reuse the structure.
| Minutes | Section | Content |
|---|---|---|
| 0–3 | The answer | The one number and its direction. No build-up. If they leave after three minutes they should have the point. |
| 3–10 | Adoption | Who uses it, how often, share of eligible volume. Segment by team — the variation is where the next quarter’s work is. |
| 10–18 | Quality | Eval trend on the frozen set, override rate, incidents. Include a regression if there was one, with what you did. |
| 18–28 | Value | The ROI range, assumptions listed, attribution caveated, redeployment named. |
| 28–38 | What is not working | The workflow with low adoption and why. Do not skip this. A review with no problems reads as a review that was not looking. |
| 38–45 | Next | Three options with effort and expected impact. Let them choose — ownership of the choice is what makes the next quarter funded. |
Where possible, your champion — with you in the room to answer detail. A win presented by the customer’s own person is worth several times the same slides presented by the vendor, because internally it is their success, and people fund their own successes.
Half of this review is your eval work
The quality section is the frozen golden set over time. If that does not exist, the review has an opinion where it needs a chart.
The economics that pay your salary
Worth understanding, because it explains which deployments get resourced and which quietly stop.
An FDE is expensive. The reason companies employ them anyway is a straightforward margin argument, and it is the argument a16z made explicitly: trade gross margin for deployment depth, because depth is defensible.8 Enterprise deals that require heavy deployment work carry contract values large enough to absorb the cost of a person, and the resulting integration is hard for a competitor to displace.
What this means for your work
- Your value is measured against the contract, not against tickets closed.
- Anything you build that generalises into the product improves the margin permanently — which is why the generalisation memo is the rung that moves careers.
- Protecting a renewal is worth more than adding a feature.
- Reducing the hours the next deployment needs is worth more still.
Signals your deployment is economically unhealthy
- Nothing you build ever lands in the product.
- Your time is measured as utilisation rather than as outcomes.
- The customer’s spend is flat while your hours rise.
- No one can name the metric this deployment is supposed to move.
The last bullet is the one to act on immediately. A deployment nobody can attach a number to is a deployment that will be cut in the next budget cycle regardless of how good the engineering was — and it is why this handbook belongs to the practitioner track rather than the commercial one. See also the services-trap checklist.
Quick answers
How do you prove the ROI of an AI deployment?
Capture the baseline in week one before anything changes, agree one primary metric with the customer up front, instrument adoption and override rate inside the product, then report hours or errors moved with the assumptions listed and attribution caveated. Pre-empt the three standard objections: that saved hours are not saved money, that other things changed too, and that the improvement may not hold.
What metrics should an FDE track for a deployment?
At least one from each of three tiers. Adoption: weekly active users of the workflow and share of eligible cases going through the system. Workflow: time per case before and after, and deflection rate. Quality: eval score on a frozen golden set over time, and override rate trending. Business value sits on top of those. Avoid vanity metrics like API calls or documents processed.
Why do most enterprise AI pilots fail to show impact?
MIT’s NANDA study found roughly 95% produced no measurable profit-and-loss impact, and the operative word is measurable. Many worked but could not be demonstrated, because no baseline was captured before the change, no metric was agreed in advance, and adoption was never instrumented — leaving only anecdotes at the budget conversation.
What is the override rate and why does it matter?
The share of system outputs a human changes before acting on them. It is simultaneously a quality signal, a trust signal, and a free source of labels for your golden set. A falling override rate over several weeks is the most persuasive chart in a business review, because it shows the users choosing to trust the system rather than being told to.
Who should present the quarterly business review?
Where possible the customer’s champion, with the forward deployed engineer in the room for detail. A result presented by the customer’s own person carries far more internal weight than the same slides from a vendor, because internally it becomes their success — and organisations fund their own successes.
Sources
Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.
Cited on this page
- a16z — Services-Led Growth — the margin-for-moat thesis behind FDE hiring. a16z.com/services-led-growth/
- levels.fyi — Palantir FDSE compensation — www.levels.fyi/companies/palantir/salaries/software-engineer/title/fdse
- MIT NANDA “State of AI in Business” — 95% of pilots with no measurable P&L impact — widely reported; figure cited via press coverage rather than a public PDF. techcrunch.com/2026/07/30/forward-deployed-engineers-are-the-ai-industrys-latest-talent-obsession/
Keep going
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Agentic AI Interview HandbookTwenty topics every senior AI engineer should be able to reason about live — from eval pipelines to reliability patterns for generative systems.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- 50 Angular Interview QuestionsA visual handbook covering components, change detection, RxJS, signals, routing, forms, performance, and testing — what interviewers actually probe for in senior Angular roles.Read →
- 50 Python Interview QuestionsFundamentals to advanced: data structures, OOP, iterators & generators, the GIL, asyncio, memory, testing, and the standard library — a visual walk through everything a Python interview touches.Read →
- 51 LLM Evals Interview QuestionsGolden sets, LLM-as-judge, regression testing, offline vs online evals, RAG evals, agent evals, red-teaming, and observability — demystified for interviews and production.Read →
Explore more from Vibe Engines
Get the next one in your inbox.
New handbooks, system-design walkthroughs, and tools — straight to your inbox. No spam, unsubscribe anytime.