On-call for a system
in someone else’s world
Production engineering assumes things an FDE does not have: a rotation, an SRE team, observability you control, and an internal audience for the post-mortem. You have a system in the customer’s VPC, one person who understands it, and a room that includes the VP who approved the spend. Everything about incidents changes under those constraints.
What is different, precisely
| Dimension | Normal production | Customer production |
|---|---|---|
| Who is paged | A rotation | Often one person — you |
| Observability | Your stack, full access | Their stack, sometimes no egress for logs29 |
| Deploy to fix | Minutes | Their change-control window, possibly next Tuesday |
| Post-mortem audience | Colleagues | The customer, including people deciding your renewal |
| Definition of down | Error rate on a dashboard | Whatever the contract says — often something you did not write |
| Escalation path | Known | Two organisations, unclear ownership, holidays that do not align |
| End state | You own it indefinitely | You must engineer your way out of owning it |
The last one. Every decision in this handbook should be judged against a single question: does this make it easier or harder for their team to run this without me in six months? An FDE who is still the only person who can fix production a year later has failed at the job, however good the uptime is.
Negotiating an SLA you can actually meet
This gets written into a contract, often before you are consulted. Read it early — being surprised by it later is the most expensive mistake in this handbook.
Define what “available” means, in their language
Not “the service returns 200”. Something a business person can check: “a submitted document produces a routing decision within 5 minutes during business hours”. If the definition is ambiguous, every incident starts with an argument about whether it counted.
Scope out what you do not control
Their network, their identity provider, their upstream systems, a model provider’s API, their change-control process. Write the dependency exclusions explicitly. Nobody enjoys this conversation and everybody prefers it to the alternative, which is being liable for their VPN.
Never let the clock start before you can be paged
The single most common trap: a four-hour response commitment where the customer reports incidents by emailing a shared inbox somebody reads at 09:00. The clock must start at the alert you receive, not at the moment their user noticed. Get the reporting channel into the same paragraph as the response time.
Separate response from resolution
You can commit to acknowledging in 30 minutes. You cannot commit to fixing an unknown bug in four hours — and a commitment to resolve in a fixed time is a commitment to ship something untested under pressure. Response time, update cadence, and best-effort resolution: three different promises.
Tie the severity ladder to business impact
Sev1 is “the workflow is stopped”, not “a service is down”. A degraded component nobody depends on at 2am is not a Sev1, and treating it as one is how a one-person rotation burns out in a quarter.
Sev1 workflow stopped for all users ack 30m · updates hourly · best-effort fix
Sev2 degraded or a subset affected ack 4h · updates daily · next release
Sev3 cosmetic / single-case ack 2d · backlog
excluded: customer network, IdP, upstream source systems, model-provider outages,
delays caused by the customer's own change-control window
Being the only one who knows
A one-person rotation is not a rotation. It is a countdown. Three things buy you time.
Write the runbook before you need it
Not architecture — symptoms. “Documents stuck in queue” → check X, then Y, then Z, and if none of those, here is who to call. Written for someone who has never seen the code, at 3am, under pressure. Every incident you handle should add one entry.
Make the system explain itself
Health endpoint that names the failing dependency. Log lines that say what to do, not just what happened. An error that says “source system returned 403 — check the service-account credential expiry” converts a page into a self-service fix and is worth more than any dashboard.
Train a second responder in month two, not month nine
Pick one of their engineers. Have them shadow every incident. Then invert it: they lead, you observe. Two incidents run that way is the difference between a handoff and a hostage situation. This is also the concrete measure of whether enablement is real.
In a VPC, on-prem, or air-gapped deployment you frequently cannot ship logs out.29 Plan for it: a local retention window, a redaction-safe bundle the customer can export and send you, and a documented procedure for producing it. Discovering this during your first Sev1 is a bad evening — and asking a customer to email you raw production logs is a security incident of its own.
Running the incident
1. acknowledge within the SLA window, to the agreed channel, with a next-update time
2. stabilise restore the workflow — rollback, feature flag, manual fallback
3. communicate on the promised cadence, even when there is nothing new
4. diagnose only after the bleeding stops
5. fix through their change control, not around it
6. write it up within 48 hours, while it is still true
Step 3 is the one engineers skip and customers judge you on. Silence during an incident is interpreted as absence. A message saying “still investigating, nothing new, next update at 15:00” costs nothing and is the difference between a customer who feels handled and one who starts calling your account executive.
Stabilise before you understand
- Roll back first; diagnose from the artefacts afterwards.
- A documented manual fallback is worth building on day one — it converts a Sev1 into a Sev2.
- Feature flags let you disable one workflow without taking everything down.
- Never debug in production while the customer’s users are blocked.
Communicate like an owner
- One channel, agreed in advance, with named people on both sides.
- State impact in their terms: how many cases, which team, since when.
- Never speculate on cause to a customer audience during the incident.
- Say what you are doing next and when you will speak again.
The post-mortem, with a client audience
Blameless works differently when the room contains the person who signed the contract.
Internally, blameless means no individual is punished. With a customer there is a second dimension: the customer must leave believing the system is more trustworthy than before, not less. Those two goals are compatible, and the way you get both is by being unusually specific.
| Section | What to write | What kills trust |
|---|---|---|
| Impact | Exact scope in their terms: 340 documents, one team, 09:12–11:40. | Vagueness. It reads as concealment even when it is not. |
| Timeline | Detection, acknowledgement, stabilisation, fix — with timestamps. | Omitting the gap between the failure and your detection. |
| Cause | The mechanism, plainly, including if it was your bug. | Passive voice. “An error occurred” fools nobody and reads as evasion. |
| Why it was not caught | The missing test, the missing alert, the untested assumption. | Skipping this. It is the section that actually builds confidence. |
| Actions | Two or three, each with an owner and a date, including any on their side. | Ten actions with no owners — a list nobody will execute. |
| Detection improvement | The alert that will catch this class next time. | Fixing only the instance, not the class. |
It often is: a credential rotated without notice, a schema changed upstream, a firewall rule. Write it neutrally as a system property rather than a person’s error — “the integration had no alert for credential expiry”, not “their team let the credential expire” — and put the corresponding action item on your side too. You are trying to fix a system, not win an argument, and the person who feels blamed in a post-mortem becomes an obstacle for the rest of the engagement.
The conversation, not the write-up
The post-mortem meeting is a customer conversation with a specific shape. The roleplay simulator covers the adjacent ones — a live failure, bad news, and a stakeholder who is not on your side.
Engineering your way out of the rotation
The goal is not heroic uptime. It is a system their team runs without you. Four milestones, in order — and a deployment that has not reached the third by month six is drifting toward permanent ownership.
Runbook covers the last five incidents
Symptom-first entries, each added the day the incident closed.
Their engineer has shadowed two incidents
Not read about them — been on the call, watched the diagnosis, asked questions.
Their engineer has led one, with you observing
The real test. Whatever they could not do alone becomes your next runbook entry or your next piece of self-explaining error handling.
You are the escalation, not the first responder
Documented, agreed, and reflected in the support arrangement. This is the handoff actually completing rather than being declared.
Track the share of incidents resolved without you, month over month. It is the only honest measure of whether the handoff is real, and it belongs in the quarterly business review next to the adoption numbers — a customer whose team can run the system is a customer who renews.
Quick answers
How do you negotiate an SLA as a forward deployed engineer?
Define availability in business terms the customer can check, explicitly exclude dependencies you do not control (their network, identity provider, upstream systems, model-provider outages, their own change-control window), make sure the response clock starts at the alert you receive rather than when a user noticed, separate response time from resolution time, and tie severity levels to business impact rather than to component health.
How do you run on-call when you are the only person who knows the system?
Three things buy time: a symptom-first runbook written for someone who has never seen the code, a system that explains itself through health checks and actionable error messages, and a second responder from the customer’s team trained in month two — shadowing incidents first, then leading one while you observe.
How do you run a post-mortem with a customer in the room?
Be unusually specific: exact impact in their terms, a timestamped timeline including the gap before detection, the mechanism stated plainly in active voice, an explicit section on why it was not caught, and two or three actions with owners and dates. When the cause is on the customer’s side, write it as a missing system property rather than a person’s error and put a corresponding action on your side.
What makes incident response different in a customer environment?
You may be the only responder, observability may be limited or unable to leave their network, fixes go through their change-control window rather than your deploy pipeline, the post-mortem audience includes people who decide your renewal, and the definition of “down” is whatever the contract says. Every decision should be judged on whether it makes it easier for their team to run the system without you.
How do you hand over on-call to the customer?
Four milestones in order: the runbook covers the last five incidents symptom-first; their engineer has shadowed two incidents live; their engineer has led one while you observed; and you are formally the escalation rather than the first responder. Track the share of incidents resolved without you month over month — it is the only honest measure that the handoff is real.
Sources
Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.
Cited on this page
- Anthropic — Forward Deployed Engineer, Applied AI (JD) — 40/30/30 split; MCP servers, sub-agents and agent skills as deliverables. jobs.menlovc.com/companies/anthropic/jobs/69674588-forward-deployed-engineer-applied-ai
- PostHog — “WTF is a forward deployed engineer?” — first-party account of the role at a product company, including travel reality. posthog.com/blog/forward-deployed-engineer
- Anthropic — Forward Deployed Engineer, Federal Civilian (JD) — air-gapped and IL-level deployment requirements. jobs.menlovc.com/companies/anthropic/jobs/76278352-forward-deployed-engineer-federal-civilian
Keep going
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Agentic AI Interview HandbookTwenty topics every senior AI engineer should be able to reason about live — from eval pipelines to reliability patterns for generative systems.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- 50 Angular Interview QuestionsA visual handbook covering components, change detection, RxJS, signals, routing, forms, performance, and testing — what interviewers actually probe for in senior Angular roles.Read →
- 50 Python Interview QuestionsFundamentals to advanced: data structures, OOP, iterators & generators, the GIL, asyncio, memory, testing, and the standard library — a visual walk through everything a Python interview touches.Read →
- 51 LLM Evals Interview QuestionsGolden sets, LLM-as-judge, regression testing, offline vs online evals, RAG evals, agent evals, red-teaming, and observability — demystified for interviews and production.Read →
Explore more from Vibe Engines
Get the next one in your inbox.
New handbooks, system-design walkthroughs, and tools — straight to your inbox. No spam, unsubscribe anytime.