Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
On-call for a system
in someone else’s world
Production engineering assumes things an FDE does not have: a rotation, an SRE team, observability you control, and an internal audience for the post-mortem. You have a system in the customer’s VPC, one person who understands it, and a room that includes the VP who approved the spend. Everything about incidents changes under those constraints.
What is different, precisely
| Dimension | Normal production | Customer production |
|---|---|---|
| Who is paged | A rotation | Often one person — you |
| Observability | Your stack, full access | Their stack, sometimes no egress for logs29 |
| Deploy to fix | Minutes | Their change-control window, possibly next Tuesday |
| Post-mortem audience | Colleagues | The customer, including people deciding your renewal |
| Definition of down | Error rate on a dashboard | Whatever the contract says — often something you did not write |
| Escalation path | Known | Two organisations, unclear ownership, holidays that do not align |
| End state | You own it indefinitely | You must engineer your way out of owning it |
The last one. Every decision in this handbook should be judged against a single question: does this make it easier or harder for their team to run this without me in six months? An FDE who is still the only person who can fix production a year later has failed at the job, however good the uptime is.
Negotiating an SLA you can actually meet
This gets written into a contract, often before you are consulted. Read it early — being surprised by it later is the most expensive mistake in this handbook.
Define what “available” means, in their language
Not “the service returns 200”. Something a business person can check: “a submitted document produces a routing decision within 5 minutes during business hours”. If the definition is ambiguous, every incident starts with an argument about whether it counted.
Scope out what you do not control
Their network, their identity provider, their upstream systems, a model provider’s API, their change-control process. Write the dependency exclusions explicitly. Nobody enjoys this conversation and everybody prefers it to the alternative, which is being liable for their VPN.
Never let the clock start before you can be paged
The single most common trap: a four-hour response commitment where the customer reports incidents by emailing a shared inbox somebody reads at 09:00. The clock must start at the alert you receive, not at the moment their user noticed. Get the reporting channel into the same paragraph as the response time.
Separate response from resolution
You can commit to acknowledging in 30 minutes. You cannot commit to fixing an unknown bug in four hours — and a commitment to resolve in a fixed time is a commitment to ship something untested under pressure. Response time, update cadence, and best-effort resolution: three different promises.
Tie the severity ladder to business impact
Sev1 is “the workflow is stopped”, not “a service is down”. A degraded component nobody depends on at 2am is not a Sev1, and treating it as one is how a one-person rotation burns out in a quarter.
Sev1 workflow stopped for all users ack 30m · updates hourly · best-effort fix
Sev2 degraded or a subset affected ack 4h · updates daily · next release
Sev3 cosmetic / single-case ack 2d · backlog
excluded: customer network, IdP, upstream source systems, model-provider outages,
delays caused by the customer's own change-control window
Being the only one who knows
A one-person rotation is not a rotation. It is a countdown. Three things buy you time.
Write the runbook before you need it
Not architecture — symptoms. “Documents stuck in queue” → check X, then Y, then Z, and if none of those, here is who to call. Written for someone who has never seen the code, at 3am, under pressure. Every incident you handle should add one entry.
Make the system explain itself
Health endpoint that names the failing dependency. Log lines that say what to do, not just what happened. An error that says “source system returned 403 — check the service-account credential expiry” converts a page into a self-service fix and is worth more than any dashboard.
Train a second responder in month two, not month nine
Pick one of their engineers. Have them shadow every incident. Then invert it: they lead, you observe. Two incidents run that way is the difference between a handoff and a hostage situation. This is also the concrete measure of whether enablement is real.
In a VPC, on-prem, or air-gapped deployment you frequently cannot ship logs out.29 Plan for it: a local retention window, a redaction-safe bundle the customer can export and send you, and a documented procedure for producing it. Discovering this during your first Sev1 is a bad evening — and asking a customer to email you raw production logs is a security incident of its own.
Running the incident
1. acknowledge within the SLA window, to the agreed channel, with a next-update time
2. stabilise restore the workflow — rollback, feature flag, manual fallback
3. communicate on the promised cadence, even when there is nothing new
4. diagnose only after the bleeding stops
5. fix through their change control, not around it
6. write it up within 48 hours, while it is still true
Step 3 is the one engineers skip and customers judge you on. Silence during an incident is interpreted as absence. A message saying “still investigating, nothing new, next update at 15:00” costs nothing and is the difference between a customer who feels handled and one who starts calling your account executive.
Stabilise before you understand
- Roll back first; diagnose from the artefacts afterwards.
- A documented manual fallback is worth building on day one — it converts a Sev1 into a Sev2.
- Feature flags let you disable one workflow without taking everything down.
- Never debug in production while the customer’s users are blocked.
Communicate like an owner
- One channel, agreed in advance, with named people on both sides.
- State impact in their terms: how many cases, which team, since when.
- Never speculate on cause to a customer audience during the incident.
- Say what you are doing next and when you will speak again.
The post-mortem, with a client audience
Blameless works differently when the room contains the person who signed the contract.
Internally, blameless means no individual is punished. With a customer there is a second dimension: the customer must leave believing the system is more trustworthy than before, not less. Those two goals are compatible, and the way you get both is by being unusually specific.
| Section | What to write | What kills trust |
|---|---|---|
| Impact | Exact scope in their terms: 340 documents, one team, 09:12–11:40. | Vagueness. It reads as concealment even when it is not. |
| Timeline | Detection, acknowledgement, stabilisation, fix — with timestamps. | Omitting the gap between the failure and your detection. |
| Cause | The mechanism, plainly, including if it was your bug. | Passive voice. “An error occurred” fools nobody and reads as evasion. |
| Why it was not caught | The missing test, the missing alert, the untested assumption. | Skipping this. It is the section that actually builds confidence. |
| Actions | Two or three, each with an owner and a date, including any on their side. | Ten actions with no owners — a list nobody will execute. |
| Detection improvement | The alert that will catch this class next time. | Fixing only the instance, not the class. |
It often is: a credential rotated without notice, a schema changed upstream, a firewall rule. Write it neutrally as a system property rather than a person’s error — “the integration had no alert for credential expiry”, not “their team let the credential expire” — and put the corresponding action item on your side too. You are trying to fix a system, not win an argument, and the person who feels blamed in a post-mortem becomes an obstacle for the rest of the engagement.
The conversation, not the write-up
The post-mortem meeting is a customer conversation with a specific shape. The roleplay simulator covers the adjacent ones — a live failure, bad news, and a stakeholder who is not on your side.
Engineering your way out of the rotation
The goal is not heroic uptime. It is a system their team runs without you. Four milestones, in order — and a deployment that has not reached the third by month six is drifting toward permanent ownership.
Runbook covers the last five incidents
Symptom-first entries, each added the day the incident closed.
Their engineer has shadowed two incidents
Not read about them — been on the call, watched the diagnosis, asked questions.
Their engineer has led one, with you observing
The real test. Whatever they could not do alone becomes your next runbook entry or your next piece of self-explaining error handling.
You are the escalation, not the first responder
Documented, agreed, and reflected in the support arrangement. This is the handoff actually completing rather than being declared.
Track the share of incidents resolved without you, month over month. It is the only honest measure of whether the handoff is real, and it belongs in the quarterly business review next to the adoption numbers — a customer whose team can run the system is a customer who renews.
The picture: one incident against the SLA clock
Worked example: does this incident breach the SLA?
Take the page’s post-mortem impact line, 340 documents, one team, 09:12–11:40, and fill in the rest (illustrative): a source-system credential expired at 09:12. The contract defines available as “a submitted document gets a routing decision within 5 minutes during business hours”, with a 99.5% monthly target. Say a business month is 22 days × 9 hours.
| Step | Question | Arithmetic | Answer |
|---|---|---|---|
| 1 | Detection gap | 09:12 failure → 09:31 alert | 19 min |
| 2 | Acknowledged inside the window? | 09:31 alert → 09:44 ack, against 30 min | 13 min: yes |
| 3 | Workflow fully stopped (Sev1) | 09:12 → 10:05 fallback | 53 min |
| 4 | Degraded on the fallback (Sev2) | 10:05 → 11:40 their team rotates the credential | 95 min |
| 5 | Business minutes in the month | 22 × 9 × 60 | 11,880 min |
| 6 | Downtime budget at 99.5% | 11,880 × 0.005 | 59.4 min |
| 7 | If the fallback counts as available | 53 min of downtime | 99.55%: target met |
| 8 | If it does not | 53 + 95 = 148 min | 98.75%: target missed |
Steps 7 and 8 are the same incident with two different answers, and the only difference is a sentence the contract never contained: whether a manually routed document meets the definition. That argument should happen while drafting the SLA, not in the post-mortem. Note also what the clock rule bought you. Measured from the failure, the ack took 32 minutes and would have missed a 30-minute promise. Measured from the alert, as negotiated, it took 13. And the cause sits in an exclusion only if “upstream source systems” was written into it.
Cost and trade-offs: what each availability number commits you to
Here Time is the downtime budget for a 30-day month around the clock, Space is what that budget demands of a small rotation, and Why is the catch.
| Case | Time | Space | Why |
|---|---|---|---|
| 99% | 432 min | Business-hours cover plus best effort | Room for one slow fix, even one that waits for their change window. |
| 99.5% | 216 min | Someone reachable out of hours for Sev1 | Survivable for one person, if the fallback is built and rehearsed. |
| 99.9% | 43.2 min | A real rotation: someone always within minutes of a laptop | One Sev1 that waits for “next Tuesday” uses the month. Rollback or fallback becomes mandatory. |
| 99.95% | 21.6 min | Automated failover; a human acknowledging is already too slow | Not a promise a one-person rotation can keep. Push back or exclude dependencies. |
| Fixed resolution time | Whatever the clause says | Shipping an untested fix under pressure | Promise response and update cadence instead, and best-effort resolution. |
The numbers go down by roughly ten times while the staffing they imply goes up by much more. Before agreeing to a target, divide its budget by the time your slowest dependency, often their change-control window, takes to move.
Common mistakes
- Alerting on components, not the workflow
Every health check is green while documents pile up, because the service is up and the upstream credential is not. Alert on the outcome the SLA defines, such as decisions per minute or queue age, so the detection gap is minutes, not a user’s phone call.
- Leaving the fallback out of the definition
As the worked example shows, the same incident is a pass or a breach depending on whether a degraded mode counts as available. Write it into the SLA, or every Sev2 starts with an argument.
- Hot-fixing around their change control
A quick patch outside their process feels like ownership during the incident and becomes the post-mortem’s second finding. Stabilise with what is already approved (rollback, flag, fallback), then ship the fix through their window.
- Starting the timeline at the acknowledgement
A post-mortem timeline that begins when you were paged hides the detection gap, which is the part the customer experienced first and the part your alerting can actually improve.
- Shadowing that never inverts
Their engineer sitting in on ten incidents is still not a second responder. Until they lead one with you watching, the knowledge stays in your head and the rotation is still one person.
Where this shows up in production and interviews
- SLOs and error budgets
Google’s SRE practice separates the internal objective (SLO) from the contractual promise (SLA) and treats the gap as an error budget. Holding your own SLO tighter than the contract gives you warning before a breach, not after.
- Service credits
Many commercial SLAs pay a missed target as a credit against the bill rather than as damages. That changes the negotiation: the credit is rarely the real cost, the renewal conversation is, and that is why the post-mortem matters more than the number.
- Status updates as a practice
Public status pages and incident channels exist because of step 3 of the loop: a timed “still investigating, next update at 15:00” is cheaper than the calls that silence produces. Agree the channel before the first incident, not during it.
- The interview version
FDE client roleplays often open with production down and an executive on the call. Interviewers listen for the order: impact in their terms, what you are doing to stabilise, when you will next speak. No guessing at the cause, and no blame.
Check yourself
The contract says “respond within 4 hours of a report”. A user emails the shared inbox at 17:50 on Friday and nobody reads it until Monday at 09:00. Breach?
Yes. The clock started at 17:50, so the window closed at 21:50 on Friday, more than two days before anyone looked. This is the trap the page warns about. The fix is in the contract: the clock starts at the alert you receive, and the reporting channel goes in the same paragraph as the response time.
A Sev1 under a 99.9% target needs a fix that can only ship in their change window, three days away. What do you do now?
Stabilise without a deploy: roll back, turn off the failing workflow with a flag, or switch on the manual fallback. The month’s budget is 43.2 minutes, so waiting three days for the window spends it many times over. The real fix then ships through the window as a planned change.
Their team rotated a credential without telling anyone. How do you write the cause, and what action goes on your side?
As a system property: “the integration had no alert for credential expiry or a spike in 403s.” Your action is that alert, ideally one that fires before expiry. Their action, if any, is notice of rotation. Nobody is named, and both sides own a fix.
Month five: their team resolved 1 of the last 8 incidents without you. Which milestone are you at, and what is next?
12.5%. If their engineer handled that one, you have just reached M3, with a month to spare before the page’s month-six warning. One incident is not a trend, so look at the share month over month. Have them lead the next one with you watching, turn whatever stops them into a runbook entry, and aim for M4, where you are only the escalation.
Quick answers
How do you negotiate an SLA as a forward deployed engineer?
Define availability in business terms the customer can check, explicitly exclude dependencies you do not control (their network, identity provider, upstream systems, model-provider outages, their own change-control window), make sure the response clock starts at the alert you receive rather than when a user noticed, separate response time from resolution time, and tie severity levels to business impact rather than to component health.
How do you run on-call when you are the only person who knows the system?
Three things buy time: a symptom-first runbook written for someone who has never seen the code, a system that explains itself through health checks and actionable error messages, and a second responder from the customer’s team trained in month two — shadowing incidents first, then leading one while you observe.
How do you run a post-mortem with a customer in the room?
Be unusually specific: exact impact in their terms, a timestamped timeline including the gap before detection, the mechanism stated plainly in active voice, an explicit section on why it was not caught, and two or three actions with owners and dates. When the cause is on the customer’s side, write it as a missing system property rather than a person’s error and put a corresponding action on your side.
What makes incident response different in a customer environment?
You may be the only responder, observability may be limited or unable to leave their network, fixes go through their change-control window rather than your deploy pipeline, the post-mortem audience includes people who decide your renewal, and the definition of “down” is whatever the contract says. Every decision should be judged on whether it makes it easier for their team to run the system without you.
How do you hand over on-call to the customer?
Four milestones in order: the runbook covers the last five incidents symptom-first; their engineer has shadowed two incidents live; their engineer has led one while you observed; and you are formally the escalation rather than the first responder. Track the share of incidents resolved without you month over month — it is the only honest measure that the handoff is real.
Sources
Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.
Cited on this page
- Anthropic — Forward Deployed Engineer, Applied AI (JD) — 40/30/30 split; MCP servers, sub-agents and agent skills as deliverables. jobs.menlovc.com/companies/anthropic/jobs/69674588-forward-deployed-engineer-applied-ai
- PostHog — “WTF is a forward deployed engineer?” — first-party account of the role at a product company, including travel reality. posthog.com/blog/forward-deployed-engineer
- Anthropic — Forward Deployed Engineer, Federal Civilian (JD) — air-gapped and IL-level deployment requirements. jobs.menlovc.com/companies/anthropic/jobs/76278352-forward-deployed-engineer-federal-civilian
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Agent Evaluations HandbookA self-contained handbook on evaluating AI agents — theory, interactive widgets, and practical guidance. Trajectory evals, tool-use scoring, LLM-as-judge, observability, and reliability for PMs, engineers, and founders.Read →
- The Agent Patterns HandbookThe design patterns behind every LLM agent — the ReAct Thought–Action–Observation loop, tool/function calling, plan-and-execute, reflection, memory, multi-agent orchestration, and the failure modes (loops, hallucinated tools, recovery, human-in-the-loop) that break agents in production.Read →
- The Kubernetes HandbookThe one idea under all the YAML — declare desired state, and a control loop makes reality match. Covers the orchestration problem, pods, deployments and replicasets, services and networking, the reconciliation loop and self-healing, the scheduler, config/secrets and health probes, autoscaling, and when you actually need Kubernetes.Read →
- The Observability HandbookSeeing inside production — monitoring vs observability, the three pillars (metrics, logs, traces) and what each answers, structured logging, metric types and the cardinality trap, distributed tracing, the golden signals and SLIs/SLOs/error budgets, alerting on symptoms not causes, and correlating all three during an incident.Read →
- The AI Security HandbookYour app now reads the internet and believes it. Why prompt injection is a design problem filters cannot solve, the lethal trifecta (private data + untrusted content + exfiltration), defense in depth from least privilege to output validation, where PII actually leaks (logs, embeddings, weights), MCP/model supply-chain trust, and the red-team eval suite that gates CI.Read →
- The Docker HandbookA container is a lie told by the kernel — namespaces (what a process sees) plus cgroups (what it uses), not a VM. Image layers and the cache-ordering rule that halves build times, multi-stage Dockerfiles that ship small, volumes vs bind mounts, name-based networking, Compose, and the handoff to Kubernetes.Read →