Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Forward Deployed Engineer / IaC for Customer Environments

FDE Track~17 min readUpdated Jul 2026

Practitioner

Their account, their cluster,
their change board

Infrastructure as code inside your own organisation is a solved problem with good defaults. Inside a customer’s account almost none of those defaults hold: you may not have permission to create a role, the VPC has no route to the internet, the state file cannot live in your bucket, and the person who inherits all of it has never used Terraform. The technology is the same. Everything around it is different.

The four questions that determine the architecture

Ask these before writing a line of HCL or YAML. Each answer eliminates half the design space, and getting one wrong late is a rewrite rather than an edit.

1. Whose account, and what can you do in it?

Full admin in a dedicated account is the easy case and is rare. More common: a shared account with a permission boundary, no ability to create IAM roles, and a landing zone with service control policies that silently forbid things. Ask for the SCP list and the permission boundary document on day one — the failure mode is discovering a prohibition at apply time, three weeks in.

2. Who holds the state?

Terraform state must live where they can reach it after you leave, and it contains secrets in plaintext by default. That means a bucket in their account, encrypted, versioned, with a lock table, and access limited to the small group who will operate this. State in your account is convenient for exactly as long as the engagement lasts.

3. Who runs it, and when?

You applying from a laptop is a phase-one answer that quietly becomes permanent. The target is their pipeline — their CI, their approval gates, their change window. Plan the migration to it explicitly, because a deployment only their departed vendor knows how to apply is a liability their platform team will eventually resolve by rebuilding it.

4. Where does the network actually go?

The single most consequential question. “Is there outbound internet access from these subnets?” If the answer is no, container images, package installs, model APIs, telemetry and certificate revocation checks all break — usually not at apply time but at first run, which is worse. See air-gapped deployment for the extreme end of this.

What a no-egress VPC actually breaks

The list is longer than people expect, and most items fail at runtime rather than at provisioning time.

What you assumed	What happens	The fix
Pull a container image	Timeout on first pod schedule	Mirror to their registry (ECR/ACR/Artifactory) and rewrite every image reference — including sidecars and init containers
`pip install` / `npm install` at build or start	Fails, sometimes only in the third environment	Vendor dependencies into the image; build outside, transfer the artefact in
Reach the S3 / storage API	Hangs — the API is public even for a private bucket	A gateway or interface VPC endpoint
Call a hosted model API	No route	PrivateLink where the provider offers it; otherwise a self-hosted model, which is a different project
Ship logs or metrics out	Silently drops, and you notice during the first incident	Local collector, retention in their environment, an export procedure they run
OCSP / CRL certificate checks	Slow TLS handshakes or hard failures nobody can explain	Configure explicitly; this one costs a day to diagnose the first time
NTP	Clock drift, then token validation failures that look like an auth bug	Their internal NTP source, set at the instance level

# the endpoint list for a typical no-egress AWS deployment
s3 (gateway) · ecr.api · ecr.dkr · logs · sts · secretsmanager · kms
ssm · ssmmessages · ec2messages # if you want Session Manager access
eks · elasticloadbalancing # if you are on EKS

# each is a resource, a security group and a private DNS decision.
# miss sts and IRSA fails with an error that names none of this.

→ Prove the network before you build on it

Deploy a trivial pod that curls every endpoint the real system will use and prints a pass/fail table. Half an hour of work, run on day one, and it converts a class of mysterious runtime failures into a list you can hand to their network team while you build something else. Keep it in the repo and run it after every environment change — it is the cheapest diagnostic you will ever write.

Structuring the code for someone else to own

The design goal is not elegance. It is that their platform engineer can read it in an hour.

# a layout that survives handover
infra/
  modules/ # reusable, versioned, the part you ship to every customer
    app/
    network/
    observability/
  envs/
    dev/ main.tf + terraform.tfvars # everything customer-specific lives here
    prod/ main.tf + terraform.tfvars
  PREREQS.md # what THEY must create before this runs
  RUNBOOK.md # apply, upgrade, roll back, destroy

# the rule: a new customer is a new envs/ directory and a tfvars file.
# if it is ever a fork of modules/, the model has broken.

Separate what you own from what they own

Their VPC, their cluster, their IdP and their KMS keys are inputs, not resources you manage. Take them as variables and use data sources. Creating a VPC inside your module means a terraform destroy can take out infrastructure that predates you — a genuinely unrecoverable mistake.

Write PREREQS.md before you write the module

The exact list of things their team must create, with the IAM policy JSON, the subnet requirements and the endpoint list included verbatim. This document is what their cloud team approves, and a specific request gets approved in days where a vague one gets scheduled for a meeting.

Version and pin everything

Provider versions, module versions, chart versions, image digests. In a customer environment “it worked last month” is a support ticket you will personally answer. Pin to digests rather than tags for images — a moved tag is an unreproducible deployment.

Make the module boring

No clever dynamic blocks, no deep for_each over computed maps, no hand-rolled abstractions. The reader is a competent engineer who did not write it and is looking at it because something is broken at 4pm on a Friday. Optimise for that reading, not for line count.

Keep Helm values thin and documented

One values file per environment, every key commented with what it does and what happens if you change it. Resist templating that requires understanding Go templates to read the output; helm template output should be inspectable by someone who does not know Helm.

Secrets you must not see

The correct number of customer credentials in your possession is zero, and getting there is a design decision.

Somebody will offer to email you a database password. Declining and giving them a better option is part of the job — and the better option is nearly always one they already have.

The pattern that works

Their secret store is the source of truth: Vault, Secrets Manager, Key Vault, Kubernetes secrets backed by one of those.
Terraform references secrets by ARN or path, never by value.
The workload reads at run time using workload identity — IRSA, managed identity, federation.
You have permission to reference, not to read.
Rotation is theirs, and nothing you built breaks when it happens.

Findings waiting to be written

A secret in terraform.tfvars, which is in git, which is in your GitHub organisation.
A secret in a Helm values file, base64 and therefore “encoded”.
A credential in your password manager two years after the engagement ended.
Terraform state in a bucket without encryption — state holds secrets in plaintext.
A shared service account whose credential three people have and nobody rotates.

→ The IAM policy you should write yourself

Do not let their cloud team guess what you need — they will over-grant to unblock you, then a security review will strip it back at the worst time. Write the policy: exact actions, exact resource ARNs, conditions where they apply. Include a one-paragraph justification per statement. It is the fastest approval you will get, and drafting it reliably reveals a permission you asked for and do not need. This is the same discipline as the least-privilege section of enterprise identity.

Handing it over so it survives

An IaC deliverable that only you can run is not a deliverable, it is a dependency. Four checks decide whether the handover is real, and they are all things you can verify rather than assert.

They apply it, with you watching

Not you applying while they watch. Their engineer, their credentials, their pipeline, a real change in a real environment. Everything that goes wrong during that hour is your remaining documentation debt, and you will find several.

A clean-room rebuild from the repo

Destroy a non-production environment and rebuild it from the repository alone, with no local files and no undocumented steps. This is the only honest test of whether the code is complete — and it always fails the first time, which is exactly why you run it before the handover rather than after.

The runbook covers the four verbs

Apply, upgrade, roll back, destroy — each with expected output and a named failure branch. Destroy matters more than people think: an environment nobody dares delete becomes an environment nobody dares change.

Ownership is written down

Which team owns the state, who approves changes, who is called when apply fails. Ambiguity here is how a deployment ends up owned by nobody, drifts for a year, and gets rebuilt by a platform team who never call you.

→ Drift is the quiet failure

Somebody will change something in the console during an incident — that is a reasonable thing to do at 2am. The failure is nobody reconciling it afterwards, so six months later terraform plan proposes destroying resources production depends on and everyone stops trusting the tool. Run plan on a schedule, alert on unexpected diffs, and treat drift as a normal operational event with a documented response rather than a moral failing.

Practise the hard part

Provisioning is the easy half

The hard half is the permission conversation, the security review and the handover. Those are practised elsewhere on the track.

Security Review Gauntlet Handoff & enablement Air-gapped & regulated POC → production hardening

The picture

The picture: what you ship, what they own, who applies

The ownership line is the design. Anything that existed before you arrived, or that they must keep after you leave, sits on their side as an input: you read it with a data source and never manage it, so a terraform destroy cannot touch it. Only the app and its observability are resources in your state, and that state lives in their bucket. The dashed box is the network fact that decides whether any of it runs.

By the numbers

Worked example: the endpoint plan for a 3-AZ no-egress EKS deployment

Take the page’s endpoint list and a cluster spread over three availability zones (illustrative). Gateway endpoints are route-table entries. Each interface endpoint puts one network interface, with one private IP, in every subnet you give it. Count what you are asking their network team for.

Step	Endpoints added	What breaks without them	Interface endpoints / IPs at 3 AZs
1	`s3` (gateway)	Image pulls too: ECR stores layers in S3	0 / 0
2	`ecr.api`, `ecr.dkr`	Registry auth and manifests: the first pod times out	2 / 6
3	`sts`	IRSA token exchange, with an error that names none of this	3 / 9
4	`logs`	Log shipping drops silently	4 / 12
5	`secretsmanager`, `kms`	Secrets read at run time, and decryption	6 / 18
6	`ssm`, `ssmmessages`, `ec2messages`	Session Manager access for you and them	9 / 27
7	`eks`, `elasticloadbalancing`	Cluster and load-balancer API calls	11 / 33
8	Network-proof pod	One pass/fail line per endpoint	12 checks

Step 1 is the one that costs a day. Teams add both ECR endpoints, still get image pull timeouts, and miss that the layers come from S3. Step 7’s total is the one to put in PREREQS.md: 11 interface endpoints, each with a security group allowing 443 from the workload and a private DNS decision, and 33 private IPs. Check the subnets before you ask. A /28 has 16 addresses and AWS reserves 5, leaving 11, so one /28 per AZ is used up entirely by the endpoints, with no room for anything else.

Trade-offs

Cost and trade-offs: the choices that decide who owns it

Here Time is the effort the choice takes now, Space is what it leaves behind after you go, and Why is the deciding argument.

Case	Time	Space	Why
State in your account	Zero setup	An orphaned dependency, with their secrets in plaintext in your bucket	Convenient for exactly as long as the engagement lasts.
State in their bucket, encrypted, with a lock	An afternoon, plus their approval	State they can reach and audit after you leave	The default. Anything else is a handover you will have to do later.
You apply from a laptop	Fastest in week one	A deployment only you know how to run	Fine for phase one, as long as the move to their CI is planned.
Their CI applies	Days to wire into their approvals and change windows	A deployment their process owns	The target. Their engineer, their credentials, their pipeline.
Fork the modules per customer	Quick for the first two customers	N copies drifting apart; every fix applied N times	If a customer needs a fork, a missing variable is the real problem.
Image tags vs digests	Tags are easier to read	A moved tag makes a rebuild run different code	Pin digests. The clean-room rebuild only proves something if the inputs cannot move.

In the wild

Where this shows up in production and interviews

Landing zones and SCPs
In an AWS Organizations landing zone, a service control policy can deny an action even to a principal with full admin in a member account. That is why the SCP list is a day-one request: an apply that fails three weeks in on a deny you never saw is a schedule problem, not a bug.
The customer security review
Your PREREQS.md and the IAM policy you wrote are what their security team approves. A request with exact actions, resource ARNs and a sentence of justification per statement is reviewed as a document. A vague one gets a meeting.
Scheduled drift checks
terraform plan -detailed-exitcode exits 0 when there are no changes, 1 on error and 2 when changes are pending. Run it on a schedule in their pipeline and alert on 2. That turns “someone changed it in the console at 2am” into a ticket rather than a surprise.
The interview version
“Deploy our product into a bank’s VPC with no internet access” is a common FDE design prompt. Lead with the four questions (whose account, who holds state, who runs it, where the network goes), then the endpoint list, then how you hand it over. The HCL is the least interesting part.

Self-check

Check yourself

You added the ecr.api and ecr.dkr endpoints, but pods still sit in ImagePullBackOff with timeouts. What is missing?

The S3 gateway endpoint. ECR serves manifests through its own endpoints, but the image layers are stored in S3, so the layer download has no route. It is a free route-table entry and the most common gap in a no-egress build.

Their SCP denies iam:CreateRole, and your module creates the workload’s IRSA role. What do you change?

Stop creating it. Take the role ARN as an input variable, and put the role (trust policy and exact permissions JSON) in PREREQS.md for their team to create. It moves to their side of the ownership line, which is where something governed by their SCPs belongs anyway.

After an incident, the scheduled plan wants to delete a security-group rule someone added in the console. Do you apply?

Not blindly. Production may now depend on that rule. Find out why it was added. If it should stay, put it in code (or import it) so the plan comes back clean. If not, remove it as a planned change. Either way drift gets a documented response, and the tool stays trusted.

The clean-room rebuild from the repo works, but the app behaves differently from production. Your images are referenced as app:1.4. What is the likeliest cause?

The tag moved. Someone re-pushed app:1.4, so the rebuild pulled different bytes under the same name. Reference images by digest (app@sha256:…) so the same repo always deploys the same code, and the rebuild proves what it claims to prove.

Frequently asked

Quick answers

Where should Terraform state live for a customer deployment?

In the customer’s account — an encrypted, versioned bucket with a lock table, access limited to the people who will operate the system. State contains secrets in plaintext by default, and it must remain reachable by the customer after the engagement ends. Keeping state in your own account is convenient for exactly as long as you are engaged and becomes an orphaned dependency afterwards.

What breaks in a no-egress VPC?

Container image pulls, package installs at build or start time, cloud storage API calls (the API endpoint is public even for a private bucket), hosted model APIs, log and metric shipping, OCSP or CRL certificate checks, and NTP. Most fail at runtime rather than at provisioning time. The fixes are a mirrored registry, vendored dependencies, VPC interface and gateway endpoints, PrivateLink where offered, and a local telemetry collector with an export procedure.

How should you handle customer secrets during a deployment?

You should end up holding none. Their secret store is the source of truth, Terraform references secrets by ARN or path rather than by value, and the workload reads them at run time using workload identity such as IRSA or managed identity. You get permission to reference, not to read, and rotation stays theirs. Secrets in tfvars, Helm values files or your password manager are all findings waiting to be written up.

How do you structure Terraform so a customer team can own it?

Reusable versioned modules that are identical across customers, plus a per-environment directory holding a tfvars file with everything customer-specific. Take their VPC, cluster, identity provider and KMS keys as inputs rather than managing them, so a destroy cannot remove infrastructure that predates you. Add a PREREQS.md listing exactly what their team must create, including the IAM policy JSON, and a runbook covering apply, upgrade, roll back and destroy.

How do you verify an infrastructure handover is real?

Four checks: their engineer applies a real change using their credentials and pipeline while you watch; a non-production environment is destroyed and rebuilt from the repository alone with no local files or undocumented steps; the runbook covers apply, upgrade, roll back and destroy with expected output and named failure branches; and ownership is written down — which team holds state, who approves changes, who is called when apply fails.

Receipts

Sources

Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.

Cited on this page

Anthropic — Forward Deployed Engineer, Applied AI (JD) — 40/30/30 split; MCP servers, sub-agents and agent skills as deliverables. jobs.menlovc.com/companies/anthropic/jobs/69674588-forward-deployed-engineer-applied-ai
Anthropic — Forward Deployed Engineer, Federal Civilian (JD) — air-gapped and IL-level deployment requirements. jobs.menlovc.com/companies/anthropic/jobs/76278352-forward-deployed-engineer-federal-civilian
Baseten — “What I learned as a forward deployed engineer at an AI startup” — www.baseten.co/blog/what-i-learned-as-a-forward-deployed-engineer-working-at-an-ai-startup/

Where this sits Customer-grade production Rung 2 of 11

What a demo has to survive to become a deployment: their cloud, their auditors, their identity provider, their on-call rota — and finally the handoff that ends your involvement.

Finished this one? 0 / 209 Handbooks done

NextFrom POC to Production: Hardening Next on the “Customer-grade production” ladder · rung 3 of 11

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

Forward Deployed Engineering