Their account, their cluster,
their change board
Infrastructure as code inside your own organisation is a solved problem with good defaults. Inside a customer’s account almost none of those defaults hold: you may not have permission to create a role, the VPC has no route to the internet, the state file cannot live in your bucket, and the person who inherits all of it has never used Terraform. The technology is the same. Everything around it is different.
The four questions that determine the architecture
Ask these before writing a line of HCL or YAML. Each answer eliminates half the design space, and getting one wrong late is a rewrite rather than an edit.
Full admin in a dedicated account is the easy case and is rare. More common: a shared account with a permission boundary, no ability to create IAM roles, and a landing zone with service control policies that silently forbid things. Ask for the SCP list and the permission boundary document on day one — the failure mode is discovering a prohibition at apply time, three weeks in.
Terraform state must live where they can reach it after you leave, and it contains secrets in plaintext by default. That means a bucket in their account, encrypted, versioned, with a lock table, and access limited to the small group who will operate this. State in your account is convenient for exactly as long as the engagement lasts.
You applying from a laptop is a phase-one answer that quietly becomes permanent. The target is their pipeline — their CI, their approval gates, their change window. Plan the migration to it explicitly, because a deployment only their departed vendor knows how to apply is a liability their platform team will eventually resolve by rebuilding it.
The single most consequential question. “Is there outbound internet access from these subnets?” If the answer is no, container images, package installs, model APIs, telemetry and certificate revocation checks all break — usually not at apply time but at first run, which is worse. See air-gapped deployment for the extreme end of this.
What a no-egress VPC actually breaks
The list is longer than people expect, and most items fail at runtime rather than at provisioning time.
| What you assumed | What happens | The fix |
|---|---|---|
| Pull a container image | Timeout on first pod schedule | Mirror to their registry (ECR/ACR/Artifactory) and rewrite every image reference — including sidecars and init containers |
pip install / npm install at build or start | Fails, sometimes only in the third environment | Vendor dependencies into the image; build outside, transfer the artefact in |
| Reach the S3 / storage API | Hangs — the API is public even for a private bucket | A gateway or interface VPC endpoint |
| Call a hosted model API | No route | PrivateLink where the provider offers it; otherwise a self-hosted model, which is a different project |
| Ship logs or metrics out | Silently drops, and you notice during the first incident | Local collector, retention in their environment, an export procedure they run |
| OCSP / CRL certificate checks | Slow TLS handshakes or hard failures nobody can explain | Configure explicitly; this one costs a day to diagnose the first time |
| NTP | Clock drift, then token validation failures that look like an auth bug | Their internal NTP source, set at the instance level |
s3 (gateway) · ecr.api · ecr.dkr · logs · sts · secretsmanager · kms
ssm · ssmmessages · ec2messages # if you want Session Manager access
eks · elasticloadbalancing # if you are on EKS
# each is a resource, a security group and a private DNS decision.
# miss sts and IRSA fails with an error that names none of this.
Deploy a trivial pod that curls every endpoint the real system will use and prints a pass/fail table. Half an hour of work, run on day one, and it converts a class of mysterious runtime failures into a list you can hand to their network team while you build something else. Keep it in the repo and run it after every environment change — it is the cheapest diagnostic you will ever write.
Structuring the code for someone else to own
The design goal is not elegance. It is that their platform engineer can read it in an hour.
infra/
modules/ # reusable, versioned, the part you ship to every customer
app/
network/
observability/
envs/
dev/ main.tf + terraform.tfvars # everything customer-specific lives here
prod/ main.tf + terraform.tfvars
PREREQS.md # what THEY must create before this runs
RUNBOOK.md # apply, upgrade, roll back, destroy
# the rule: a new customer is a new envs/ directory and a tfvars file.
# if it is ever a fork of modules/, the model has broken.
Separate what you own from what they own
Their VPC, their cluster, their IdP and their KMS keys are inputs, not resources you manage. Take them as variables and use data sources. Creating a VPC inside your module means a terraform destroy can take out infrastructure that predates you — a genuinely unrecoverable mistake.
Write PREREQS.md before you write the module
The exact list of things their team must create, with the IAM policy JSON, the subnet requirements and the endpoint list included verbatim. This document is what their cloud team approves, and a specific request gets approved in days where a vague one gets scheduled for a meeting.
Version and pin everything
Provider versions, module versions, chart versions, image digests. In a customer environment “it worked last month” is a support ticket you will personally answer. Pin to digests rather than tags for images — a moved tag is an unreproducible deployment.
Make the module boring
No clever dynamic blocks, no deep for_each over computed maps, no hand-rolled abstractions. The reader is a competent engineer who did not write it and is looking at it because something is broken at 4pm on a Friday. Optimise for that reading, not for line count.
Keep Helm values thin and documented
One values file per environment, every key commented with what it does and what happens if you change it. Resist templating that requires understanding Go templates to read the output; helm template output should be inspectable by someone who does not know Helm.
Secrets you must not see
The correct number of customer credentials in your possession is zero, and getting there is a design decision.
Somebody will offer to email you a database password. Declining and giving them a better option is part of the job — and the better option is nearly always one they already have.
The pattern that works
- Their secret store is the source of truth: Vault, Secrets Manager, Key Vault, Kubernetes secrets backed by one of those.
- Terraform references secrets by ARN or path, never by value.
- The workload reads at run time using workload identity — IRSA, managed identity, federation.
- You have permission to reference, not to read.
- Rotation is theirs, and nothing you built breaks when it happens.
Findings waiting to be written
- A secret in
terraform.tfvars, which is in git, which is in your GitHub organisation. - A secret in a Helm values file, base64 and therefore “encoded”.
- A credential in your password manager two years after the engagement ended.
- Terraform state in a bucket without encryption — state holds secrets in plaintext.
- A shared service account whose credential three people have and nobody rotates.
Do not let their cloud team guess what you need — they will over-grant to unblock you, then a security review will strip it back at the worst time. Write the policy: exact actions, exact resource ARNs, conditions where they apply. Include a one-paragraph justification per statement. It is the fastest approval you will get, and drafting it reliably reveals a permission you asked for and do not need. This is the same discipline as the least-privilege section of enterprise identity.
Handing it over so it survives
An IaC deliverable that only you can run is not a deliverable, it is a dependency. Four checks decide whether the handover is real, and they are all things you can verify rather than assert.
They apply it, with you watching
Not you applying while they watch. Their engineer, their credentials, their pipeline, a real change in a real environment. Everything that goes wrong during that hour is your remaining documentation debt, and you will find several.
A clean-room rebuild from the repo
Destroy a non-production environment and rebuild it from the repository alone, with no local files and no undocumented steps. This is the only honest test of whether the code is complete — and it always fails the first time, which is exactly why you run it before the handover rather than after.
The runbook covers the four verbs
Apply, upgrade, roll back, destroy — each with expected output and a named failure branch. Destroy matters more than people think: an environment nobody dares delete becomes an environment nobody dares change.
Ownership is written down
Which team owns the state, who approves changes, who is called when apply fails. Ambiguity here is how a deployment ends up owned by nobody, drifts for a year, and gets rebuilt by a platform team who never call you.
Somebody will change something in the console during an incident — that is a reasonable thing to do at 2am. The failure is nobody reconciling it afterwards, so six months later terraform plan proposes destroying resources production depends on and everyone stops trusting the tool. Run plan on a schedule, alert on unexpected diffs, and treat drift as a normal operational event with a documented response rather than a moral failing.
Provisioning is the easy half
The hard half is the permission conversation, the security review and the handover. Those are practised elsewhere on the track.
Quick answers
Where should Terraform state live for a customer deployment?
In the customer’s account — an encrypted, versioned bucket with a lock table, access limited to the people who will operate the system. State contains secrets in plaintext by default, and it must remain reachable by the customer after the engagement ends. Keeping state in your own account is convenient for exactly as long as you are engaged and becomes an orphaned dependency afterwards.
What breaks in a no-egress VPC?
Container image pulls, package installs at build or start time, cloud storage API calls (the API endpoint is public even for a private bucket), hosted model APIs, log and metric shipping, OCSP or CRL certificate checks, and NTP. Most fail at runtime rather than at provisioning time. The fixes are a mirrored registry, vendored dependencies, VPC interface and gateway endpoints, PrivateLink where offered, and a local telemetry collector with an export procedure.
How should you handle customer secrets during a deployment?
You should end up holding none. Their secret store is the source of truth, Terraform references secrets by ARN or path rather than by value, and the workload reads them at run time using workload identity such as IRSA or managed identity. You get permission to reference, not to read, and rotation stays theirs. Secrets in tfvars, Helm values files or your password manager are all findings waiting to be written up.
How do you structure Terraform so a customer team can own it?
Reusable versioned modules that are identical across customers, plus a per-environment directory holding a tfvars file with everything customer-specific. Take their VPC, cluster, identity provider and KMS keys as inputs rather than managing them, so a destroy cannot remove infrastructure that predates you. Add a PREREQS.md listing exactly what their team must create, including the IAM policy JSON, and a runbook covering apply, upgrade, roll back and destroy.
How do you verify an infrastructure handover is real?
Four checks: their engineer applies a real change using their credentials and pipeline while you watch; a non-production environment is destroyed and rebuilt from the repository alone with no local files or undocumented steps; the runbook covers apply, upgrade, roll back and destroy with expected output and named failure branches; and ownership is written down — which team holds state, who approves changes, who is called when apply fails.
Sources
Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.
Cited on this page
- Anthropic — Forward Deployed Engineer, Applied AI (JD) — 40/30/30 split; MCP servers, sub-agents and agent skills as deliverables. jobs.menlovc.com/companies/anthropic/jobs/69674588-forward-deployed-engineer-applied-ai
- Anthropic — Forward Deployed Engineer, Federal Civilian (JD) — air-gapped and IL-level deployment requirements. jobs.menlovc.com/companies/anthropic/jobs/76278352-forward-deployed-engineer-federal-civilian
- Baseten — “What I learned as a forward deployed engineer at an AI startup” — www.baseten.co/blog/what-i-learned-as-a-forward-deployed-engineer-working-at-an-ai-startup/
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Agentic AI Interview HandbookTwenty topics every senior AI engineer should be able to reason about live — from eval pipelines to reliability patterns for generative systems.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- 50 Angular Interview QuestionsA visual handbook covering components, change detection, RxJS, signals, routing, forms, performance, and testing — what interviewers actually probe for in senior Angular roles.Read →
- 50 Python Interview QuestionsFundamentals to advanced: data structures, OOP, iterators & generators, the GIL, asyncio, memory, testing, and the standard library — a visual walk through everything a Python interview touches.Read →
- 51 LLM Evals Interview QuestionsGolden sets, LLM-as-judge, regression testing, offline vs online evals, RAG evals, agent evals, red-teaming, and observability — demystified for interviews and production.Read →
Explore more from Vibe Engines
Get the next one in your inbox.
New handbooks, system-design walkthroughs, and tools — straight to your inbox. No spam, unsubscribe anytime.