No internet, no telemetry,
no second chance
Some of the highest-value forward deployed work happens in environments where none of your normal tools reach: no package registry, no model API, no logs leaving the building, and a deploy window that opens once a fortnight. Anthropic’s federal FDE role names air-gapped and impact-level deployment directly.29 This is what the work actually involves, and why engineers who can do it are scarce.
The spectrum, because “air-gapped” is used loosely
Customers say “air-gapped” to mean five different things, and the difference decides your entire architecture. Establish which one you are in during discovery, with a concrete question: can a machine in this environment reach a public IP address, and if so, through what?
| Mode | What is actually true | What it costs you |
|---|---|---|
| Their cloud tenancy | Full internet, their account, their controls | Almost nothing architecturally. This is normal work. |
| Private VPC, no public ingress | Outbound egress usually allowed, inbound blocked | Bastion access, PrivateLink endpoints, a VPN conversation. |
| No-egress VPC | Nothing reaches the internet, including your package installs | Mirrored registries, vendored dependencies, no external model API. A real re-architecture. |
| On-prem, disconnected | Their datacentre, updates arrive by an approved transfer process | Everything above, plus their hardware, their OS baseline, their change board. |
| True air gap | Physically separated network; data crosses on removable media through a formal review | Weeks of latency on every change. Your debugging loop is the constraint, not your code. |
“When you deploy an update to an existing system in this environment today, walk me through what happens and how long it takes.” Ask an operator, not an architect. The answer — a fortnightly window, a signed media transfer, a two-week security scan — is your actual iteration speed, and every estimate you give should be built on it rather than on how fast you can write the code.
The compliance vocabulary, for engineers
You do not need to be a compliance specialist. You do need to know what the words constrain.
An authorisation programme for cloud services sold to US federal agencies, with baselines at Low, Moderate and High reflecting the impact of a breach. Practical consequence for an engineer: the service must run inside an authorised boundary, every component in that boundary is inventoried, and changes go through a documented process. A dependency you add is a boundary change, not a pull request.
A parallel scale for US Department of Defense data. IL2 is public/non-critical. IL4 is controlled unclassified information. IL5 raises the bar again, typically requiring dedicated infrastructure and US-person access controls. IL6 covers classified up to SECRET and lives on separate networks entirely. Federal FDE roles reference these levels directly.29 The engineering consequence is a ladder: as the level rises, shared infrastructure, remote access and outbound telemetry each disappear in turn.
SOC 2 Type II — an audited report on controls over a period; what most enterprise buyers ask for first. ISO 27001 — the international management-system standard, common outside the US. HIPAA — US health data, with a business associate agreement that assigns you real liability. GDPR and data residency — where data may physically live, which quietly rules out a lot of architectures. CMMC — the defence supply chain’s certification of the same ideas.
All of these frameworks ask the same three questions in different vocabularies: who can reach the data, can you prove what happened, and can you show the control worked over time. Identity, audit logging and evidence — the subjects of enterprise identity. Build those three well and most of a questionnaire answers itself; build them late and every framework costs you separately.
Getting model weights into a disconnected network
The step people underestimate, because in every other context it is a download.
An open-weights model of useful size is tens to hundreds of gigabytes. Moving that across an approved boundary is a logistics exercise with a security review attached, and it is normally the long pole in the first delivery.
1. licence confirm the weights may be used in this environment, at this level
2. acquire download once, in a clean staging environment you control
3. verify checksums against the publisher — and record them; you will be asked
4. scan malware + supply-chain scan of weights, container images and deps
5. package one artefact: weights + runtime + config + manifest + hashes
6. transfer approved media or a one-way diode, with a signed manifest
7. re-verify checksums again inside the boundary, before anything runs
8. register the artefact enters their inventory as a controlled component
# elapsed time for step 6 alone, in a real programme: days to weeks
Package once, correctly
- Everything in one signed artefact — weights, runtime, model config, tokeniser, dependencies.
- A manifest listing every file with its hash and its licence.
- A version string that is meaningful to their change board, not a git SHA.
- Offline install instructions that assume no registry, no DNS, and an operator who has not met you.
- A rollback artefact, transferred at the same time. There will be no second transfer this week.
Discovered the hard way
- A container image that pulls a base layer at run time. Nothing pulls anything in there.
- A Python package that compiles from source and needs a toolchain nobody installed.
- A model runtime that phones home for a licence check on start-up.
- Hardware that does not match: different GPU generation, different driver, different quantisation support.
- Forgetting the tokeniser, or shipping a config that references a HuggingFace path.
Before any transfer, install your package on a machine with networking physically disabled. It takes an afternoon and it catches nearly every failure above. The alternative is discovering the missing base layer inside a facility where you cannot download anything, cannot take your laptop in, and cannot try again until the next window.
Updates without an update channel
Every convenience of continuous delivery is gone. What replaces it is a slower loop that must be made reliable, because you get few attempts and each one is expensive.
Batch on their cadence, not yours
If the window is fortnightly, your release train is fortnightly. Bundle changes, test the bundle as a unit, and stop thinking in individual commits. A change that misses the window waits, so scope each bundle to what you are confident about rather than what is finished.
Make configuration a separate, faster path
Prompts, thresholds, routing rules and feature flags should be data an authorised operator can change in the environment — not code requiring a full transfer. This single design decision is the difference between tuning a system in a day and tuning it in a month, and it is much harder to retrofit than to build.
Ship diagnostics with the product
You will not be able to attach a debugger, and often cannot see a log. Build a self-test command that exercises every dependency and prints a readable report, and a redaction-safe bundle an operator can produce and pass to you through the approved channel. Design what goes in that bundle deliberately — it is your only window.
Make rollback a first-class artefact
Versioned, transferred alongside the update, tested. In a connected environment rollback is a deploy command; here it is a physical process, and if the rollback package is not already inside the boundary it does not exist.
Write for an operator who cannot ask you a question
Runbooks in these environments are read by someone with no ability to phone a friend. Every step deterministic, every expected output shown, every failure branch named. This is enablement under the hardest possible constraint.
Evals. The loop everyone relies on — observe production, collect failures, improve — assumes you can see production. You cannot. The substitute is a golden set that lives inside the boundary, run by their operators against every release, with only the aggregate scores coming back out through the approved channel. Design that at the start; retrofitting an eval harness into a disconnected environment is far harder than building one. See evals for client work for the harness itself.
Working inside the constraints
The environment shapes the day, not just the architecture.
What is genuinely different
- You may not bring a phone or a laptop into the facility. Your notes are paper.
- You cannot search the internet mid-problem. Bring documentation in.
- Access clearance can take months and gates the work entirely.
- You may be unable to discuss specifics of the problem with your own colleagues.
- Pairing with a cleared operator often replaces pairing with an engineer.
Habits that make it survivable
- Prepare an offline kit: docs, dependency mirrors, reference implementations, a working local model.
- Write hypotheses down before each session — you get one experiment run, not twenty.
- Make the system explain itself, because you cannot inspect it from outside.
- Over-communicate outside the boundary, so the people who cannot see the work still trust it.
- Keep a decision log. Six months later nobody will remember why a threshold is 0.7.
This work pays well and is genuinely scarce, and it has real costs the recruiting conversation tends to skip: clearance processes that constrain travel and personal life, work you cannot put in a portfolio or describe in an interview, on-site requirements that are not negotiable, and skills that are deep but narrow. It suits some people enormously. Decide with the trade-offs visible — is forward deployed engineering worth it covers the wider version of this question.
Regulated deployments are won and lost in the review
Quick answers
What does air-gapped actually mean in an AI deployment?
It is used loosely for five different modes: their cloud tenancy with full internet; a private VPC with no public ingress but outbound egress; a no-egress VPC where nothing reaches the internet including package installs; on-premise and disconnected, where updates arrive through an approved transfer process; and a true air gap where networks are physically separated and data crosses on removable media. Establish which one during discovery by asking whether any machine can reach a public IP address.
What are FedRAMP and DoD impact levels?
FedRAMP is the US federal cloud authorisation programme with Low, Moderate and High baselines reflecting breach impact. DoD Impact Levels are a parallel scale for defence data: IL2 for public and non-critical, IL4 for controlled unclassified information, IL5 typically requiring dedicated infrastructure and US-person access controls, and IL6 for classified up to SECRET on separate networks. As the level rises, shared infrastructure, remote access and outbound telemetry each disappear in turn.
How do model weights get into an air-gapped environment?
Confirm licensing for that environment, download once in a clean staging area, verify checksums against the publisher and record them, scan the weights and images, package everything into one signed artefact with a manifest, transfer via approved media or a one-way diode, re-verify checksums inside the boundary before anything runs, and register the artefact in their inventory. Transfer alone commonly takes days to weeks. Always transfer a tested rollback artefact at the same time.
How do you ship updates when there is no update channel?
Batch releases onto the customer’s change window rather than your own cadence, and make configuration — prompts, thresholds, routing rules, feature flags — a separate faster path that an authorised operator can change in place without a full transfer. Ship a self-test command and a redaction-safe diagnostic bundle, treat rollback as a first-class artefact transferred alongside the update, and write runbooks for an operator who cannot ask you a question.
What breaks first in a disconnected AI deployment?
Evaluation. The normal loop of observing production, collecting failures and improving assumes you can see production, and you cannot. The substitute is a golden set that lives inside the boundary, run by the customer’s operators against every release, with only aggregate scores returning through the approved channel. Design this at the start — retrofitting an eval harness into a disconnected environment is substantially harder than building one.
Sources
Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.
Cited on this page
- Anthropic — Forward Deployed Engineer, Applied AI (JD) — 40/30/30 split; MCP servers, sub-agents and agent skills as deliverables. jobs.menlovc.com/companies/anthropic/jobs/69674588-forward-deployed-engineer-applied-ai
- Anthropic — Forward Deployed Engineer, Federal Civilian (JD) — air-gapped and IL-level deployment requirements. jobs.menlovc.com/companies/anthropic/jobs/76278352-forward-deployed-engineer-federal-civilian
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Agentic AI Interview HandbookTwenty topics every senior AI engineer should be able to reason about live — from eval pipelines to reliability patterns for generative systems.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- 50 Angular Interview QuestionsA visual handbook covering components, change detection, RxJS, signals, routing, forms, performance, and testing — what interviewers actually probe for in senior Angular roles.Read →
- 50 Python Interview QuestionsFundamentals to advanced: data structures, OOP, iterators & generators, the GIL, asyncio, memory, testing, and the standard library — a visual walk through everything a Python interview touches.Read →
- 51 LLM Evals Interview QuestionsGolden sets, LLM-as-judge, regression testing, offline vs online evals, RAG evals, agent evals, red-teaming, and observability — demystified for interviews and production.Read →
Explore more from Vibe Engines
Get the next one in your inbox.
New handbooks, system-design walkthroughs, and tools — straight to your inbox. No spam, unsubscribe anytime.