The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why deploy in the customer’s walls?
A regulated customer — a bank, a hospital, a defense contractor — wants your AI product, but their data cannot leave their environment to hit a public API. Compliance, contracts, and sometimes law forbid it. A hosted endpoint, however good, is a non-starter for them.
Bring the model to the data: deploy the LLM inside the customer’s own environment (their VPC, their data center, or fully air-gapped). Right-size it to their hardware, serve it efficiently, run it with no network egress, and patch and observe it without anything leaving. It’s the deployment shape that turns a hard "no" into a signed deal.
Step 1 · The skeleton
Run the model where the data is
The naive plan is your cloud endpoint — but the customer’s data can’t reach it. So the request path itself has to move: their apps must call a model that lives inside their network, not across the internet.
Stand up an Inference Gateway and Inference Server inside the customer’s environment, loading local Model Weights. To the customer’s apps it looks like the same API — but every byte of prompt and response stays in their walls.
Why this piece earns its place
The gateway is a seam you will never be on the other side of — and neither is anything else here, because the whole deployment sits behind walls you have no account on. That changes what debuggable means. A fault reaches you second-hand, described by someone reading a screen you cannot see, so the gateway has to answer the first diagnostic questions in one place and without a debugger: a health endpoint that prints the loaded model, its precision, the VRAM headroom and the bundle version, in output an operator can paste into a ticket. Anything that needs a flag only you know how to set is unreachable in practice. The gateway is also where identity stops being yours to design. There is no account system to build — it delegates to whatever provider they already run, and writes who prompted what to a local audit log, which a regulated buyer asks to see during procurement rather than after go-live. And the seam is what lets there be a second deployment at all. Each site pins its own version on its own schedule, so there is no single current release; the request and response shape at this boundary is the contract you have to keep compatible with the oldest instance still in the field.
What the new pieces do
- Customer Appsclient
- The customer’s own users and applications, calling the model from inside their network. They never talk to a public API — the whole point is that the request and the data never leave their environment.
- Inference Gatewaybackend
- The API surface that lives inside the customer’s environment. It looks like the cloud endpoint your app already calls, so integration is a config change — but every byte stays local.
- Inference Servercache
- The serving runtime (vLLM, TGI, llama.cpp) that loads the weights and answers requests. Continuous batching and a KV cache keep the limited GPUs busy so throughput is usable on modest hardware.
- Model Weightsdata
- The model files, stored on the customer’s own disk and loaded by the server. They never come from a live download at request time — they’re part of the signed bundle that was imported.
Step 2 · Make it fit
Right-size to their hardware
In the cloud you pick the GPU. On-prem, the customer already owns fixed, often modest hardware — and you can’t add more on demand. Ship the biggest full-precision model and it won’t load, or OOMs on the first long prompt.
The customer has two mid-range GPUs with fixed VRAM. How do you choose what to deploy?
The largest model in full precision won’t fit fixed, modest VRAM — it fails to load or OOMs on a long prompt. On-prem you can’t just provision a bigger GPU, so the biggest model is the wrong default: it has to fit the hardware first.
Sometimes true, but it’s slow, expensive, and often not an option in a locked-down data center. A good FDE first right-sizes to what they have — model choice plus quantization — and only escalates hardware if the requirement genuinely can’t be met.
Right-sizing is the on-prem skill: choose a model whose weights, quantized as needed, fit their VRAM with room for the KV cache, and validate latency on their actual hardware. Fit first, then maximize quality within that envelope.
Add a Right-Sizing step: pick the model and precision that fit the customer’s VRAM — a smaller and/or quantized (INT8/INT4) model — leaving headroom for the KV cache, and validate latency on their hardware before you commit.
Why this piece earns its place
Quantization is where this step is won, and the bill arrives somewhere you cannot look. Fit is settled in seconds — weights load or they do not. Quality loss is not evenly spread: it concentrates on code, on long-context reasoning, on languages and formats that were thin in whatever calibration set produced the quantized weights, which is precisely the tail a demo never reaches. So the comparison has to be scoped to what can actually run in the room. The full-precision arm the arithmetic above already ruled out cannot be the baseline; the candidates are the ones that fit — this model at INT8 against it at INT4, or a smaller model at higher precision against a larger one at lower. Their prompts cannot come to you, so their people run the comparison and their people score it, which means what you are really writing is a rubric and a harness, not a result. INT4 is also not one format, and the kernels that make a given scheme fast exist per GPU generation and driver version, so a mixed fleet can leave you serving different precisions on different boxes in the same rack. Leave headroom now, while headroom is free: acceptance is signed against one model at one precision, and changing it afterwards reopens that whole cycle.
- fixed VRAMnot elastic
- INT4 / INT8quantize to fit
- validateon their box
What the new pieces do
- Right-Sizingservice
- Chooses the model and precision that fit the hardware the customer actually has — a smaller model and/or quantized weights (INT8/INT4) so it loads in their VRAM and hits an acceptable latency.
Back of the envelope
- 70B × 2 bytes/param = 140 GB of weights at FP16
- more than any single card holds — this is the “won’t load” case
- INT4 is ½ byte/param ⇒ 70B × 0.5 = 35 GB
- same model, a quarter of the VRAM — the trade is quality, not capability
- KV cache ≈ 0.33 MB/token (2 × 80 layers × 8 KV heads × 128 dim × 2 bytes) ⇒ ≈2.7 GB per 8K request
- the weights load fine and it still OOMs on the first long prompt — this is why
- 80 GB card − 35 GB weights = 45 GB ⇒ 45 ÷ 2.7 ≈ 16 concurrent 8K requests
- the batch size is not a guess — it falls out of the arithmetic, and Step 3 tunes it
- then benchmark on their hardware
- the numbers above size the box; they don’t predict its throughput
Step 3 · Keep the GPUs busy
Serve efficiently on limited GPUs
A right-sized model that answers one request at a time wastes the very GPUs you fought to fit it on. On modest hardware you can’t brute-force throughput with more machines — you have to use what’s there well.
Serve through a runtime built for it: continuous batching packs concurrent requests through the GPU Pool, and a KV cache (with paged attention) makes each token cheap. You get usable throughput and latency from fixed hardware instead of one-at-a-time serving.
Why this piece earns its place
Continuous batching does not create capacity. It converts a capacity problem into a queueing problem, and queueing is the part that shows up in production: mean throughput improves while the tail gets worse, and there is no autoscaler here to absorb a burst. The pool is the pool. So the first thing this step owes the design is admission control — a queue-depth ceiling that refuses fast and says so, because a caller left hanging is worse for the customer app than an explicit rejection it can retry. The second is less obvious. When the KV cache fills, the scheduler does not politely stop admitting; it preempts, evicting a running sequence and recomputing its tokens later, so a burst costs you work you had already paid for and a saturated pool degrades faster than a utilization graph suggests. And because this is one shared pool inside one organisation, the workloads collide: an interactive caller wants a short time to first token, a nightly job wants tokens per second, and batching them together makes both worse than either alone. Separate queues, or separate hours. The compensation is that you know the organisation — its working day, its quiet night — which no multi-tenant service does, so the heavy work can be scheduled rather than bought.
- continuousbatching
- KV cachecheap tokens
- fixed poolwell used
What the new pieces do
- GPU Pooldata
- The physical accelerators the customer owns — a fixed, often modest amount of VRAM you cannot expand on demand. Serving choices (batch size, context length, quantization) are all bounded by this budget.
Back of the envelope
- continuous batching
- many requests, full GPU
- paged KV cache
- long contexts without waste
- tune batch + context
- to the VRAM budget
Step 4 · Nothing leaves
No egress / air-gap
Everything so far runs locally — but a single outbound call breaks the promise. A telemetry beacon, an online license check, or a model tool that fetches a URL can carry prompts and outputs out of the customer’s environment. The requirement isn’t "encrypted egress" — it’s no egress.
The customer requires that data never leave their network. What must the deployment guarantee?
The guarantee is zero egress: block outbound traffic at the boundary, make licensing and config fully offline, and disable any model tool that can fetch a URL. If nothing can call out, nothing can leak — which is the exact requirement that justified running on-prem.
Encryption protects data in transit, but the requirement is that the data doesn’t leave at all. An encrypted beacon still exfiltrates prompts/outputs to a third party — a compliance failure regardless of TLS. No egress means no outbound call, period.
"Anonymized" still means data crosses the boundary, and prompts are notoriously hard to truly anonymize. For a customer whose whole reason for going on-prem is that nothing leaves, any outbound flow — even aggregated — breaks the deal. Keep telemetry local (Step 6).
Enforce a No-Egress Boundary: deny outbound traffic, make licensing and configuration fully offline, and disable any tool or feature that can fetch a URL. Verify it — a locked-down customer will test that the box truly can’t call home.
Why this piece earns its place
Almost nobody writes the outbound call themselves. A dependency does, on import, by default — a crash reporter that ships enabled, a TLS stack fetching a revocation list, a package index refresh, a container runtime pulling a layer it does not have locally. So the work is an inventory of what every transitive dependency does to the network, and verification only means anything if it is continuous: the whole stack brought up with the interface down, in CI, on every build, rather than once at installation on a machine that still had a route out. The detail that bites afterwards is the shape of the failure. A default-deny policy drops packets rather than rejecting them, so an attempted phone-home never raises an error anyone would grep for — it surfaces as a request that hangs until something times out, often a name lookup first, on a code path with no obvious relationship to the call. Set short connect timeouts everywhere, and ask for denied flows to be logged at the boundary, because their firewall log is the only place the truth gets written and you will be reading it over someone’s shoulder. Test a cold restart too, not just a running box: a deployment that has been up for months can fail to come back because something it needs at start has been quietly resolving a hostname since install day.
- denyoutbound
- offlinelicensing
- noURL fetches
What the new pieces do
- No-Egress Boundaryservice
- Enforces that nothing calls out: no telemetry beacon, no online license check, no model tool that fetches a URL. Licensing and config are offline. Blocking outbound traffic is the requirement that justifies the whole deployment.
Back of the envelope
- block outbound at the boundary
- default-deny egress
- offline licensing + config
- no phone-home
- disable URL-fetching tools
- no covert exfil path
Step 5 · Ship fixes without internet
Updates in a locked-down env
A model with no egress also can’t auto-update. But it still needs new model versions and security patches — and CVEs don’t wait. You need a way to get changes in that respects the same no-egress rule you just enforced.
Distribute Offline Updates as signed, verifiable bundles the customer’s Ops team imports on their schedule — weights, runtime, and patches together — with a clear rollback. Signing lets them trust the artifact without trusting a network connection to you.
Why this piece earns its place
Signing settles authenticity and creates a key problem. With no network there is no revocation check, so a compromised signing key cannot be withdrawn from a deployed instance — rotation has to travel the way everything else does, with bundle N carrying the key that will verify bundle N+1, and the first trust root established by hand at install. Rolling back is the other thing that is not what it looks like. Reverting a binary is not reverting a deployment: if the new runtime migrated anything on disk — a cache format, an index layout, a config schema — the old version comes back up reading files it does not understand, and they discover that mid-window, under their own change control, with you unreachable. Either ship migrations that reverse, or keep the previous version able to read the new format. Underneath both sits a constraint worth naming out loud: their maintenance window sets your CVE response time, not your release cadence. You cannot patch quickly, so the real defence is having less to patch — a minimal image, fewer packages, fewer lines in the scanner report you are contractually obliged to answer within a fixed number of days.
- signedbundles
- customerimports
- cleanrollback
What the new pieces do
- Customer Opsclient
- The customer’s own operations and security team. After handoff they run the box: apply updates you ship, watch health, and hold the keys. Your deployment has to be operable by them, not only by you.
- Offline Updatesservice
- Ships new model versions and security patches into a locked-down environment without internet — signed, verifiable bundles the customer’s team imports and rolls back, instead of an auto-updater phoning home.
Back of the envelope
- signed, verifiable artifacts
- trust without a network
- import on the customer’s schedule
- their change control
- versioned with rollback
- a bad update reverts
Step 6 · See without exfiltrating
Local observability
You still need to know the deployment is healthy — latency, errors, GPU saturation — but you can’t stream metrics back to yourself without breaking no-egress. Flying blind isn’t acceptable either; a stuck GPU or a bad rollout has to be visible.
Collect Local Telemetry that stays in the customer’s environment on a dashboard they own. When you need to debug, the customer exports and shares exactly what they choose. You get observability; they keep control of every byte that leaves.
Why this piece earns its place
The signal you most want is the one that can never leave. When an answer is wrong, the prompt and the output are what explain it, and those are the exact bytes this deployment exists to keep in place. So it only works if the telemetry is content-free by construction: shapes, not contents. Token counts, sequence lengths, finish reasons, queue depth, preemption counts, error codes, model and bundle version. Redacting after the fact is the thing that fails, because eventually someone pastes a stack trace carrying a prompt into a ticket. Design the exportable slice to be safe on its face, publish its schema, and have it approved once — then sharing is a file their team sends, not a fresh legal review each time. Alerting is the harder half, because it is not yours. Thresholds have to arrive in the bundle with defaults already set: nobody on their side knows what a bad p99 looks like for this model on this hardware, and the person who does know is not on the rotation. There is also no fleet to compare against — one site, one configuration, no other instance quietly telling you what normal is. The acceptance benchmark is the entire baseline, so if nobody wrote those numbers down on install day, the first incident has no before.
- metricsstay local
- theyown it
- exporton request
What the new pieces do
- Local Telemetryservice
- Health, latency, and error metrics collected and stored inside the customer’s environment, on a dashboard they own. You get observability without any data leaving — the customer decides what, if anything, they share back.
Back of the envelope
- local metrics + logs
- health without egress
- customer-owned dashboard
- they hold the data
- share by explicit export
- never an auto-stream
You did it
You just deployed an LLM in a customer’s environment.
Everything you assembled, in order
- When data can’t leave, move the model to the data — deploy inside the customer’s walls.
- Right-size the model and precision to the fixed hardware they actually own; fit first.
- Serve with continuous batching + a KV cache to use limited GPUs well — utilization, not more boxes.
- Enforce no egress: offline licensing, no beacons, no URL fetches — and prove it.
- Ship updates as signed bundles the customer imports and can roll back — no auto-update.
- Keep telemetry local on a dashboard they own; share only by explicit export.