The big idea
Why deploy in the customer’s walls?
A regulated customer — a bank, a hospital, a defense contractor — wants your AI product, but their data cannot leave their environment to hit a public API. Compliance, contracts, and sometimes law forbid it. A hosted endpoint, however good, is a non-starter for them.
Bring the model to the data: deploy the LLM inside the customer’s own environment (their VPC, their data center, or fully air-gapped). Right-size it to their hardware, serve it efficiently, run it with no network egress, and patch and observe it without anything leaving. It’s the deployment shape that turns a hard "no" into a signed deal.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · The skeleton
Run the model where the data is
The naive plan is your cloud endpoint — but the customer’s data can’t reach it. So the request path itself has to move: their apps must call a model that lives inside their network, not across the internet.
Stand up an Inference Gateway and Inference Server inside the customer’s environment, loading local Model Weights. To the customer’s apps it looks like the same API — but every byte of prompt and response stays in their walls.
Move the model to the data, not the data to the model: When data can’t travel, compute has to. The endpoint the app calls is identical; only its location changes — which is exactly what unblocks the regulated customer.
Step 2 · Make it fit
Right-size to their hardware
In the cloud you pick the GPU. On-prem, the customer already owns fixed, often modest hardware — and you can’t add more on demand. Ship the biggest full-precision model and it won’t load, or OOMs on the first long prompt.
Design decision: The customer has two mid-range GPUs with fixed VRAM. How do you choose what to deploy?
The call: Pick a model and precision (e.g. INT4/INT8 quantization) that fit their VRAM at acceptable latency. — Right-sizing is the on-prem skill: choose a model whose weights, quantized as needed, fit their VRAM with room for the KV cache, and validate latency on their actual hardware. Fit first, then maximize quality within that envelope.
Add a Right-Sizing step: pick the model and precision that fit the customer’s VRAM — a smaller and/or quantized (INT8/INT4) model — leaving headroom for the KV cache, and validate latency on their hardware before you commit.
Fit the hardware first, then maximize quality: On-prem the GPU budget is fixed and unexpandable. Model size and quantization are the dials you turn to fit it; quality is maximized inside that envelope, not above it.
Step 3 · Keep the GPUs busy
Serve efficiently on limited GPUs
A right-sized model that answers one request at a time wastes the very GPUs you fought to fit it on. On modest hardware you can’t brute-force throughput with more machines — you have to use what’s there well.
Serve through a runtime built for it: continuous batching packs concurrent requests through the GPU Pool, and a KV cache (with paged attention) makes each token cheap. You get usable throughput and latency from fixed hardware instead of one-at-a-time serving.
Utilization is the lever when you can’t add hardware: In the cloud you scale out; on-prem you scale up utilization. Continuous batching and a KV cache are how a fixed GPU pool serves many users at once.
Step 4 · Nothing leaves
No egress / air-gap
Everything so far runs locally — but a single outbound call breaks the promise. A telemetry beacon, an online license check, or a model tool that fetches a URL can carry prompts and outputs out of the customer’s environment. The requirement isn’t "encrypted egress" — it’s no egress.
Design decision: The customer requires that data never leave their network. What must the deployment guarantee?
The call: No outbound network calls at all — offline licensing, no telemetry beacon, no URL-fetching tools. — The guarantee is zero egress: block outbound traffic at the boundary, make licensing and config fully offline, and disable any model tool that can fetch a URL. If nothing can call out, nothing can leak — which is the exact requirement that justified running on-prem.
Enforce a No-Egress Boundary: deny outbound traffic, make licensing and configuration fully offline, and disable any tool or feature that can fetch a URL. Verify it — a locked-down customer will test that the box truly can’t call home.
No egress is the deliverable, not a setting: The deployment’s core promise is that data can’t leave. That means zero outbound calls — offline licensing, no beacons, no URL fetches — and proving it, because the customer will check.
Step 5 · Ship fixes without internet
Updates in a locked-down env
A model with no egress also can’t auto-update. But it still needs new model versions and security patches — and CVEs don’t wait. You need a way to get changes in that respects the same no-egress rule you just enforced.
Distribute Offline Updates as signed, verifiable bundles the customer’s Ops team imports on their schedule — weights, runtime, and patches together — with a clear rollback. Signing lets them trust the artifact without trusting a network connection to you.
Updates come in as signed artifacts, not live downloads: No egress cuts both ways: you can’t push, so you hand over signed bundles the customer imports and can roll back. Trust comes from the signature, not from a live channel.
Step 6 · See without exfiltrating
Local observability
You still need to know the deployment is healthy — latency, errors, GPU saturation — but you can’t stream metrics back to yourself without breaking no-egress. Flying blind isn’t acceptable either; a stuck GPU or a bad rollout has to be visible.
Collect Local Telemetry that stays in the customer’s environment on a dashboard they own. When you need to debug, the customer exports and shares exactly what they choose. You get observability; they keep control of every byte that leaves.
Observability the customer controls: Metrics live where the model lives. The customer owns the dashboard and decides what, if anything, is shared back — so you can operate without ever violating no-egress.
You did it
You just deployed an LLM in a customer’s environment.
- When data can’t leave, move the model to the data — deploy inside the customer’s walls.
- Right-size the model and precision to the fixed hardware they actually own; fit first.
- Serve with continuous batching + a KV cache to use limited GPUs well — utilization, not more boxes.
- Enforce no egress: offline licensing, no beacons, no URL fetches — and prove it.
- Ship updates as signed bundles the customer imports and can roll back — no auto-update.
- Keep telemetry local on a dashboard they own; share only by explicit export.