The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Shared infrastructure, zero cross-tenant leakage
Serving many separate customer organizations from one shared platform is efficient — until tenant A's private data ends up retrieved into tenant B's answer, or tenant C's traffic spike makes every other tenant's product feel broken. Multi-tenancy done right is invisible to every tenant; done wrong, it's a data breach or an outage waiting to happen.
We'll build isolation as a first-class, multi-layer property — not application-level trust — plus the operational realities of running one platform for many customers: customization without duplication, noisy-neighbor prevention, cost attribution, and compliance-driven data residency.
What the new pieces do
- Tenant Userclient
- An end user at one customer organization, whose data and queries must never be visible to any other tenant sharing the same platform.
Step 1 · The baseline
One shared config, one shared index
Simplest version: every tenant's documents go into one shared vector index, and everyone uses the same prompt and configuration. What's the danger, specifically for a retrieval-augmented system?
All tenants' documents share one unscoped vector index. What's the specific danger?
Storage capacity is a real, separate scaling concern, but it's not the IMMEDIATE danger — the immediate danger is far more serious than running out of disk.
Without tenant scoping enforced on retrieval, similarity search has no concept of "whose data this is" — it just finds the closest-matching vectors regardless of tenant, meaning one tenant's private, confidential documents can genuinely surface in another tenant's query results. For a B2B SaaS product, this is often the single most catastrophic possible failure.
Quality degradation from a larger shared index is a real but comparatively minor concern next to the much more severe risk of one tenant's confidential data literally appearing in another tenant's results.
An unscoped shared index risks one tenant's private documents surfacing directly in another tenant's retrieval results — a severe data breach, not a minor quality issue. Tenant isolation has to be enforced at the data layer, from the very first design decision, not treated as an afterthought.
Why this piece earns its place
The decision hiding under "one shared index" is a physical layout choice, and it is expensive to reverse. One index with a tenant field on every vector keeps memory in one place and one recall profile, but the filter has to be applied inside the search: applied to the neighbors after they come back, it can return nothing at all while that tenant’s own matching documents sit just outside the fetched window. An index per tenant makes the boundary structural instead, at the price of a fixed memory and build cost each, paid in full by your smallest customer. Neither shape is free to walk back later — moving one tenant across is a rebuild of their vectors while their users are still querying them. And a shared index shares its writes: one tenant loading their entire document history reshapes the neighborhood every other tenant searches and lengthens the rebuild everyone waits on. Size it for the largest corpus the plan allows a tenant to bring, not the one the median tenant actually brings.
What the new pieces do
- Tenant-Aware APIgateway
- Every request carries a tenant id that must be enforced at this layer AND every layer downstream — never trusted from just one place.
- Tenant-Isolated Datadata
- Documents, embeddings, and history — starts shared and unscoped; becomes independently tenant-checked at every single query.
Step 2 · Enforce isolation at the data layer
Every query, independently scoped
Fix Step 1: every retrieval and every data access must be scoped to exactly one tenant. Should this scoping be trusted to the application layer alone?
Tenant scoping needs to be enforced somewhere. Is trusting the application/API layer alone sufficient?
Trusting a SINGLE layer means any bug in that one layer's logic — a missed check, a copy-paste error, an edge case in how a tenant id gets threaded through — becomes a full cross-tenant data breach with no independent backstop.
Independent, redundant enforcement at the data layer means even if the API layer's own tenant check has a bug, the data store's OWN check still blocks the cross-tenant access — a genuine defense-in-depth property that a single point of enforcement can never provide.
Making tenant isolation opt-in treats a fundamental correctness guarantee as an optional add-on — every tenant's data needs to be protected from every other tenant by default, not as a premium feature some tenants might not select.
Enforce tenant scoping independently at the data layer, not just trusted from the API — every query to Tenant-Isolated Data carries and verifies a tenant id, checked by the store itself. This gives genuine defense-in-depth: a bug at any single layer doesn't automatically mean a full cross-tenant breach.
Why this piece earns its place
The check has to fail closed, and most data layers default to the opposite: a query built without a tenant predicate returns every row rather than raising. So what you want is not a WHERE tenant_id = ? added at each call site — that is the single layer again wearing a different hat — but a store that refuses a query carrying no tenant identity at all. Row-level security with the tenant set per session is the usual shape, and it brings its own trap: pooled connections outlive requests, so a session variable left set by the previous checkout is inherited by the next one. Set it inside the transaction, clear it on release, and exercise the pool under concurrency, because that bug only appears under load. The other cost is organizational. Once the store demands an identity, every background job, every analytics query and every support tool needs one too, and those are the paths that quietly earn an exemption. Keep exemptions as a short, named list of service identities rather than a branch that skips the check, because a list can be reviewed and a branch cannot.
Step 3 · Customization without duplicated infrastructure
Different prompts, one shared runtime
Tenants want different system prompts, different knowledge-base scopes, maybe different feature sets. Does each tenant need its own separately-deployed copy of the whole application stack?
Tenants want meaningfully different behavior. Does each need a fully separate deployed stack?
Deploying an entire separate stack per tenant doesn't scale operationally past a small number of customers — every code update, bug fix, or new feature would need to be deployed N times, once per tenant, which becomes unmanageable quickly.
A shared runtime with tenant-scoped CONFIGURATION resolved per request gives each tenant meaningfully different behavior — different prompts, different feature flags, different retrieval scope — without needing to deploy or maintain a separate copy of the actual application for each one. Updates and fixes ship once, to everyone, automatically.
No customization at all fails real B2B SaaS requirements — most enterprise customers reasonably expect some degree of tailoring (their own knowledge base, their own tone/prompt) that a one-size-fits-all product can't provide.
Use one shared runtime with Per-Tenant Config resolved at request time — prompts, retrieval scope, and feature flags differ per tenant, while the underlying application code and infrastructure remain shared. Tenant admins configure their own instance through this layer, without needing platform engineering to deploy anything separately.
Why this piece earns its place
Making behavior data rather than code means a tenant admin now performs something very like a deploy — no review, no staging, no rollback — every time they edit a prompt. Treat the config store as versioned and append-only for that reason, and stamp the resolved config version onto each request record beside the tenant id. Without it, "the product started answering strangely at four" is unanswerable, because the config that produced those answers has already been overwritten by the one you are reading. The hot path needs care too. A lookup per request makes the config store a hard dependency of every answer, so it gets cached, and the cache turns an admin’s correction into something that takes effect after a delay rather than at once — which they will report as a bug unless the interface says so plainly. Per-tenant flags also quietly multiply what you ship: there is no longer one product under test but one per combination of flags, and CI only covers the combinations somebody wrote down. Keep the flag set small, and delete a flag once its rollout is done.
What the new pieces do
- Tenant Adminclient
- Configures their organization's instance — prompts, knowledge base, quotas — without needing platform engineering involvement.
- Per-Tenant Configdata
- Each tenant's customization — system prompts, retrieval scope, feature flags — resolved at request time from one shared runtime.
Step 4 · Prevent noisy neighbors
One tenant's spike shouldn't degrade everyone else
All tenants share the same underlying infrastructure. One tenant suddenly sends 50x their normal traffic. Should that be allowed to affect other tenants' latency or availability?
One tenant spikes traffic dramatically on shared infrastructure. Should other tenants feel the effect?
Accepting cross-tenant impact as "unavoidable" abandons a solvable problem — per-tenant quotas are specifically designed to prevent exactly this, so it isn't actually unavoidable, just unaddressed without them.
Per-tenant quotas contain the blast radius of any single tenant's traffic pattern to that tenant alone — the spiking tenant experiences their own throttling, while the shared infrastructure's capacity for every other tenant stays exactly as available as before.
Full suspension is a disproportionate response to a legitimate traffic spike (which might just mean the tenant's own product is having a busy day) — throttling to a fair quota, not shutting the tenant out entirely, is the appropriate response.
Add a Quota Enforcer: per-tenant rate limits and resource quotas mean a traffic spike from one tenant is contained to throttling at THEIR boundary, with zero impact on the latency or availability every other tenant experiences on the same shared infrastructure.
Why this piece earns its place
The unit you meter decides whether any of this works. Requests per second is the reflex and the wrong dial for an inference product, because one request can be a short lookup and the next a long document summarization — a tenant comfortably inside an RPS limit can still hold most of the shared model capacity. Meter the thing that actually runs out: tokens over a window, or concurrent in-flight inference slots. The enforcer then needs shared counter state on the hot path of every request, so decide explicitly what it does when that state is unreachable. Fail closed and a counter outage has become a platform outage. Fail open and a spike arriving inside that window is unbounded — on a platform that bills what it serves, that is attributable spend nobody authorized, arriving in the one window when the protection is off. Neither works as the only answer, which is the case for a local allowance each instance can enforce alone: loose, wrong at the edges, but bounded. The last call is queue or reject. Queuing a throttled tenant reads as friendlier, but held connections and buffered requests are themselves shared capacity, and a deep enough queue lets the neighbor back in through another door.
- per-tenant quotacontains the blast radius
- spiking tenantthrottled at their own limit
- every other tenantunaffected
What the new pieces do
- Quota Enforcerservice
- Enforces per-tenant rate limits and resource quotas so one tenant's traffic can't degrade every other tenant's experience.
Step 5 · Attribute cost per tenant
Know exactly what each tenant actually costs
Tenants have vastly different usage patterns — some send a handful of requests a day, others thousands. Should pricing be flat regardless of actual usage?
Tenant usage varies enormously. Should billing be a flat rate regardless of actual consumption?
Flat pricing across wildly different usage levels either significantly overcharges light users (who may churn, feeling it's not worth the cost) or loses real money on heavy users (whose actual LLM/infra cost far exceeds what they're paying) — neither is sustainable at scale.
Precise per-tenant cost tracking means pricing can actually reflect real consumption — light users pay less, heavy users pay proportionally more, and the platform's margins stay predictable regardless of how usage patterns vary across the tenant base.
Aggregate-only cost tracking tells you the platform's overall margin but gives no way to price INDIVIDUAL tenants fairly or identify which specific tenants are actually profitable versus running at a loss.
Track Cost Attribution precisely per tenant, per request — the real LLM and infrastructure cost each tenant actually generates. This becomes the basis for usage-based billing that fairly reflects real consumption, rather than a flat rate that over- or under-charges depending on how each tenant actually uses the product.
Why this piece earns its place
Attribution is easy for costs that arrive per request and hard for everything else. Tokens through a model carry a tenant. The memory a vector index holds, an always-on embedding service, a reserved capacity floor you pay for whether or not anyone calls it — none of those decompose per request, so what you really have is a measured direct cost plus an allocated share, and the allocation rule you pick (by request count, by stored volume, by peak concurrency) decides which tenants look profitable. Write the rule down and keep it still, because changing it re-sorts the customer list with nothing real having moved. The subtler trap is that this meter and the Quota Enforcer look like one counter and are not. Enforcement needs a number in milliseconds and survives losing the odd event; an invoice needs a number that is exact at month end and can take an hour to settle. Build one counter for both and you get something too slow to gate a request with and too lossy to bill from. So run two paths off the same measurement, and decide in advance which of them the tenant is shown — because the throttling they felt this morning and the usage on next month’s invoice will not agree.
What the new pieces do
- Cost Attributiondata
- Tracks exact LLM/infra cost per tenant per request — the basis for accurate usage-based billing, not a flat guess.
Step 6 · Data residency and compliance
Some tenants' data can't leave a region
An enterprise tenant in the EU requires their data to stay within EU infrastructure for regulatory reasons. Can they be served from the same global deployment as every other tenant?
Add a Region Router that directs a tenant's requests and storage to region-specific infrastructure when their compliance requirements demand it — a tenant with data-residency requirements is transparently routed to and stored in the correct region, while tenants without such requirements continue using the default global deployment. This is a per-tenant configuration choice, not a platform-wide architectural fork.
Why this piece earns its place
Routing is the easy half. The hard half is drawing the line between the control plane and customer content, because the platform needs one global view of who its tenants are, how they are configured and what they owe, while nothing they typed or uploaded may leave. Write that split down explicitly — identity and billing metadata global, content regional — because every feature built afterwards will be built by someone who has not thought about it, and a cross-region analytics table or a single support console listing everyone’s recent questions is the shape that fails an audit. The boundary therefore belongs in the data model, where a schema forces the question, and not in a routing table nobody opens. Two things get missed routinely. The inference call itself leaves the region if the model provider serves it from elsewhere, so your contract with that provider is part of the guarantee, not an implementation detail. And logs become customer content the moment a request body lands in one. Moving an existing tenant between regions is a migration with downtime, which is why the question is asked at signup.
What the new pieces do
- Region Routerservice
- Routes a tenant's requests and storage to region-specific infrastructure when their compliance requirements demand it.
Step 7 · Automated tenant onboarding
A new customer shouldn't need custom engineering
A new enterprise customer signs up. Should provisioning their isolated configuration, storage, and quotas require manual, bespoke engineering work each time?
Make onboarding self-service and automated: creating a new tenant's isolated config entry, data-store scope, and default quotas is a fast, automated provisioning flow the Tenant Admin can trigger themselves — not a manual, per-customer engineering task that doesn't scale as the platform grows its customer base.
Why this piece earns its place
Provisioning is a write across several systems — a config entry, a data-store scope, quota defaults, a billing record, an identity-provider entry — with no transaction spanning them. A failure halfway leaves a tenant that half exists, which is worse than one that does not, because some code paths find it and others do not. So make the flow idempotent and keyed on the tenant id, safe to re-run from the beginning after any failure, and keep the tenant unusable until a final step marks provisioning complete. Build it as a declared sequence of steps rather than a script and the audit comes free: what a tenant has is what the sequence says it has, so a drifted tenant is a diff rather than an investigation. The first tenants to show a diff are the ones created before the flow existed, assembled by hand and carrying a missing quota row or a config key nothing sets any more — backfill those the day you automate, because nothing else will ever look at them again. And self-service means a sign-up form now creates infrastructure. Put verification and a rate limit in front of it, or the tenant table becomes an abuse surface, and every abandoned tenant still carries whatever a tenant costs to hold.
Step 8 · The sharp edges
Defense in depth, and genuine deletion
Two of the highest-stakes real-world requirements: a bug in any single isolation layer must not cause a full cross-tenant breach, and when a tenant leaves, their data needs to actually, verifiably disappear.
Isolation is enforced independently at multiple layers — API-level tenant scoping AND data-layer tenant verification — so a bug in either one alone doesn't produce a full breach, as demonstrated by the Step 2 chaos scenario. For tenant offboarding, deletion must be genuine and verifiable across every store the tenant's data touched (documents, embeddings, config, logs, backups) — not simply marking the tenant inactive while their data quietly persists, which fails compliance requirements like the GDPR right to erasure.
Why this piece earns its place
Two layers only buy you something if they can be wrong independently, and the easy mistake is to draw one layer twice. If the API check and the store check both read the tenant id out of the same header through the same shared helper, a bug in that helper is agreed with by both of them and the second check adds nothing but comfort. Independence has a price worth naming out loud: the identity the store verifies has to reach it in a form the API cannot simply retype — a token minted when the user authenticated and signed, carried with the request and checked by the store against its own key; or, where the store can only know what the connection tells it, credentials issued per tenant, which buys the independence by giving up the one shared pool. Then a wrong id upstream fails verification instead of being agreed with, and that disagreement is the whole product of the design. Deletion has the symmetric property: it is the one path that destroys data on purpose, so a scoping bug there is the only unrecoverable one in the system. Give it a grace period before anything really goes, and make it print what it would remove before it removes anything — the one code path worth a dry run every single time it runs.
You did it
You just designed a multi-tenant AI SaaS platform.
Everything you assembled, in order
- An unscoped shared index risks one tenant's private data surfacing directly in another tenant's results — a severe breach, not a minor bug.
- Tenant isolation is enforced independently at multiple layers (API and data store), so a bug in any one alone doesn't cause a full breach.
- Per-tenant customization lives in configuration resolved at request time, on one shared runtime — not a separately deployed stack per tenant.
- Per-tenant quotas contain the blast radius of any single tenant's traffic spike to that tenant alone.
- Precise per-tenant cost attribution is the prerequisite for fair, usage-based billing across widely varying tenant usage patterns.
- A region router transparently serves tenants with data-residency requirements from appropriate regional infrastructure.
- Automated, self-service tenant onboarding is what lets the customer base scale independent of platform-engineering headcount.
- Tenant offboarding requires genuine, verifiable deletion across every data store, not just marking a tenant inactive.
