The whole design, in writing
Learn system design by building a feature flag / experimentation platform like LaunchDarkly or Flagsmith step by step. An interactive guide covering why redeploying to toggle a feature is slow and risky, a flag store + dashboard that lifts flags out of code, an SDK that evaluates flags locally, targeting rules with consistent-hash percentage rollouts, low-latency delivery via local cache + streaming + edge, experimentation with exposure events, and the unhappy paths (sticky bucketing, offline fallback, flag debt, kill switch).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a feature flag system?
You want to ship a risky new checkout flow to 1% of users in Canada, watch the metrics, ramp to 100% over a week — and if it breaks, turn it off in seconds, at 3am, without a code deploy or a rollback. A hardcoded if (newCheckout) can't do any of that: changing it means a full build, review, and redeploy.
A feature flag platform decouples deploy from release. Features ship dark behind flags; a runtime config (with targeting rules) decides who sees what, evaluated in-app by an SDK. You release gradually, target segments, run A/B tests, and kill a bad feature instantly — all by changing config, never code.
Step 1 · The redeploy trap
if/else + redeploy
The naive approach: guard the feature with if (FLAG) where FLAG is a constant, and change the constant to release. What can't this do that a real release needs?
A hardcoded boolean guards the feature. What can't it do?
A constant can't target segments, ramp a percentage, differ per user, or turn off without a redeploy. Real releases need runtime, per-user, reversible control.
Aesthetics aside, the real gaps are functional: no gradual rollout, no targeting, no instant kill, and every change needs a full build+deploy cycle.
A compile-time constant is fixed until the next deploy, applies to everyone equally, and can't be reversed in seconds during an incident. Lifting the decision to runtime config unlocks all of it.
A compile-time constant is fixed until the next deploy, applies to everyone equally, and can't be reversed fast when something breaks. Releasing safely needs the decision to live in runtime config: per-user, gradual, targetable, and instantly reversible — separate from the code that ships the feature.
Why this piece earns its place
Lifting the decision to runtime costs you the compiler. A flag key is a string looked up while the program is running, so a typo in it — or a rename that landed in the dashboard and not in the code — compiles cleanly, passes review, ships, and then resolves to nothing at all. That is a bug with no stack trace and no failed test, and it is usually found by a person asking why the feature they were promised never arrived. What I would do about it is make the keys typed: generate constants from the flag store so code cannot reference a flag that does not exist, and have the SDK complain loudly in non-production the first time it is asked for a key it has never seen. The second cost is that both branches are now live in production at once. The off path is what most users run, the on path is what someone turns on at 3am, and a build that only ever exercises one of them has tested half the program — so the suite has to run the flagged paths in both states, not just the state that happens to be configured today.
Step 2 · Lift flags out of code
A flag store + dashboard
If the decision is runtime config, that config needs to live somewhere editable by humans (including non-engineers) without touching the codebase. Where, and how is it managed?
Where should the flag decisions live if not in code?
A redeployed file is just the redeploy trap again with extra steps — no instant change, no non-engineer control. The config needs a live store edited through a UI.
A small, read-heavy config store holds every flag and its targeting rules; a dashboard lets PMs/engineers create flags, write rules, ramp rollouts and kill features — no deploy. This is the control plane.
Per-environment constants still can't change at runtime or target users, and multiply the maintenance. Centralize the decisions in a managed store instead.
Put the decisions in a flag store — a small, read-heavy config holding every flag, its targeting rules, rollout percentages and segments — and manage it through a dashboard. Engineers and PMs create flags, write rules, ramp rollouts, and flip the kill switch from the UI. The control plane is now separate from your application code entirely.
Why this piece earns its place
The moment humans edit flags through a UI, what matters most is which copy of the world they are editing. A flag key is the same string in development, staging and production, and if it resolves to a single record, then a rollout someone ramped while testing is a rollout they just ramped for real customers. So environment belongs in the store’s identity rather than in a field on a rule: separate records, separate SDK credentials, separate permissions, so a client configured for staging physically cannot read the production ruleset. That buys safety and creates the next problem, which is promotion. Rules written and verified in staging do not move to production by themselves, and retyping a targeting rule by hand at the end of a long day is exactly where a wrong rule comes from. I would make promotion an explicit action in the dashboard — copy this flag’s rules from staging to production, shown as a diff before it applies — so the configuration that ships is the one that was tested, not a fresh one written under pressure.
What the new pieces do
- Flag Storedata
- The source of truth: every flag, its targeting rules, rollout percentages and segments. Small, read-heavy config.
- Dashboardservice
- Where humans create flags, write targeting rules, ramp rollouts and hit the kill switch — no code deploy needed.
Step 3 · Evaluate without a network call
The SDK evaluates locally
Your app checks flags on the hot path — potentially thousands of times per request across many flags. If each check were a network round trip to the flag service, you'd add latency everywhere and make the flag service a hard dependency of every app. How do checks stay instant?
Apps check flags constantly on the hot path. How do checks avoid a network round trip each time?
A network call per check adds latency to every request and makes the flag service a hard dependency — its outage would break your app. Evaluation must be local.
A frozen per-user cache can't reflect a rules change or a kill switch, and doesn't handle new users. You want the ruleset locally and evaluate fresh each check.
The SDK pulls the whole (small) flag config once, then evaluates any flag for any user in-memory with zero network calls per check. It refreshes the config in the background, so checks are instant and the flag service isn't on the request path.
The SDK evaluates locally. It downloads the (small) flag ruleset once from the delivery layer, caches it in memory, and then evaluates any flag for any user in-process — zero network calls per check, microsecond latency. It refreshes the config in the background. Flag checks become as cheap as a local function call, and the flag service is never on the request hot path.
Why this piece earns its place
Shipping the rules to the SDK means shipping the rules to wherever the SDK runs. In a backend service that is fine. In a browser bundle or a mobile app it means anyone who opens devtools can read every flag key, every segment definition and every rollout percentage you have — including the names of features you have not announced, and, if rules are written against personal attributes, the shape of how you segment your customers. None of that can be encrypted away; it is config the client needs in order to evaluate at all. So the defense is scope, not obfuscation: client SDKs get their own credential that returns only the flags marked client-visible, and anything sensitive either stays server-side or moves to server-side evaluation, where the delivery layer takes a user’s attributes and returns that user’s values instead of the rules that produced them. The same boundary protects the other assumption in this step, which is that the ruleset is small. It stays small only while segments are expressed as rules; the day someone pastes a list of individual user ids into a segment, the thing every client downloads starts growing with your user base.
- 0network calls per check
- µsevaluation latency
- backgroundconfig refresh
What the new pieces do
- App + SDKclient
- Your application with the flag SDK embedded. It evaluates flags in-process for each user — no network call per check.
- Flag Deliverygateway
- Serves the evaluated flag config to SDKs, cached at the edge so millions of clients bootstrap fast and cheaply.
Step 4 · Who gets it?
Targeting & percentage rollouts
"1% of users in Canada" and "everyone on the beta team" require the SDK to decide, per user, deterministically. Random each call would flip a user in and out of the feature between page loads. How do you roll out to a stable percentage?
How do you put a *stable* 1% of users into a feature (not flip them each request)?
Per-request randomness means the same user flips in and out between requests — a broken, inconsistent experience. The decision must be sticky per user.
Hashing (user id + flag key) to a stable bucket gives each user a fixed position, so the same user always gets the same answer, and ramping the % just admits more buckets. Combine with rule matching (country=CA, segment=beta) for targeting.
A per-user stored decision works but needs a lookup and storage for every user and flag — heavy and stateful. Consistent hashing gets stable bucketing for free, statelessly, in the SDK.
Combine rule matching with consistent bucketing. Rules match attributes (country=CA, segment=beta, plan=pro). For a percentage rollout, the SDK hashes (userId + flagKey) into a stable bucket 0–99; the user is "in" if their bucket is below the rollout %. Same user → same bucket → same answer every time, and ramping the % just admits more buckets — no one flips back out.
Why this piece earns its place
Consistent hashing is only as stable as the identifier you feed it. A logged-in user id is stable. An anonymous visitor does not have one, so you bucket on a device or session id, and that changes when somebody clears storage, switches browser, or opens the app on a second device — the same human re-buckets and can land on the other side of the rollout. The sharpest version is login itself: if you bucket an anonymous visitor on a device id and then switch to their account id the moment they sign in, the assignment can flip at precisely the point you were trying to measure. So the bucketing key is a per-flag decision I would make deliberately rather than inherit. Account id for anything billing- or entitlement-shaped, where two people sharing an account must see the same thing. Device id for anonymous interface changes. And where a flow crosses login, carry the pre-login identifier through and keep hashing on it, so the answer does not change halfway through the journey it is being judged on.
- hash(user+flag)stable bucket
- bucket < %user is in
- stickysame answer every time
Step 5 · Change takes effect now
Low-latency delivery: cache + stream
The kill switch is only useful if flipping it reaches every running SDK fast. Polling every 5 minutes is too slow when a feature is melting production. But you also can't have millions of SDKs hammering the store. How do changes propagate in milliseconds without overload?
A kill switch must reach millions of SDKs fast, without hammering the store. How?
Millions of SDKs polling per second would crush the store and cost a fortune. You want the source of truth to push changes, and to serve bootstrap reads from a cache/edge.
Waiting for restarts means a kill switch could take hours — useless in an incident. Changes must reach running SDKs live.
On start, an SDK loads config (cached at the edge for millions of cheap reads), then keeps a streaming connection (SSE/WebSocket) that pushes any change instantly. A toggle propagates in milliseconds, polling is only a fallback, and the store isn't hammered.
Three layers. SDKs keep a local cache so they always have an answer. They subscribe to a stream (SSE/WebSocket) that pushes a change the instant it's saved, so a kill switch reaches every client in milliseconds. And the initial config is served from an edge/CDN cache so millions of SDKs bootstrap cheaply without touching the store. Polling remains as a fallback if the stream drops.
Why this piece earns its place
A push-based delivery layer is calm in steady state and dangerous at the moment it comes back. Every SDK holds an open connection, so when the relay tier restarts or a region drops, you do not get a trickle of reconnections — you get all of them in the same second, each one immediately followed by a config bootstrap, which is the heaviest request in the system. That is a second outage, self-inflicted, arriving on top of the first. So reconnects need randomized backoff with jitter from the very first retry, and the bootstrap they perform has to be absorbed by the edge rather than reaching the store. The fallback poll deserves the same arithmetic before anyone picks its interval: that number is not a freshness preference, it is a capacity decision, because the only time every client is polling is the time the stream is down and every client is polling at once. Choose an interval the delivery tier can serve with streaming gone entirely, and let the stream be what makes changes feel instant rather than what makes them possible.
- ~mskill-switch propagation
- edgecheap bootstrap
- pollfallback only
What the new pieces do
- Streambus
- Pushes flag changes to every connected SDK in real time (SSE/WebSocket) so a toggle takes effect in milliseconds.
Step 6 · Millions of clients
Scale delivery globally
Your SDK now runs in every user's browser, mobile app and backend service — potentially millions of concurrent evaluators worldwide, all needing config and change streams with low latency everywhere.
Serve flag config from a globally distributed edge (CDN/PoPs) so every SDK bootstraps from a nearby copy, and terminate streaming connections at regional relays that fan out changes — the store publishes once, the edge fans out to millions. Keep the config small (it's just rules), version it, and let SDKs bootstrap from an embedded default so a brand-new client is never blocked waiting for config.
Why this piece earns its place
An edge cache is a cache, which means the millisecond kill switch is a promise about the stream path only. A client that is already connected gets the change pushed to it. A client starting up during the incident — and during an incident clients are restarting — fetches its first config from the nearest point of presence, and that copy is exactly as old as whatever the cache was told to hold. So the config should not live at a stable URL that gets mutated in place and then purged; a purge is a request to many locations that succeeds unevenly and cannot be confirmed from where you are standing. Publish each change as a new versioned object instead, so a fetch either finds the new version or does not find it yet, and nothing can serve a half-updated copy. The SDK then reconciles: whatever version it booted with, the first thing the stream tells it is the current version, and it refetches if they differ. That also covers the case nobody notices — propagation is not simultaneous everywhere, so for a short window two users in two regions can get different answers for the same flag. Fine for a feature; not fine for anything where the two must agree.
Step 7 · Measure the release
Experiments, events & audit
Rolling out is half the job — you also want to know if the new variation actually helped (an A/B test), and you need a trustworthy record of every flag change for safety and compliance.
Emit exposure events ("user X saw variation B") asynchronously, and join them with outcome metrics in an experimentation engine to measure lift with statistical significance — flags and experiments share the same bucketing. Keep an immutable audit log of every flag change (who, what, when) and require the kill switch to be one obvious click. Add guardrails: approvals for prod, and automatic rollback if a key metric craters during a ramp.
Why this piece earns its place
Watch what the exposure event does to the property bought back in step 3. Evaluation is local and free, which is precisely why it happens constantly — the same flag, for the same user, on every render and every request. One event per evaluation would put the network back on the hot path and produce a stream of data larger than the traffic that caused it. So the SDK has to deduplicate before it emits, keeping one exposure per user, flag and variation for the session, and batch what survives into a background flush that is allowed to fail quietly. The subtler half is when it fires. Emit on evaluation and you are counting users the code merely asked about, including flags evaluated for a screen the user never reached, and every one of those inflates the denominator with people who never saw the variation. Emit at the point of use, where the variation is actually rendered or acted on. That distinction changes nothing about any user’s experience and changes every number the analysis produces.
What the new pieces do
- Exposure Eventsevents
- Records which variation each user was exposed to, streamed asynchronously to power experiment analysis.
- A/B Analysisservice
- Joins exposures with outcome metrics to measure whether a variation actually moved the needle, with significance.
- Audit Logdata
- An immutable record of who changed which flag when — essential for safe, reversible releases and compliance.
Step 8 · The sharp edges
Stickiness, offline & flag debt
Reality bites: a user's bucket must stay stable even as you change the rollout %, an SDK may start with no network, dead flags accumulate as "flag debt," and a subtle bug in evaluation could change behavior for everyone.
Preserve sticky assignment across ramps by only ever adding buckets as the % grows (never re-shuffling), so no one who has the feature loses it. Ship in-code defaults so an SDK with no network still evaluates safely (fail-closed for risky features). Fight flag debt with owners, expiry dates and cleanup workflows — a stale flag is a landmine. And treat the SDK/evaluation as critical: test it hard, since a bug there changes production behavior instantly.
Why this piece earns its place
The in-code default is the same decision written down in two places, and on the day the two disagree, the code wins silently. That makes deleting a flag a release, not a cleanup. Remove a flag from the store while the branch is still in the code and every SDK falls through to its compiled default — usually off — so retiring a flag that had been ramped to everyone can turn the feature back off for everyone, with no deploy to blame and no error to trace. The order that survives this is: set the flag to its final value and leave it there, then delete the dead branch and ship code that no longer asks, and only then remove the flag from the store once nothing references it. The same mechanism is live far more often than people expect, because the default is also what an SDK returns in the window between process start and the first config landing. Either wait for the client to report ready before serving traffic, or make sure every default is a value you would be content to serve to everybody.
You did it
You just designed a feature flag system.
Everything you assembled, in order
- A hardcoded boolean can't target, ramp or kill without a redeploy — lift the decision to runtime config.
- A flag store + dashboard is the control plane; humans define flags and rules without touching code.
- The SDK evaluates flags locally in-process — zero network calls per check, outage-resilient.
- Targeting rules + consistent-hash bucketing give sticky, stateless percentage rollouts.
- Local cache + streaming push + edge delivery = millisecond kill switch without hammering the store.
- Fan out config/streams at the global edge to reach millions of SDKs.
- Exposure events power A/B experiments; an audit log + kill switch + guardrails keep releases safe.