The big idea

What is a feature flag system?

You want to ship a risky new checkout flow to 1% of users in Canada, watch the metrics, ramp to 100% over a week — and if it breaks, turn it off in seconds, at 3am, without a code deploy or a rollback. A hardcoded if (newCheckout) can't do any of that: changing it means a full build, review, and redeploy.

A feature flag platform decouples deploy from release. Features ship dark behind flags; a runtime config (with targeting rules) decides who sees what, evaluated in-app by an SDK. You release gradually, target segments, run A/B tests, and kill a bad feature instantly — all by changing config, never code.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the platform grow, and at the end kill delivery and kill streaming to see why a flag outage never takes your app down. Hit Begin.

Step 1 · The redeploy trap

if/else + redeploy

The naive approach: guard the feature with if (FLAG) where FLAG is a constant, and change the constant to release. What can't this do that a real release needs?

Design decision: A hardcoded boolean guards the feature. What can't it do?

The call: Change at runtime, target a segment, ramp a %, or kill it without redeploying. — A compile-time constant is fixed until the next deploy, applies to everyone equally, and can't be reversed in seconds during an incident. Lifting the decision to runtime config unlocks all of it.

A compile-time constant is fixed until the next deploy, applies to everyone equally, and can't be reversed fast when something breaks. Releasing safely needs the decision to live in runtime config: per-user, gradual, targetable, and instantly reversible — separate from the code that ships the feature.

Deploy ≠ release: Deploy = get the code onto servers (dark). Release = turn it on for users. Flags split these, so you deploy continuously and release deliberately — ramping, targeting, and rolling back independently of your deploy pipeline.

Step 2 · Lift flags out of code

A flag store + dashboard

If the decision is runtime config, that config needs to live somewhere editable by humans (including non-engineers) without touching the codebase. Where, and how is it managed?

Design decision: Where should the flag decisions live if not in code?

The call: In a flag store, edited via a dashboard (flags, rules, rollout %, segments). — A small, read-heavy config store holds every flag and its targeting rules; a dashboard lets PMs/engineers create flags, write rules, ramp rollouts and kill features — no deploy. This is the control plane.

Put the decisions in a flag store — a small, read-heavy config holding every flag, its targeting rules, rollout percentages and segments — and manage it through a dashboard. Engineers and PMs create flags, write rules, ramp rollouts, and flip the kill switch from the UI. The control plane is now separate from your application code entirely.

Control plane vs data plane: The control plane (dashboard + store) is where flags are defined and changed by humans — low traffic, needs consistency and audit. The data plane (SDK evaluation) is where flags are read millions of times a second. Designing them separately is the key structural choice.

Step 3 · Evaluate without a network call

The SDK evaluates locally

Your app checks flags on the hot path — potentially thousands of times per request across many flags. If each check were a network round trip to the flag service, you'd add latency everywhere and make the flag service a hard dependency of every app. How do checks stay instant?

Design decision: Apps check flags constantly on the hot path. How do checks avoid a network round trip each time?

The call: The SDK downloads the flag ruleset and evaluates locally, in-process, per user. — The SDK pulls the whole (small) flag config once, then evaluates any flag for any user in-memory with zero network calls per check. It refreshes the config in the background, so checks are instant and the flag service isn't on the request path.

The SDK evaluates locally. It downloads the (small) flag ruleset once from the delivery layer, caches it in memory, and then evaluates any flag for any user in-process — zero network calls per check, microsecond latency. It refreshes the config in the background. Flag checks become as cheap as a local function call, and the flag service is never on the request hot path.

Local evaluation: Shipping the rules to the SDK (not asking the server per check) is what makes flags free to use everywhere. The server-side alternative (evaluate on a backend) exists for secrecy/large rulesets, but local evaluation is the default because it's fast and outage-resilient.

Step 4 · Who gets it?

Targeting & percentage rollouts

"1% of users in Canada" and "everyone on the beta team" require the SDK to decide, per user, deterministically. Random each call would flip a user in and out of the feature between page loads. How do you roll out to a stable percentage?

Design decision: How do you put a *stable* 1% of users into a feature (not flip them each request)?

The call: Hash the user id into a bucket 0–99; if bucket < rollout %, they're in — consistently. — Hashing (user id + flag key) to a stable bucket gives each user a fixed position, so the same user always gets the same answer, and ramping the % just admits more buckets. Combine with rule matching (country=CA, segment=beta) for targeting.

Combine rule matching with consistent bucketing. Rules match attributes (country=CA, segment=beta, plan=pro). For a percentage rollout, the SDK hashes (userId + flagKey) into a stable bucket 0–99; the user is "in" if their bucket is below the rollout %. Same user → same bucket → same answer every time, and ramping the % just admits more buckets — no one flips back out.

Sticky, stateless bucketing: Hashing user+flag to a bucket gives deterministic, sticky assignment with no server state — every SDK computes the same answer independently. Salting by flag key means a user isn't correlated across different flags/experiments. This is the mathematical heart of gradual rollouts and A/B tests.

Step 5 · Change takes effect now

Low-latency delivery: cache + stream

The kill switch is only useful if flipping it reaches every running SDK fast. Polling every 5 minutes is too slow when a feature is melting production. But you also can't have millions of SDKs hammering the store. How do changes propagate in milliseconds without overload?

Design decision: A kill switch must reach millions of SDKs fast, without hammering the store. How?

The call: SDKs hold a local cache and subscribe to a stream that pushes changes; edge/CDN serves the initial config. — On start, an SDK loads config (cached at the edge for millions of cheap reads), then keeps a streaming connection (SSE/WebSocket) that pushes any change instantly. A toggle propagates in milliseconds, polling is only a fallback, and the store isn't hammered.

Three layers. SDKs keep a local cache so they always have an answer. They subscribe to a stream (SSE/WebSocket) that pushes a change the instant it's saved, so a kill switch reaches every client in milliseconds. And the initial config is served from an edge/CDN cache so millions of SDKs bootstrap cheaply without touching the store. Polling remains as a fallback if the stream drops.

Push + cache + edge: Cache makes the SDK resilient and instant. Streaming makes changes near-real-time. Edge delivery makes bootstrap scale to millions. Together they give the two things that seem to conflict — instant global change AND never overloading the source of truth.

Step 6 · Millions of clients

Scale delivery globally

Your SDK now runs in every user's browser, mobile app and backend service — potentially millions of concurrent evaluators worldwide, all needing config and change streams with low latency everywhere.

Serve flag config from a globally distributed edge (CDN/PoPs) so every SDK bootstraps from a nearby copy, and terminate streaming connections at regional relays that fan out changes — the store publishes once, the edge fans out to millions. Keep the config small (it's just rules), version it, and let SDKs bootstrap from an embedded default so a brand-new client is never blocked waiting for config.

Fan-out at the edge: The control plane writes rarely; the data plane reads constantly. So you push the read/stream load to the edge: one change published centrally is fanned out by edge relays to millions of subscribers. The store stays small and calm; the edge absorbs the scale.

Step 7 · Measure the release

Experiments, events & audit

Rolling out is half the job — you also want to know if the new variation actually helped (an A/B test), and you need a trustworthy record of every flag change for safety and compliance.

Emit exposure events ("user X saw variation B") asynchronously, and join them with outcome metrics in an experimentation engine to measure lift with statistical significance — flags and experiments share the same bucketing. Keep an immutable audit log of every flag change (who, what, when) and require the kill switch to be one obvious click. Add guardrails: approvals for prod, and automatic rollback if a key metric craters during a ramp.

Flags → experiments: Because assignment is already deterministic bucketing, a feature flag is an experiment: split users into variations, record exposures, compare metrics. The audit log + kill switch turn "changing prod behavior at runtime" from scary into safe and reversible.

Step 8 · The sharp edges

Stickiness, offline & flag debt

Reality bites: a user's bucket must stay stable even as you change the rollout %, an SDK may start with no network, dead flags accumulate as "flag debt," and a subtle bug in evaluation could change behavior for everyone.

Preserve sticky assignment across ramps by only ever adding buckets as the % grows (never re-shuffling), so no one who has the feature loses it. Ship in-code defaults so an SDK with no network still evaluates safely (fail-closed for risky features). Fight flag debt with owners, expiry dates and cleanup workflows — a stale flag is a landmine. And treat the SDK/evaluation as critical: test it hard, since a bug there changes production behavior instantly.

Design for the unhappy path: Ramp changes → additive buckets, never reshuffle. No network → safe in-code default. Old flags → expire and delete them. Eval bug → it's now your most safety-critical code. The power to change prod instantly cuts both ways; these guards keep it a scalpel, not a footgun.

You did it

You just designed a feature flag system.

• A —
• A —
• T — h
• T — a
• L — o
• F — a
• E — x