The whole design, in writing
Learn system design by building a real-time ad click aggregator step by step. An interactive guide covering high-volume event ingestion, stream processing with windowed aggregation, exactly-once counting, a batch reconciliation (lambda) path, fraud detection, and handling late events at scale.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What are we counting?
Millions of ad clicks per second, and advertisers want to see totals — by campaign, by minute, by region — almost instantly, to decide where to spend. The counts must be fast, accurate (it’s billing), and queryable in many ways.
Treat clicks as a stream: ingest them cheaply, aggregate them in real time into rollups advertisers can query, and run a slower batch pass to guarantee the numbers are exactly right. Speed from the stream, truth from the batch.
What the new pieces do
- Ad Clickevent
- A single click event among millions per second. It must be recorded fast and counted accurately — advertisers pay per click, so every count is money.
Step 1 · Catch the firehose
Ingest to a stream
At millions of events per second, writing each click straight to a database would melt it, and any slow downstream step would drop clicks — which means dropping revenue.
Millions of clicks/sec, and every click is billable revenue. How do you ingest without dropping any?
A database can’t absorb millions of writes/sec, and any slow downstream step backs up and drops clicks — dropping revenue. Ingestion must not be coupled to processing speed.
In-memory counts vanish on restart and can’t be replayed or reconciled — fatal for a billing system. You need a durable record of every raw click.
Validate and append to Kafka, then return — ingestion is a fast append that absorbs spikes, buffers durably, and lets many consumers read independently. Nothing is lost if a consumer lags.
A thin Ingest API validates each click and appends it to a durable Event Stream (Kafka), then returns. The stream absorbs spikes, buffers durably, and lets multiple consumers read independently. Ingestion is now just a fast append.
- millions/sclicks
- appendonly
- replayablebuffer
What the new pieces do
- Ingest APIbackend
- A thin, massively scaled endpoint that validates a click and writes it to the event stream. It does almost nothing else, so it can absorb huge spikes.
- Event Streambus
- The durable buffer between fast ingestion and slower processing. It absorbs spikes, allows replay, and feeds every downstream consumer independently.
Step 2 · Count as it flows
Stream aggregation
Advertisers won’t wait for a nightly job — they need totals within seconds. But you also can’t store every raw click in a way that’s fast to query by campaign and minute.
Advertisers want totals within seconds, queryable by campaign/minute. How do you produce them?
Scanning billions of raw clicks per dashboard query is far too slow and expensive. You want to pre-summarize the firehose, not re-scan it on every read.
Flink/Spark consumes the log and rolls clicks into per-minute, per-campaign counts written to an OLAP store. Dashboards query small pre-summed buckets — orders of magnitude less data, seconds fresh.
A nightly job can’t give within-seconds freshness — advertisers manage live spend. Batch is the truth layer (step 5), but the real-time number comes from the stream.
A Stream Processor consumes the log and continuously aggregates clicks into time windows (per-minute, per-campaign counts), writing rollups to an Aggregates DB (OLAP/time-series). Dashboards query small, pre-summed buckets, not raw events.
- per-minutewindows
- rollupsnot raw
- secondsfreshness
What the new pieces do
- Stream Processorworker
- Consumes clicks and continuously aggregates them into per-minute, per-campaign counts (Flink/Spark Streaming) — turning a firehose into rollups.
- Aggregates DBstore
- Stores the rolled-up counts by dimension and time bucket, optimized for the slice-and-dice queries advertiser dashboards run.
Back of the envelope
- store counts, not clicks
- orders of magnitude less data
- tumbling / sliding windows
- infinite stream → finite queryable buckets
- per-minute rollups ⇒ seconds fresh
- dashboards read pre-summed rows
Step 3 · Let them see it
The query path
Advertisers need to slice the data many ways — by campaign, region, device, time range — and fast, while ingestion and aggregation keep hammering at full volume.
Advertisers slice by campaign, region, device, time — while ingestion hammers at full volume. How do you keep dashboards fast?
Querying the raw firehose for every slice is slow and pits read load against the write path. Dashboards should read summaries, never the raw events.
The processor’s working state is transient and tuned for aggregation, not flexible slice-and-dice, and reads would contend with processing. Persist rollups to a query-optimized store instead.
Reads hit small pre-aggregated, time-bucketed rows built for slice-and-dice, never the firehose — so query load and write load stay isolated. Separate read and write models.
Point dashboards at the Aggregates DB, which is built for exactly these slice-and-dice reads over time-bucketed data. Reads hit small pre-aggregated rows and never touch the raw firehose, so query load and write load stay isolated.
What the new pieces do
- Advertiserclient
- Wants near-real-time totals — clicks by campaign, minute, region — to manage spend. Reads aggregates, never the raw firehose.
Step 4 · Count it once
Exactly-once aggregation
Streams deliver at least once, and processors restart and replay. Naively, a replayed click gets counted twice — and since clicks are billed, double-counting is charging advertisers for clicks that never happened.
The stream delivers at-least-once and processors replay on restart. Clicks are billed — how do you not double-count?
Exactly-once delivery is impossible across processor restarts and retries — a replayed click would inflate billed totals. The counting itself must be resilient to redelivery.
Track click ids already counted (or use the engine’s exactly-once state + idempotent writes) so reprocessing the same click leaves totals unchanged. At-least-once delivery → exactly-once counts.
Guessing and subtracting duplicates is inaccurate — unacceptable for billing. You need precise per-click dedup, not a statistical correction.
Give every click a unique id and dedup on it: the processor keeps track of ids already counted (or uses the framework’s exactly-once state + idempotent writes) so reprocessing the same click leaves the totals unchanged.
What the new pieces do
- Dedupservice
- Recognizes a click it has already counted (by a unique click id) so retries and replays don’t inflate totals — the key to accurate, exactly-once counts.
Back of the envelope
- at-least-once + replay
- a naive count double-counts on retry
- dedup on unique click id
- reprocessing leaves totals unchanged
- = exactly-once counts
- the effect happens once, not the delivery
Step 5 · Trust the numbers
Batch reconciliation
The fast stream path can still drift — a bug, a dropped window, a late-arriving correction. For a billing system, “probably right” isn’t good enough; you need a number you can defend.
The fast stream path can drift (a bug, a dropped window). For a billing system, how do you get numbers you can defend?
Even a well-tested stream path drifts from dropped windows, dedup gaps or late data — and "trust us" doesn’t survive a billing dispute. You need an independent, authoritative recomputation.
Manual reconciliation doesn’t scale to billions of events and erodes trust. The system itself must produce a defensible number automatically.
Store every raw click in a durable lake and re-aggregate from scratch periodically to correct the fast path. Stream gives speed, batch gives authoritative truth — a lambda architecture you can always recompute from.
Keep every raw click in a durable Event Lake, and run a periodic Batch Recompute that re-aggregates from scratch and corrects the Aggregates DB. The stream gives speed; the batch gives an authoritative, reconciled source of truth (a lambda architecture).
What the new pieces do
- Batch Recomputeworker
- Periodically re-aggregates the raw events from scratch to correct any drift in the fast path — the slow, authoritative source of truth.
- Raw Event Lakestore
- A durable, append-only archive of every raw click. The ground truth that batch jobs replay and that lets you recompute anything later.
Back of the envelope
- fast stream = speed, approximate
- near-real-time, but may drift
- batch over raw lake = truth
- re-aggregate from scratch, reconcile
- keep raw forever
- you can always recompute anything later
Step 6 · Real clicks only
Fraud detection
A big slice of ad traffic is bots and click farms. Counting fraudulent clicks bills advertisers for nothing and corrupts the metrics they make decisions on — left unchecked, it poisons the whole product.
A big slice of ad traffic is bots and click farms. How do you keep from billing advertisers for fraud?
Billing fraud then refunding erodes trust, is operationally costly, and corrupts the live metrics advertisers decide on. Fraud must be excluded before billing, not after.
A fraud filter flags impossible rates, suspicious IPs/devices and bot patterns, excluding them from billable counts while keeping them in raw data. Real-time catches obvious abuse, batch models catch the subtle kind.
Aggressively dropping "unusual" clicks throws away real human clicks (false positives) — also lost revenue. Keep everything, bill only on validated clicks, and let fraud rules evolve against retained evidence.
Run a Fraud Filter over the stream: flag impossible click rates, suspicious IPs/devices, and known bot patterns, excluding them from billable counts (while keeping them in raw data for analysis). Real-time signals catch obvious abuse; batch models catch the subtle kind.
What the new pieces do
- Fraud Filterservice
- Flags non-genuine clicks — bots, click farms, impossible rates — so advertisers aren’t billed for fraud and counts reflect real humans.
Step 7 · Scale & late events
The sharp edges
A viral campaign creates a hot partition that overwhelms one processor, and a phone offline for an hour sends its click late — after its window already closed and was reported.
Spread load by partitioning on a high-cardinality key (and pre-aggregating at the edge for hot keys). Handle stragglers with watermarks and a grace period: keep windows open briefly for late events, then emit corrections that the batch layer ultimately reconciles.
You did it
You just designed an ad click aggregator.
Everything you assembled, in order
- A thin ingest API appends clicks to a durable, replayable event stream.
- A stream processor aggregates into time windows, storing rollups in an OLAP DB.
- Dashboards query pre-aggregated buckets, isolated from the write path.
- Per-click dedup keys turn at-least-once delivery into exactly-once counts.
- A raw event lake + batch recompute reconcile the fast path (lambda architecture).
- A fraud filter separates observed clicks from billable, valid ones.
- Partitioning handles hot keys; watermarks and event-time handle late events.