The big idea
What are we counting?
Millions of ad clicks per second, and advertisers want to see totals — by campaign, by minute, by region — almost instantly, to decide where to spend. The counts must be fast, accurate (it’s billing), and queryable in many ways.
Treat clicks as a stream: ingest them cheaply, aggregate them in real time into rollups advertisers can query, and run a slower batch pass to guarantee the numbers are exactly right. Speed from the stream, truth from the batch.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · Catch the firehose
Ingest to a stream
At millions of events per second, writing each click straight to a database would melt it, and any slow downstream step would drop clicks — which means dropping revenue.
Design decision: Millions of clicks/sec, and every click is billable revenue. How do you ingest without dropping any?
The call: A thin Ingest API appends each click to a durable event stream. — Validate and append to Kafka, then return — ingestion is a fast append that absorbs spikes, buffers durably, and lets many consumers read independently. Nothing is lost if a consumer lags.
A thin Ingest API validates each click and appends it to a durable Event Stream (Kafka), then returns. The stream absorbs spikes, buffers durably, and lets multiple consumers read independently. Ingestion is now just a fast append.
Decouple with a log: A durable log between producers and consumers turns a brittle synchronous pipeline into a resilient one: ingestion can’t be slowed by processing, and nothing is lost if a consumer lags.
Step 2 · Count as it flows
Stream aggregation
Advertisers won’t wait for a nightly job — they need totals within seconds. But you also can’t store every raw click in a way that’s fast to query by campaign and minute.
Design decision: Advertisers want totals within seconds, queryable by campaign/minute. How do you produce them?
The call: A stream processor aggregates clicks into time windows continuously. — Flink/Spark consumes the log and rolls clicks into per-minute, per-campaign counts written to an OLAP store. Dashboards query small pre-summed buckets — orders of magnitude less data, seconds fresh.
A Stream Processor consumes the log and continuously aggregates clicks into time windows (per-minute, per-campaign counts), writing rollups to an Aggregates DB (OLAP/time-series). Dashboards query small, pre-summed buckets, not raw events.
Windowed aggregation: Tumbling/sliding windows turn an infinite stream into finite, queryable buckets. You store counts, not clicks — orders of magnitude less data, and exactly what dashboards ask for.
Step 3 · Let them see it
The query path
Advertisers need to slice the data many ways — by campaign, region, device, time range — and fast, while ingestion and aggregation keep hammering at full volume.
Design decision: Advertisers slice by campaign, region, device, time — while ingestion hammers at full volume. How do you keep dashboards fast?
The call: Point dashboards at the OLAP Aggregates DB of pre-summed buckets. — Reads hit small pre-aggregated, time-bucketed rows built for slice-and-dice, never the firehose — so query load and write load stay isolated. Separate read and write models.
Point dashboards at the Aggregates DB, which is built for exactly these slice-and-dice reads over time-bucketed data. Reads hit small pre-aggregated rows and never touch the raw firehose, so query load and write load stay isolated.
Separate read and write models: The write path optimizes for ingesting and aggregating; the read path optimizes for flexible queries. Pre-aggregation is what keeps interactive dashboards fast over billions of events.
Step 4 · Count it once
Exactly-once aggregation
Streams deliver at least once, and processors restart and replay. Naively, a replayed click gets counted twice — and since clicks are billed, double-counting is charging advertisers for clicks that never happened.
Design decision: The stream delivers at-least-once and processors replay on restart. Clicks are billed — how do you not double-count?
The call: Dedup on a unique click id (or framework exactly-once state). — Track click ids already counted (or use the engine’s exactly-once state + idempotent writes) so reprocessing the same click leaves totals unchanged. At-least-once delivery → exactly-once counts.
Give every click a unique id and dedup on it: the processor keeps track of ids already counted (or uses the framework’s exactly-once state + idempotent writes) so reprocessing the same click leaves the totals unchanged.
Idempotent counting: Exactly-once isn’t about perfect delivery — it’s about making the effect of an event happen once. Dedup keys plus idempotent aggregate updates turn at-least-once delivery into exactly-once counts.
Step 5 · Trust the numbers
Batch reconciliation
The fast stream path can still drift — a bug, a dropped window, a late-arriving correction. For a billing system, “probably right” isn’t good enough; you need a number you can defend.
Design decision: The fast stream path can drift (a bug, a dropped window). For a billing system, how do you get numbers you can defend?
The call: Keep raw clicks; a periodic batch recompute reconciles the DB. — Store every raw click in a durable lake and re-aggregate from scratch periodically to correct the fast path. Stream gives speed, batch gives authoritative truth — a lambda architecture you can always recompute from.
Keep every raw click in a durable Event Lake, and run a periodic Batch Recompute that re-aggregates from scratch and corrects the Aggregates DB. The stream gives speed; the batch gives an authoritative, reconciled source of truth (a lambda architecture).
Lambda architecture: A fast, approximate streaming layer for now, plus a slow, exact batch layer for truth. Store the raw events so you can always recompute — the ultimate backstop against any fast-path mistake.
Step 6 · Real clicks only
Fraud detection
A big slice of ad traffic is bots and click farms. Counting fraudulent clicks bills advertisers for nothing and corrupts the metrics they make decisions on — left unchecked, it poisons the whole product.
Design decision: A big slice of ad traffic is bots and click farms. How do you keep from billing advertisers for fraud?
The call: Filter clicks; keep two counts — observed and billable-valid. — A fraud filter flags impossible rates, suspicious IPs/devices and bot patterns, excluding them from billable counts while keeping them in raw data. Real-time catches obvious abuse, batch models catch the subtle kind.
Run a Fraud Filter over the stream: flag impossible click rates, suspicious IPs/devices, and known bot patterns, excluding them from billable counts (while keeping them in raw data for analysis). Real-time signals catch obvious abuse; batch models catch the subtle kind.
Two counts: raw and valid: Keep everything, but bill only on validated clicks. Separating “what we observed” from “what counts” lets fraud rules evolve without losing the underlying evidence.
Step 7 · Scale & late events
The sharp edges
A viral campaign creates a hot partition that overwhelms one processor, and a phone offline for an hour sends its click late — after its window already closed and was reported.
Spread load by partitioning on a high-cardinality key (and pre-aggregating at the edge for hot keys). Handle stragglers with watermarks and a grace period: keep windows open briefly for late events, then emit corrections that the batch layer ultimately reconciles.
Event time vs processing time: Aggregate by when the click happened, not when it arrived. Watermarks bound how long to wait for late data; the reconciliation path fixes anything that slips past.
You did it
You just designed an ad click aggregator.
- A thin ingest API appends clicks to a durable, replayable event stream.
- A stream processor aggregates into time windows, storing rollups in an OLAP DB.
- Dashboards query pre-aggregated buckets, isolated from the write path.
- Per-click dedup keys turn at-least-once delivery into exactly-once counts.
- A raw event lake + batch recompute reconcile the fast path (lambda architecture).
- A fraud filter separates observed clicks from billable, valid ones.
- Partitioning handles hot keys; watermarks and event-time handle late events.