The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What are we counting?
Millions of ad clicks per second, and advertisers want to see totals — by campaign, by minute, by region — almost instantly, to decide where to spend. The counts must be fast, accurate (it’s billing), and queryable in many ways.
Treat clicks as a stream: ingest them cheaply, aggregate them in real time into rollups advertisers can query, and run a slower batch pass to guarantee the numbers are exactly right. Speed from the stream, truth from the batch.
What the new pieces do
- Ad Clickevent
- A single click event among millions per second. It must be recorded fast and counted accurately — advertisers pay per click, so every count is money.
Step 1 · Catch the firehose
Ingest to a stream
At millions of events per second, writing each click straight to a database would melt it, and any slow downstream step would drop clicks — which means dropping revenue.
Millions of clicks/sec, and every click is billable revenue. How do you ingest without dropping any?
A database can’t absorb millions of writes/sec, and any slow downstream step backs up and drops clicks — dropping revenue. Ingestion must not be coupled to processing speed.
In-memory counts vanish on restart and can’t be replayed or reconciled — fatal for a billing system. You need a durable record of every raw click.
Validate and append to Kafka, then return — ingestion is a fast append that absorbs spikes, buffers durably, and lets many consumers read independently. Nothing is lost if a consumer lags.
A thin Ingest API validates each click and appends it to a durable Event Stream (Kafka), then returns. The stream absorbs spikes, buffers durably, and lets multiple consumers read independently. Ingestion is now just a fast append.
Why this piece earns its place
The word doing all the work here is durably, and it hides an acknowledgement policy. If the Ingest API returns the moment a leader broker accepts the append, the click is acknowledged but not yet replicated — lose that broker in the next instant and the click is gone, and unlike a user-facing request there is nobody left to retry it, because the browser has already followed the ad to the landing page. So wait for replication to a quorum before returning and accept the extra time at the edge; an unacknowledged drop here is revenue that never appears in any audit, because no record of it was ever written. The second thing this step quietly assumes is retention. The log is a buffer, not an archive, so "nothing is lost while a consumer lags" is only true inside the retention window — a consumer stalled longer than that resumes at an offset that no longer exists. That is the real reason the lake write in step 5 is the consumer you never let fall behind: it is the only copy with no expiry date on it.
- millions/sclicks
- appendonly
- replayablebuffer
What the new pieces do
- Ingest APIbackend
- A thin, massively scaled endpoint that validates a click and writes it to the event stream. It does almost nothing else, so it can absorb huge spikes.
- Event Streambus
- The durable buffer between fast ingestion and slower processing. It absorbs spikes, allows replay, and feeds every downstream consumer independently.
Step 2 · Count as it flows
Stream aggregation
Advertisers won’t wait for a nightly job — they need totals within seconds. But you also can’t store every raw click in a way that’s fast to query by campaign and minute.
Advertisers want totals within seconds, queryable by campaign/minute. How do you produce them?
Scanning billions of raw clicks per dashboard query is far too slow and expensive. You want to pre-summarize the firehose, not re-scan it on every read.
Flink/Spark consumes the log and rolls clicks into per-minute, per-campaign counts written to an OLAP store. Dashboards query small pre-summed buckets — orders of magnitude less data, seconds fresh.
A nightly job can’t give within-seconds freshness — advertisers manage live spend. Batch is the truth layer (step 5), but the real-time number comes from the stream.
A Stream Processor consumes the log and continuously aggregates clicks into time windows (per-minute, per-campaign counts), writing rollups to an Aggregates DB (OLAP/time-series). Dashboards query small, pre-summed buckets, not raw events.
Why this piece earns its place
The knob this step hands you is the grain of the rollup key, and it costs more than it looks. Per-minute, per-campaign is one counter per campaign per minute. Add region and device — because step 3’s dashboard slices by them — and you are holding the cross-product of every combination that saw traffic, all of it live in the processor’s window state and all of it copied into every checkpoint. Checkpointing is what fails first: its duration grows with state, and once it grows past the checkpoint interval the job spends its life snapshotting instead of counting, and then takes forever to recover when it does restart. So pick the grain from the queries advertisers actually run rather than from every dimension you happen to have collected, and push the rare high-cardinality slices — creative id, placement, individual publisher — down to the batch path in step 5, which can afford a full scan of the lake and does not have to hold anything open while it waits.
- per-minutewindows
- rollupsnot raw
- secondsfreshness
What the new pieces do
- Stream Processorworker
- Consumes clicks and continuously aggregates them into per-minute, per-campaign counts (Flink/Spark Streaming) — turning a firehose into rollups.
- Aggregates DBstore
- Stores the rolled-up counts by dimension and time bucket, optimized for the slice-and-dice queries advertiser dashboards run.
Back of the envelope
- store counts, not clicks
- orders of magnitude less data
- tumbling / sliding windows
- infinite stream → finite queryable buckets
- per-minute rollups ⇒ seconds fresh
- dashboards read pre-summed rows
Step 3 · Let them see it
The query path
Advertisers need to slice the data many ways — by campaign, region, device, time range — and fast, while ingestion and aggregation keep hammering at full volume.
Advertisers slice by campaign, region, device, time — while ingestion hammers at full volume. How do you keep dashboards fast?
Querying the raw firehose for every slice is slow and pits read load against the write path. Dashboards should read summaries, never the raw events.
The processor’s working state is transient and tuned for aggregation, not flexible slice-and-dice, and reads would contend with processing. Persist rollups to a query-optimized store instead.
Reads hit small pre-aggregated, time-bucketed rows built for slice-and-dice, never the firehose — so query load and write load stay isolated. Separate read and write models.
Point dashboards at the Aggregates DB, which is built for exactly these slice-and-dice reads over time-bucketed data. Reads hit small pre-aggregated rows and never touch the raw firehose, so query load and write load stay isolated.
Why this piece earns its place
This step quietly assumes every metric on the dashboard is additive — that a day is the sum of its minutes and a country is the sum of its cities. Clicks are, which is the only reason pre-aggregation works at all. The assumption breaks the first time someone asks a question the stored shape cannot answer by summing: unique users reached by a campaign, or click-through rate against impressions. Distinct counts do not add up — one person clicking in two different minutes is one user, not two — so rolling buckets together overcounts, and you cannot repair it at read time, because the identities were discarded at write time. Both defenses have to be chosen here, in the rollup: keep a sketch such as HyperLogLog per bucket so distincts can be merged approximately, and for any ratio store the numerator and denominator as separate additive columns and divide after summing. Never store the ratio itself — averaging pre-computed rates across buckets is the classic wrong answer, and the dashboard renders it just as confidently as a right one.
What the new pieces do
- Advertiserclient
- Wants near-real-time totals — clicks by campaign, minute, region — to manage spend. Reads aggregates, never the raw firehose.
Step 4 · Count it once
Exactly-once aggregation
Streams deliver at least once, and processors restart and replay. Naively, a replayed click gets counted twice — and since clicks are billed, double-counting is charging advertisers for clicks that never happened.
The stream delivers at-least-once and processors replay on restart. Clicks are billed — how do you not double-count?
Exactly-once delivery is impossible across processor restarts and retries — a replayed click would inflate billed totals. The counting itself must be resilient to redelivery.
Track click ids already counted (or use the engine’s exactly-once state + idempotent writes) so reprocessing the same click leaves totals unchanged. At-least-once delivery → exactly-once counts.
Guessing and subtracting duplicates is inaccurate — unacceptable for billing. You need precise per-click dedup, not a statistical correction.
Give every click a unique id and dedup on it: the processor keeps track of ids already counted (or uses the framework’s exactly-once state + idempotent writes) so reprocessing the same click leaves the totals unchanged.
Why this piece earns its place
Remembering ids is not free, and the question this step skips is how long you remember them. You cannot keep every id forever — that set grows with every click ever ingested, and it lives in the same processor state that has to be checkpointed — so in practice the dedup set is a bounded window, and a duplicate that arrives after it expires is counted again. The width is the knob, and it has a floor: it has to cover the whole envelope in which a redelivery can plausibly arrive, which includes a client retrying, a processor replaying from its last committed offset after a long outage, and the stragglers of step 7 landing well after their window. Set it from the observed lateness of real traffic, not a round number, and know which failure each direction buys — too narrow double-counts and overbills, too wide inflates state and slows every checkpoint. It is also why the guarantee here is scoped rather than absolute: the unbounded backstop is the batch pass in step 5, which dedups across the entire lake at once and has no window to fall out of.
What the new pieces do
- Dedupservice
- Recognizes a click it has already counted (by a unique click id) so retries and replays don’t inflate totals — the key to accurate, exactly-once counts.
Back of the envelope
- at-least-once + replay
- a naive count double-counts on retry
- dedup on unique click id
- reprocessing leaves totals unchanged
- = exactly-once counts
- the effect happens once, not the delivery
Step 5 · Trust the numbers
Batch reconciliation
The fast stream path can still drift — a bug, a dropped window, a late-arriving correction. For a billing system, “probably right” isn’t good enough; you need a number you can defend.
The fast stream path can drift (a bug, a dropped window). For a billing system, how do you get numbers you can defend?
Even a well-tested stream path drifts from dropped windows, dedup gaps or late data — and "trust us" doesn’t survive a billing dispute. You need an independent, authoritative recomputation.
Manual reconciliation doesn’t scale to billions of events and erodes trust. The system itself must produce a defensible number automatically.
Store every raw click in a durable lake and re-aggregate from scratch periodically to correct the fast path. Stream gives speed, batch gives authoritative truth — a lambda architecture you can always recompute from.
Keep every raw click in a durable Event Lake, and run a periodic Batch Recompute that re-aggregates from scratch and corrects the Aggregates DB. The stream gives speed; the batch gives an authoritative, reconciled source of truth (a lambda architecture).
Why this piece earns its place
Lambda’s real cost is not running two systems, it is that the definition of a billable click now lives in two places. The stream and the batch job have to agree on the dedup key, the window boundaries, the timezone, and exactly which of step 6’s fraud rules disqualify a click. Let one drift and reconciliation reports a difference every single cycle, and you will lose days deciding whether you are looking at a data problem or a code problem — the diff looks identical either way. So derive both paths from one shared definition wherever the engines allow it, and treat a persistent gap as a defect to explain rather than a number to accept. The other thing this step needs is a policy nobody asks for at the whiteboard: when does a number stop moving? Advertisers screenshot dashboards and finance invoices off them, so a bucket has to seal at some point, after which a correction becomes a line on the next invoice instead of a silent rewrite of yesterday. Reconciliation that rewrites history forever is indistinguishable from a system that cannot count.
What the new pieces do
- Batch Recomputeworker
- Periodically re-aggregates the raw events from scratch to correct any drift in the fast path — the slow, authoritative source of truth.
- Raw Event Lakestore
- A durable, append-only archive of every raw click. The ground truth that batch jobs replay and that lets you recompute anything later.
Back of the envelope
- fast stream = speed, approximate
- near-real-time, but may drift
- batch over raw lake = truth
- re-aggregate from scratch, reconcile
- keep raw forever
- you can always recompute anything later
Step 6 · Real clicks only
Fraud detection
A big slice of ad traffic is bots and click farms. Counting fraudulent clicks bills advertisers for nothing and corrupts the metrics they make decisions on — left unchecked, it poisons the whole product.
A big slice of ad traffic is bots and click farms. How do you keep from billing advertisers for fraud?
Billing fraud then refunding erodes trust, is operationally costly, and corrupts the live metrics advertisers decide on. Fraud must be excluded before billing, not after.
A fraud filter flags impossible rates, suspicious IPs/devices and bot patterns, excluding them from billable counts while keeping them in raw data. Real-time catches obvious abuse, batch models catch the subtle kind.
Aggressively dropping "unusual" clicks throws away real human clicks (false positives) — also lost revenue. Keep everything, bill only on validated clicks, and let fraud rules evolve against retained evidence.
Run a Fraud Filter over the stream: flag impossible click rates, suspicious IPs/devices, and known bot patterns, excluding them from billable counts (while keeping them in raw data for analysis). Real-time signals catch obvious abuse; batch models catch the subtle kind.
Why this piece earns its place
The knob here is where the line between observed and billable sits, and its two errors neither cost the same nor announce themselves the same way. A miss bills an advertiser for a bot: visible, disputed, corrosive to trust. A false positive throws away a real human click: nobody complains, because no one notices a charge they never received, and the loss surfaces only as revenue you quietly never invoiced. That asymmetry pushes teams to tighten the filter until the invisible error is the larger one, and nothing in the metrics will tell you that happened. The only defense is a labelled evidence set — sampled clicks properly adjudicated, crucially including the ones you rejected — because without labels you can measure how much you blocked but never how much of it was real. The second thing to say out loud is that this is the only adversarial component on the page. Whoever is on the other side gets to observe the effect of every rule you ship, so a fraud path needs a release cadence and a periodically retrained batch model, not a rule list written once and left alone.
What the new pieces do
- Fraud Filterservice
- Flags non-genuine clicks — bots, click farms, impossible rates — so advertisers aren’t billed for fraud and counts reflect real humans.
Step 7 · Scale & late events
The sharp edges
A viral campaign creates a hot partition that overwhelms one processor, and a phone offline for an hour sends its click late — after its window already closed and was reported.
Spread load by partitioning on a high-cardinality key (and pre-aggregating at the edge for hot keys). Handle stragglers with watermarks and a grace period: keep windows open briefly for late events, then emit corrections that the batch layer ultimately reconciles.
Why this piece earns its place
A watermark is the disciplined version of a guess, and the guess is the grace period — so the useful thing to say is how you would set it. It comes from the measured distribution of arrival delay: find the point where waiting longer stops recovering a meaningful number of clicks, because every extra moment of grace delays the emission of every window, including the overwhelming majority that were complete on time. That is the direct trade against step 2’s promise of totals within seconds, and erring short is survivable — it only pushes more clicks into the correction path and makes yesterday’s number visibly move once. The failure worth naming is the stalled watermark. A watermark advances with the slowest partition’s event time, so one partition with no traffic — a quiet region overnight, or a shard left empty after a partitioning change — holds it back, and every window in the job stays open waiting for a straggler that will never come. Emit idle markers from silent partitions, so no data reads as no data rather than as no time passing.
You did it
You just designed an ad click aggregator.
Everything you assembled, in order
- A thin ingest API appends clicks to a durable, replayable event stream.
- A stream processor aggregates into time windows, storing rollups in an OLAP DB.
- Dashboards query pre-aggregated buckets, isolated from the write path.
- Per-click dedup keys turn at-least-once delivery into exactly-once counts.
- A raw event lake + batch recompute reconcile the fast path (lambda architecture).
- A fraud filter separates observed clicks from billable, valid ones.
- Partitioning handles hot keys; watermarks and event-time handle late events.