The whole design, in writing
Learn system design by building a metrics and monitoring system like Prometheus, Datadog or Grafana step by step. An interactive guide covering why SSH-and-grep can't scale, instrumentation and metric types, pull vs push collection, storing metrics in a time-series database, alerting rules with dedupe and routing, dashboards, high-availability collectors and federation, and the unhappy paths (alert storms, cardinality, monitoring the monitor).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a monitoring system?
You run thousands of services across thousands of machines. When something breaks at 3am — latency spikes, a disk fills, error rate climbs — you need to know before your users do, see where it's happening, and get paged. SSHing into boxes to tail logs doesn't scale past a handful of servers.
A monitoring system turns a fleet's raw signals into awareness. It collects metrics from every target, stores them in a time-series database, lets humans query and visualize them, and — most importantly — alerts a human when something crosses a line. Collect → store → query → alert.
What the new pieces do
- Targetsclient
- The services, hosts and containers you're watching. Each exposes its current metrics at an endpoint (or ships them out).
Step 1 · The reactive trap
SSH-and-grep doesn't scale
The starting point for many teams: when something seems wrong, log into servers and check. It works for three servers. Why is it hopeless — and dangerous — for three thousand?
Manually checking servers when something "seems wrong". Why does it fail at scale?
Bandwidth isn't the issue. The problems are that it's reactive (you look only after noticing), unscalable (you can't watch thousands of boxes by hand), and has no history to compare against.
Log size is a symptom. The core failure is that manual, reactive inspection can't continuously watch a fleet or catch problems before users do — you need automated, always-on measurement.
You only look after something breaks, you can't manually watch thousands of hosts, and you have no baseline to compare "now" against. Monitoring must be automatic, continuous, historical, and page you proactively.
Manual inspection is reactive (you look only after noticing), unscalable (no human watches thousands of hosts), and historyless (no baseline for "is this normal?"). Monitoring flips all three: continuously measure everything, store the history, and automatically alert so the system tells you — before the users do.
Why this piece earns its place
Worth saying out loud what continuous measurement gives up. SSH-and-grep was open-ended: at three in the morning you could look at anything on that box, including the thing nobody thought to record. A metrics pipeline only ever sees what was instrumented in advance, so the first incident caused by something you never emitted is invisible on every dashboard you own — and the graphs will look calm while the service burns. That isn’t an argument for keeping the manual path as the primary one; it’s an argument for two habits. Keep an exploratory escape hatch — logs, traces, and the ability to get onto a host — for the questions metrics structurally cannot answer. And treat every incident the metrics missed as an instrumentation bug, with a fix that ships alongside the code fix. The pipeline improves only if someone deliberately closes the gaps each incident reveals; otherwise you’ve automated exactly the visibility you had a year ago, and quietly stopped looking anywhere else.
Step 2 · What to measure
Instrumentation & metric types
Before collecting anything, decide what a "metric" even is. You want cheap-to-store numbers that summarize behavior — not every event. What primitive metric types cover what you need to know?
What form should the measurements take?
Storing and counting every event is the log/event pipeline's job — far more expensive and detailed than you need for health. Metrics are cheap pre-aggregated numbers: counters, gauges, histograms.
A counter only goes up (requests, errors); a gauge goes up and down (memory, queue depth, temperature); a histogram buckets a distribution (request latency) so you can compute p50/p99. These few primitives, sampled over time, describe system health compactly.
One boolean throws away all the signal you need to see trends, distributions and leading indicators. You want quantitative metrics over time, not a binary.
Instrument code to emit a few numeric metric types: a counter (monotonic — requests, errors), a gauge (up/down — memory, queue depth, temperature), and a histogram/summary (a bucketed distribution — request latency, so you can compute p50/p99). Each metric carries labels (service, region, endpoint) for slicing. Cheap numbers sampled over time, not raw events.
Why this piece earns its place
Pre-aggregation is the whole win here, and it’s also irreversible. A histogram’s buckets and a metric’s labels are chosen at emit time, in code, and you can’t recover a distribution you never bucketed — if the top latency bucket ends below where your real tail lives, every slow request lands in one indistinguishable lump, and the day p99 matters you’re shipping a code change and waiting on a deploy to see it. It cuts both ways: each extra bucket multiplies the series this metric produces, on every instance, forever. The subtler trap is arithmetic. Percentiles don’t average. The mean of each host’s p99 is not the fleet’s p99, it isn’t any meaningful number at all, which is why latency has to be emitted as bucket counts that get summed across hosts before the quantile is computed — a summary that pre-computes quantiles per process can’t be aggregated afterwards. So pick buckets around the thresholds you intend to alert on, and accept that changing them is a deploy, not a query.
Step 3 · Getting the numbers
Pull vs push collection
Metrics live inside thousands of processes. Something has to gather them centrally. Should the central system pull (scrape targets on a schedule) or should targets push their metrics out? This choice shapes the whole system.
How does the central system get metrics off thousands of processes?
Push works and suits some cases (short-lived jobs, serverless), but pure push makes it hard to know if a target is down (silence vs. never-existed) and lets a misbehaving client flood you. Pull gives the collector control and free liveness detection.
Manual export is the reactive trap again. Collection must be automatic and frequent — every few seconds — via pull or push, not a person.
In the pull model each target exposes its current metrics at an endpoint and the collector scrapes them every N seconds. Pull gives the collector control over rate, makes "target down" obvious (scrape fails), and simplifies discovery. Short-lived batch jobs that vanish between scrapes push to a gateway that the collector then scrapes.
Favor a pull model (like Prometheus): each target exposes its current metrics at a /metrics endpoint, and the collector scrapes it every few seconds. Pull gives the collector control of the sampling rate, makes a down target obvious (the scrape simply fails), and pairs naturally with service discovery. For short-lived batch jobs that die between scrapes, add a push gateway they push to and the collector scrapes.
Why this piece earns its place
The knob this step hands you is the scrape interval — this page picks fifteen seconds — and it isn’t a free parameter. Shorten it and every target pays more CPU serialising its metrics, the collector does proportionally more work, and the sample firehose the next step has to store grows in step with the change. Lengthen it and two things bite. Anything shorter than the interval simply isn’t sampled: a gauge like queue depth can spike and fully recover between two scrapes and leave no trace, which is a good reason to reach for a counter for anything you genuinely need counted, since a counter carries the whole delta no matter when you read it. And the interval sets the floor on detection — no alert can know about a condition sooner than the next successful scrape — so it has to stay small relative to how long you make alerts hold before firing, or the two compose into a delay nobody budgeted for. Set it per job rather than globally: a busy front end and a nightly batch host don’t deserve the same rate.
- pullcollector-paced
- scrape fail= target down
- push gwephemeral jobs
What the new pieces do
- Push Gatewayservice
- A catch for short-lived jobs that can't be scraped: they push their metrics here, and the collector scrapes the gateway.
- Collectorgateway
- Periodically pulls metrics from every target, adds labels, and writes the samples into the time-series store.
Step 4 · Where it all lives
Store it in a TSDB
Scraping thousands of targets every few seconds produces a firehose of timestamped samples — millions of points a second, kept for months, queried by time range. A general database can't hold this. Where do the samples go?
Millions of samples/sec, kept for months, queried by range. Where do they live?
A row-per-sample table melts on write amplification, index bloat and slow range scans at this volume — exactly the problem a specialized store exists to solve.
The TSDB is purpose-built for this: append-only ingest, columnar compression (~1–2 bytes/point), time partitioning, downsampling and retention. Collectors write samples in; queries aggregate ranges out.
Months of millions-per-second samples can't live in RAM, and it wouldn't survive a restart. Recent data sits in memory but is flushed durably to the compressed on-disk TSDB.
Store samples in a time-series database — the specialized engine for append-heavy, range-queried numeric data. Collectors write samples in; dashboards and alert rules query ranges out. The TSDB handles the hard parts (columnar compression to ~1–2 bytes/point, LSM ingest, downsampling, retention) so the rest of the pipeline just reads and writes it.
Why this piece earns its place
The assumption this store quietly bakes in is that samples arrive in time order. Append-only ingest into a writable head block is what makes the compression and the time partitioning work, and the price is that a sample older than what the head already holds is typically refused, not stored. That happens more often than it sounds: a target that was unreachable and now exposes catch-up values, a federated scrape pulling data that is already minutes old, and above all clock skew — a host whose clock runs fast writes points from the future, then appears to travel backwards the moment it’s corrected. The defence is mostly discipline. Timestamp at scrape time using the collector’s clock rather than trusting each target’s, keep the collectors’ own clocks disciplined, and — the part people skip — expose the count of rejected and out-of-order samples as a metric and alert on it. Otherwise the loss is silent: the write is dropped, the query still succeeds, and the resulting gap looks exactly like a period when nothing happened.
- appendsample firehose
- ~1–2 Bper point
- rangeaggregation reads
What the new pieces do
- Time-Series DBdata
- The specialized store for all the samples: append-heavy writes, range-aggregation reads. The heart of the system.
Step 5 · The whole point
Alerting: rules, dedupe & routing
Data sitting in a database doesn't wake anyone up. The real job is: when error rate exceeds a threshold for 5 minutes, get the right person paged — once, not a thousand times, and not for a blip. How?
How do you turn "error rate high for 5 min" into exactly the right page?
Alerting on raw points floods on-call with noise and false pages from momentary blips. You need a rule that must hold FOR a duration, and a layer that dedupes/groups before notifying.
Humans staring at dashboards is the reactive trap and can't cover 24/7. Alerts must be automatic, evaluated continuously against rules.
A rule like "error_rate > 5% FOR 5m" is evaluated on a schedule; when it holds it fires an alert. A separate alert manager deduplicates identical alerts, groups related ones, honors silences/maintenance windows, and routes to the right on-call via the right channel — one clear page, not a storm.
Split it in two. A rule evaluator runs alerting queries against the TSDB on a schedule; an alert fires only when a condition holds FOR a duration (so blips don't page). A separate alert manager then deduplicates identical alerts, groups related ones, honors silences/maintenance windows, and routes each to the right on-call via the right channel (page, Slack, ticket). Signal in, one clear page out.
Why this piece earns its place
The FOR duration is the knob, and five minutes is a choice, not a default. Too short and you’re back to paging on blips. Too long and you’ve written a fixed delay into every incident that rule covers — the condition is true, users are already affected, and the page is being deliberately withheld. The failure that surprises people sits between the two: a rule whose condition flaps around the threshold may never fire at all, because each dip back under resets the timer, so a service oscillating in and out of trouble stays invisible for as long as it never fails continuously. That’s an argument about what goes in the expression, not just how long you wait. Alerting on a ratio computed over a window — the share of requests failing over the last while — integrates the flapping instead of resetting on it, and crosses the line faster the worse the incident is, which is the behaviour you actually want from a pager. Keep instantaneous thresholds with short FOR durations for conditions that can’t flap, like a disk that only fills in one direction.
- FOR 5mno blip pages
- dedupe+groupone page, not 500
- routeright on-call
What the new pieces do
- Rule Evaluatorservice
- Runs alerting queries against the TSDB on a schedule; when a condition holds for long enough, it fires an alert.
- Alert Managerservice
- Deduplicates, groups, silences and routes firing alerts to the right people through the right channels.
- Notifynotify
- The channels an alert lands in: pager/on-call, chat, email, tickets. The last mile from signal to human.
Step 6 · Make it visible
Dashboards & queries
Humans need to see trends, compare "now" to last week, and drill into an incident across dimensions. Alerts tell you something is wrong; how do people investigate?
Add a query language over the TSDB (PromQL-style) and dashboards (Grafana-style) that visualize it: time-series panels, heatmaps, SLO burn-down, and label-based drill-down. A good dashboard turns "the site is slow" into "p99 latency for the checkout service in eu-west jumped when deploy X shipped." Alerts detect; dashboards diagnose.
Why this piece earns its place
Dashboards read from the same TSDB the rule evaluator reads from, and they read badly: a wall-mounted panel refreshing a month-long range across every label is one of the heaviest queries this system will ever run, and it runs forever whether or not anyone is looking at it. During an incident that load goes up — every engineer opens every dashboard and widens the range — and what those queries compete with is alert evaluation. A monitoring system that stops evaluating rules because humans are investigating has inverted its own priorities at the worst possible moment. So give the two read paths separate budgets: query limits and timeouts a dashboard hits well before the evaluator does, and, where you can, a separate read replica for interactive queries. Then stop recomputing the expensive expressions at all. A recording rule evaluates a costly aggregation once on a schedule and writes the result back as its own series, so both the dashboard and the alert read a cheap precomputed number — and they keep agreeing with each other, which is the other half of why the same expression should back both.
What the new pieces do
- Dashboardsservice
- Grafana-style panels that query the TSDB to visualize trends, SLOs and system health for humans.
Step 7 · Watch the whole fleet
Scale: HA collectors & federation
One collector can't scrape a whole datacenter, and if your monitoring has a single point of failure it will blind you exactly when an incident takes that node down. How do you scale collection and keep it reliable?
Run collectors in redundant pairs (both scrape the same targets) for high availability, and shard scraping so each collector owns a slice of targets (by service/region), discovered via service discovery. Federate: local collectors keep full-resolution data, and a global layer scrapes aggregated rollups from them for a fleet-wide view. Cluster the alert manager so alerts survive a node loss. And keep cardinality in check — labels with unbounded values (user/request ids) will blow up the TSDB.
Why this piece earns its place
Two collectors scraping the same targets do not produce one dataset. They scrape at different instants, so the two copies of a series disagree sample by sample — you can’t merge them and shouldn’t try. The usual arrangement is to let both replicas run the same rules independently and lean on the alert manager’s deduplication to collapse the two identical alerts into a single page, which is a second reason that component earns its place, and to pin dashboards to one replica at query time rather than blending them, accepting that a panel’s data shifts slightly when the chosen replica changes. Federation has its own edge: it is itself a scrape, with a timeout, and a scrape that doesn’t finish in time yields nothing — not a partial result. A global layer pointed at a raw regional endpoint works right up until that region grows, then fails wholesale and takes your fleet-wide view with it. So federate from recording-rule output only, keep the federated series count deliberately small, and stay honest about what you bought: the global layer can tell you a region is unhealthy, and only the regional collector can tell you which instance.
Step 8 · The sharp edges
Alert storms & monitoring the monitor
Monitoring's own failure modes are brutal: one root cause fires a thousand alerts (an alert storm), a metric's cardinality explodes, and — worst — the monitoring system itself goes down silently and you never find out.
Tame alert storms with grouping, inhibition (a "datacenter down" alert suppresses its 500 child alerts), and rate-limited routing. Enforce cardinality limits and reject/relabel high-cardinality labels. Set sane thresholds + FOR durations to cut false pages, and alert on symptoms (user-facing SLOs) over causes. Crucially, monitor the monitor: a separate, independent dead-man's switch that pages you if the pipeline stops producing data or the heartbeat alert stops firing.
Why this piece earns its place
Everything in this step shares one property that makes it uniquely dangerous: its failure mode is silence. An inhibition rule that has stopped matching, a silence that outlived the maintenance window it was written for, and a dead-man’s switch whose delivery path is broken all look identical to a quiet night. And the dependency model behind inhibition is hand-maintained — rename an alert or add a label and the parent stops suppressing its children, with no error anywhere, until the next datacenter event brings back the storm you thought you’d fixed. So two rules. The watchdog must not share fate with what it watches: if the dead-man’s switch is evaluated by your rule evaluator, routed by your alert manager and delivered by your paging integration, it is certifying the health of a pipeline it is part of, and a whole-region outage takes the alarm out along with everything else. Put its receiving end somewhere you don’t operate. And exercise all of this on purpose — fire a synthetic parent alert and check the children stay quiet, stop the heartbeat and check you get woken — because these are the only rules on the page that no incident will ever test for you.
You did it
You just designed a monitoring system.
Everything you assembled, in order
- SSH-and-grep is reactive, unscalable and historyless — measure continuously and alert automatically.
- Instrument with counters, gauges and histograms plus labels; watch the golden signals.
- Prefer pull (scrape /metrics) for control + free liveness; add a push gateway for ephemeral jobs.
- Store the sample firehose in a time-series database — the compressed, range-queried heart.
- Split alerting: a rule evaluator fires (FOR a duration); an alert manager dedupes, groups, routes.
- Dashboards + a query language turn detection into diagnosis across labels.
- Scale with HA + sharded collectors and federation; cap cardinality; and monitor the monitor.