The big idea
What is a monitoring system?
You run thousands of services across thousands of machines. When something breaks at 3am — latency spikes, a disk fills, error rate climbs — you need to know before your users do, see where it's happening, and get paged. SSHing into boxes to tail logs doesn't scale past a handful of servers.
A monitoring system turns a fleet's raw signals into awareness. It collects metrics from every target, stores them in a time-series database, lets humans query and visualize them, and — most importantly — alerts a human when something crosses a line. Collect → store → query → alert.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end kill a scraper and kill the alert manager to see why you must monitor the monitor. Hit Begin.
Step 1 · The reactive trap
SSH-and-grep doesn't scale
The starting point for many teams: when something seems wrong, log into servers and check. It works for three servers. Why is it hopeless — and dangerous — for three thousand?
Design decision: Manually checking servers when something "seems wrong". Why does it fail at scale?
The call: It's reactive, unscalable, and has no history or automatic alerting. — You only look after something breaks, you can't manually watch thousands of hosts, and you have no baseline to compare "now" against. Monitoring must be automatic, continuous, historical, and page you proactively.
Manual inspection is reactive (you look only after noticing), unscalable (no human watches thousands of hosts), and historyless (no baseline for "is this normal?"). Monitoring flips all three: continuously measure everything, store the history, and automatically alert so the system tells you — before the users do.
Proactive, not reactive: The goal is MTTD/MTTR — detect and resolve fast. That requires always-on measurement and automated alerting, not a human periodically checking. You can't improve what you don't measure, and you can't react to what you don't detect.
Step 2 · What to measure
Instrumentation & metric types
Before collecting anything, decide what a "metric" even is. You want cheap-to-store numbers that summarize behavior — not every event. What primitive metric types cover what you need to know?
Design decision: What form should the measurements take?
The call: A few numeric types: counters, gauges, and histograms/summaries. — A counter only goes up (requests, errors); a gauge goes up and down (memory, queue depth, temperature); a histogram buckets a distribution (request latency) so you can compute p50/p99. These few primitives, sampled over time, describe system health compactly.
Instrument code to emit a few numeric metric types: a counter (monotonic — requests, errors), a gauge (up/down — memory, queue depth, temperature), and a histogram/summary (a bucketed distribution — request latency, so you can compute p50/p99). Each metric carries labels (service, region, endpoint) for slicing. Cheap numbers sampled over time, not raw events.
The golden signals: A useful default of what to measure: latency, traffic, errors, saturation (the "golden signals"), or USE (utilization/saturation/errors) for resources. Instrument these everywhere and most incidents become visible. Labels give you the dimensions to drill in.
Step 3 · Getting the numbers
Pull vs push collection
Metrics live inside thousands of processes. Something has to gather them centrally. Should the central system pull (scrape targets on a schedule) or should targets push their metrics out? This choice shapes the whole system.
Design decision: How does the central system get metrics off thousands of processes?
The call: The collector pulls (scrapes) each target's /metrics endpoint on a schedule; a push gateway catches jobs that can't be scraped. — In the pull model each target exposes its current metrics at an endpoint and the collector scrapes them every N seconds. Pull gives the collector control over rate, makes "target down" obvious (scrape fails), and simplifies discovery. Short-lived batch jobs that vanish between scrapes push to a gateway that the collector then scrapes.
Favor a pull model (like Prometheus): each target exposes its current metrics at a /metrics endpoint, and the collector scrapes it every few seconds. Pull gives the collector control of the sampling rate, makes a down target obvious (the scrape simply fails), and pairs naturally with service discovery. For short-lived batch jobs that die between scrapes, add a push gateway they push to and the collector scrapes.
Pull vs push: Pull: collector-controlled rate, free liveness detection, easy discovery — great for long-lived services. Push: better for ephemeral/serverless jobs and networks where targets can't be reached. Most systems are mostly-pull with a push escape hatch; some (Datadog agents, StatsD) are push-first. Know the trade.
Step 4 · Where it all lives
Store it in a TSDB
Scraping thousands of targets every few seconds produces a firehose of timestamped samples — millions of points a second, kept for months, queried by time range. A general database can't hold this. Where do the samples go?
Design decision: Millions of samples/sec, kept for months, queried by range. Where do they live?
The call: A time-series database: columnar, compressed, time-partitioned, append-only. — The TSDB is purpose-built for this: append-only ingest, columnar compression (~1–2 bytes/point), time partitioning, downsampling and retention. Collectors write samples in; queries aggregate ranges out.
Store samples in a time-series database — the specialized engine for append-heavy, range-queried numeric data. Collectors write samples in; dashboards and alert rules query ranges out. The TSDB handles the hard parts (columnar compression to ~1–2 bytes/point, LSM ingest, downsampling, retention) so the rest of the pipeline just reads and writes it.
The TSDB is the heart: Everything upstream feeds it and everything downstream reads it. Its properties — compression, cardinality limits, retention, downsampling — set the cost and performance of the whole system. (That store is a system design in its own right.)
Step 5 · The whole point
Alerting: rules, dedupe & routing
Data sitting in a database doesn't wake anyone up. The real job is: when error rate exceeds a threshold for 5 minutes, get the right person paged — once, not a thousand times, and not for a blip. How?
Design decision: How do you turn "error rate high for 5 min" into exactly the right page?
The call: A rule evaluator fires when a query holds for a duration; an alert manager dedupes, groups, silences and routes it. — A rule like "error_rate > 5% FOR 5m" is evaluated on a schedule; when it holds it fires an alert. A separate alert manager deduplicates identical alerts, groups related ones, honors silences/maintenance windows, and routes to the right on-call via the right channel — one clear page, not a storm.
Split it in two. A rule evaluator runs alerting queries against the TSDB on a schedule; an alert fires only when a condition holds FOR a duration (so blips don't page). A separate alert manager then deduplicates identical alerts, groups related ones, honors silences/maintenance windows, and routes each to the right on-call via the right channel (page, Slack, ticket). Signal in, one clear page out.
Rules vs routing: Keep detection (is something wrong? — rule evaluation over metrics) separate from notification (who hears about it, how, and de-duplicated). This separation is why the same alert from 500 hosts becomes one grouped page, and why you can silence a whole service during a deploy.
Step 6 · Make it visible
Dashboards & queries
Humans need to see trends, compare "now" to last week, and drill into an incident across dimensions. Alerts tell you something is wrong; how do people investigate?
Add a query language over the TSDB (PromQL-style) and dashboards (Grafana-style) that visualize it: time-series panels, heatmaps, SLO burn-down, and label-based drill-down. A good dashboard turns "the site is slow" into "p99 latency for the checkout service in eu-west jumped when deploy X shipped." Alerts detect; dashboards diagnose.
Query language is leverage: A metric query language lets you compute rates, aggregate across labels, and derive SLOs on the fly (rate(errors[5m]) / rate(requests[5m])). The same expressions power both dashboards and alert rules, so what you graph is what you alert on.
Step 7 · Watch the whole fleet
Scale: HA collectors & federation
One collector can't scrape a whole datacenter, and if your monitoring has a single point of failure it will blind you exactly when an incident takes that node down. How do you scale collection and keep it reliable?
Run collectors in redundant pairs (both scrape the same targets) for high availability, and shard scraping so each collector owns a slice of targets (by service/region), discovered via service discovery. Federate: local collectors keep full-resolution data, and a global layer scrapes aggregated rollups from them for a fleet-wide view. Cluster the alert manager so alerts survive a node loss. And keep cardinality in check — labels with unbounded values (user/request ids) will blow up the TSDB.
HA + federation: HA: duplicate collectors so no single scraper is a blind spot. Sharding: split targets across collectors to scale. Federation: a hierarchy where a global view aggregates from regional collectors. The monitoring system must be more reliable than the things it monitors.
Step 8 · The sharp edges
Alert storms & monitoring the monitor
Monitoring's own failure modes are brutal: one root cause fires a thousand alerts (an alert storm), a metric's cardinality explodes, and — worst — the monitoring system itself goes down silently and you never find out.
Tame alert storms with grouping, inhibition (a "datacenter down" alert suppresses its 500 child alerts), and rate-limited routing. Enforce cardinality limits and reject/relabel high-cardinality labels. Set sane thresholds + FOR durations to cut false pages, and alert on symptoms (user-facing SLOs) over causes. Crucially, monitor the monitor: a separate, independent dead-man's switch that pages you if the pipeline stops producing data or the heartbeat alert stops firing.
Design for the unhappy path: Storm → group + inhibit. Cardinality → cap + relabel. Noise → symptom-based alerts with FOR. Monitor down → an external dead-man's switch. The cruelest failure is the one where your monitoring dies quietly — so its liveness must be watched from outside itself.
You did it
You just designed a monitoring system.
- SSH-and-grep is reactive, unscalable and historyless — measure continuously and alert automatically.
- Instrument with counters, gauges and histograms plus labels; watch the golden signals.
- Prefer pull (scrape /metrics) for control + free liveness; add a push gateway for ephemeral jobs.
- Store the sample firehose in a time-series database — the compressed, range-queried heart.
- Split alerting: a rule evaluator fires (FOR a duration); an alert manager dedupes, groups, routes.
- Dashboards + a query language turn detection into diagnosis across labels.
- Scale with HA + sharded collectors and federation; cap cardinality; and monitor the monitor.