Observability, and
the three ways to see.

Monitoring watches for failure modes you anticipated: predefined dashboards and alerts for "CPU high," "disk full," "error rate up." It answers known-unknowns — questions you thought to ask in advance. Invaluable, but it can only catch what you predicted.

Observability is a property: the ability to understand your system's internal state from its external outputs, and to ask new questions without shipping new code. It's for the unknown-unknowns — the novel failure you never imagined, where you need to slice and pivot the data in ways you didn't pre-plan ("why are these specific users in this region seeing slow checkouts only when they use coupon X?"). Monitoring tells you something is wrong; observability lets you figure out what, even if you've never seen it before.

They complement each other precisely because each has a blind spot. A metric alert ("p99 latency spiked") tells you something's wrong but not where. A trace of a slow request reveals which service caused it. The trace links to the relevant logs from that service, which tell you exactly what happened. Metric → trace → log is the canonical debugging path.

A log line like "payment failed for user 42 after 3011ms" is human-readable but hard to query. Structured logging emits logs as data — JSON with typed fields: level=ERROR, service=payments, user_id=42, latency_ms=3011, plus a trace_id. Now you can filter and aggregate precisely ("all payment errors over 3s in eu"), and full-text search still works on the message.

The single most valuable field is a correlation id — a trace_id or request_id — that ties one request's log lines together across services. Without it, logs are a million disconnected sentences; with it, you can pull the whole story of a single request. Structure early: a field you set at the source is reliable; one you regex out of free text later is fragile.

Three metric types cover most needs: a counter only goes up (requests, errors), a gauge goes up and down (memory, queue depth), and a histogram buckets a distribution (request latency, so you can compute p50/p99). Each metric carries labels (service, region, endpoint) to slice it.

Here's the trap that topples metrics systems: cardinality. A metrics store's cost scales with the number of distinct time series, which is the number of unique label-value combinations — not the number of data points. Attach a high-cardinality label like a user_id, request_id, or full URL, and you mint a new series per value, exploding into millions and crushing the store. The rule: labels must be bounded, low-cardinality sets (region, status, method) — never unbounded ids. Put those in logs or traces, which are built for high cardinality.

In a microservices system, one user request touches a dozen services. A trace follows it end to end. Each service records a span — its slice of work with timing — and all spans from the request share one trace id, linked by parent-child relationships into a waterfall that shows exactly where the time went.

The magic that makes it work is context propagation: the trace id is passed in request headers from service to service (the W3C traceparent standard), so every service's spans join the same trace. Miss it at one hop and the trace breaks. Because full tracing is expensive, systems sample — ideally tail-based, keeping the interesting traces (errors, slow ones) and a baseline of normal ones. Traces answer the question metrics and logs can't: which service, on this exact request, was the problem?

You can measure infinite things; measure the ones that reflect user experience. The golden signals are a proven default: latency (how slow), traffic (how much demand), errors (how many fail), and saturation (how full your resources are). Instrument those everywhere and most incidents become visible.

Formalize the important ones as reliability targets. An SLI (service level indicator) is a measured health signal — say, the fraction of successful requests. An SLO (objective) is a target for it — e.g. 99.9% success. The gap below 100% is your error budget: the failure you're allowed to spend. Within budget, ship features fast; if you've burned it, slow down and stabilize. SLOs turn "be reliable" into a number both engineers and product can reason about — the core of SRE practice.

Good observability is wasted if the alerts are noise. Two principles keep alerts useful. First, alert on symptoms, not causes: page on what users feel (error rate up, latency past your SLO) rather than every internal cause (a single CPU spike). Symptom alerts fire when something actually matters, and they don't multiply — one root cause is one page, not fifty.

Second, fight alert fatigue: if a "critical" alert doesn't need immediate human action, it shouldn't page. Every noisy page erodes trust until real ones get ignored. Use SLO-based (burn-rate) alerts so you're paged when you're actually spending your error budget too fast, group and deduplicate related alerts, and reserve paging for the actionable. A quiet, trustworthy pager beats a loud, ignored one.

The three pillars pay off when they're connected. A real incident flows across them: an SLO/metric alert fires (error rate breached) → you open a trace of a failing request and see the payments service is timing out → you jump to the logs for that service and trace id and find the exact exception. Minutes, not hours — because a trace id threads the metric, the trace, and the logs into one story.

That's the real goal of observability: not three separate tools, but a connected picture where you can pivot fluidly from "something's wrong" to "here's exactly what and where." Instrument the golden signals, log with structure and a trace id, trace across services, keep metric cardinality sane, and alert on symptoms — and 3am debugging becomes a guided walk instead of a guessing game.