The big idea
What is distributed tracing?
A user says "checkout is slow." The request touched the gateway, auth, cart, pricing, inventory, payments and three databases — across a dozen services owned by different teams. Metrics say "p99 is up"; logs are a million disconnected lines. Which hop actually made this request slow? Neither can tell you.
Distributed tracing follows a single request end to end. Each service records a span (its slice of work, with timing), all tagged with one shared trace id, and the system stitches them into a waterfall that shows exactly where the time went. It's the third pillar of observability — metrics say something is wrong, logs say what happened, traces say where.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end kill the collector and disable sampling to see why tracing stays off the hot path and can't keep everything. Hit Begin.
Step 1 · Why logs & metrics fall short
The needle across a haystack of services
You have per-service metrics and mountains of logs. Yet you can't answer "why was this specific request slow?" What's fundamentally missing?
Design decision: Metrics + logs exist. Why can't they explain one slow cross-service request?
The call: Metrics aggregate away the single request; logs aren't linked across services. — A metric like p99 is a statistic over many requests — it can't point at one. Logs record events per service with no shared identifier tying one request's lines together across a dozen services. Tracing adds exactly that missing thread.
Metrics aggregate — they tell you p99 rose but not which request or where. Logs are per-service and disconnected — no shared thread ties one request's lines together across services. Tracing supplies the missing primitive: a single trace id that follows one request through every service, turning scattered events into one causal timeline.
The three pillars: Metrics: aggregate health (cheap, always-on). Logs: discrete events (detailed, per-service). Traces: the causal path of one request across services. They complement each other — an alert (metric) leads you to an exemplar trace, which links to the relevant logs.
Step 2 · The unit of work
Spans and traces
To reconstruct a request's journey you need a data model. What's the smallest useful record, and how do many of them add up to one request's story?
Design decision: What's the right data model for one request across services?
The call: A span per operation (name, start, duration, tags, parent id), many spans sharing a trace id form a trace. — Each unit of work is a span with timing, tags and a parent-span id; all spans from one request carry the same trace id, so they assemble into a tree/waterfall showing the full path and where time was spent.
The unit is a span: one operation (an HTTP handler, a DB query) with a name, start time, duration, tags, and a parent span id. Every span from one request shares a single trace id. Assemble all spans with the same trace id by their parent links and you get a trace — a tree/waterfall of the entire request, with each bar's width showing where the time went.
Span tree = trace: A trace is a directed tree of spans: the root is the entry request; each child is a downstream call. Parent-child links reconstruct causality; start+duration reconstruct timing. The waterfall you stare at during an incident is just this tree laid out on a time axis.
Step 3 · Carry the id across the wire
Context propagation
Service A calls B calls C, each in a different process (or language). For all their spans to share one trace id, that id has to travel with the request across every network hop. How does the id get from A to C?
Design decision: A→B→C are separate processes. How do all their spans share one trace id?
The call: Propagate the trace context (trace id + parent span id) in request headers on every call. — When A calls B, it injects the trace id and its own span id into standard headers (W3C traceparent); B reads them, makes its span a child, and injects updated context when calling C. The id rides the request through every hop, so all spans link into one trace.
Propagate the trace context in request headers. When A calls B, it injects the trace id and its current span id into standard headers (the W3C traceparent format). B reads them, starts its span as a child, and injects updated context when it calls C. The identity rides in-band with the request across every hop and language, so every span links into one trace with no central coordination.
In-band context: Propagation is the make-or-break detail: miss it at one hop and the trace breaks into disconnected fragments. Standard formats (W3C Trace Context, B3) and auto-instrumentation libraries handle injection/extraction at RPC boundaries so it "just works" across a polyglot fleet.
Step 4 · Where spans come from
Instrumentation
Someone has to actually create spans, start the timer, attach tags, and read/write the context headers — in every service, ideally without developers hand-writing it everywhere. How do spans get produced?
Use instrumentation libraries (OpenTelemetry SDKs). Auto-instrumentation hooks common frameworks and clients (HTTP servers, gRPC, DB drivers) to create spans and propagate context automatically at the boundaries; developers add manual spans and tags around important business logic. Spans are emitted asynchronously to a local agent so instrumentation adds negligible latency to the request itself.
Auto + manual: Auto-instrumentation gives you the whole request skeleton (every inbound/outbound call) for free; manual spans add the meaningful in-process detail ("cache lookup", "risk check"). Emitting spans off-thread keeps the observed system fast — you must not slow the request to trace it.
Step 5 · Get spans out of the app
Agent → collector ingest
Spans are produced inside thousands of short-lived processes. You need to get them to central storage without coupling the app to the pipeline, adding request latency, or losing the app if the tracing backend hiccups.
Design decision: How do you ship spans off thousands of processes without risking the app?
The call: Services emit spans fire-and-forget to a local agent, which forwards to a scalable collector tier. — A local agent (sidecar/daemon) buffers spans off the request path and ships them to a horizontally-scaled collector that validates, batches and forwards to sampling/storage. If the backend is down, spans buffer and then drop — the app never notices.
Two decoupling layers. Services emit spans fire-and-forget to a local agent (sidecar/daemon) that buffers off the request path. Agents forward to a horizontally-scaled collector tier that validates, batches, and hands spans to sampling and storage. Because emission is asynchronous and buffered, a tracing-backend outage causes dropped traces at worst — never a slowed or broken request.
Off the critical path: The golden rule of tracing: observing the system must not endanger it. Async emit → local agent buffer → collector fleet means backpressure or failure downstream degrades trace completeness, not application latency or availability. OpenTelemetry's Collector is exactly this tier.
Step 6 · You can't keep it all
Sampling: head vs tail
At scale, tracing every request would multiply your ingest, storage and cost by orders of magnitude — and 99% of traces are boring, fast, successful requests nobody will ever look at. But the 1% you do want are exactly the errors and slow outliers. How do you keep the interesting traces cheaply?
Design decision: Tracing 100% is far too expensive, but you want the errors/slow ones. How do you sample?
The call: Tail-based sampling: buffer each full trace, then keep errors, slow ones, and a small random baseline. — Head sampling decides up front (cheap, but can't know if a trace will error). Tail sampling waits until the trace is complete, then keeps it if it errored or was slow, plus a small random sample of normal traces for baseline — capturing the interesting minority while dropping the boring majority.
Sample. Head-based sampling decides at the start (cheap, but you don't yet know if the request will error or be slow). Tail-based sampling buffers each complete trace and then keeps the interesting ones — errors, high latency — plus a small random baseline of normal traces, dropping the boring majority. Tail sampling costs more to run (you buffer everything briefly) but keeps exactly the traces you'll actually investigate.
Bias toward the interesting: The whole game is spending your storage budget on traces someone will look at. Tail sampling can say "keep every trace that errored or took >1s, plus 1% of the rest." A consistent sampling decision must be propagated so an entire trace is kept or dropped together — never half a trace.
Step 7 · Store for trace lookup
The trace store
Kept spans must be stored so you can (a) pull an entire trace by its id instantly, and (b) find traces by service, duration, error, or tag. Those are different access patterns. How do you store spans?
Store spans indexed by trace id so all spans of a trace are fetched together to render the waterfall, and maintain secondary indexes (service, operation, duration, status, tags) so you can search "slow checkout traces with error=true in the last hour." Use a scalable backend (Cassandra/Elasticsearch/columnar) with retention — traces are high-volume and short-lived in usefulness, so keep them days, not forever.
Two access patterns: Get-by-id: assemble one trace's spans (the waterfall). Search: find candidate traces by attributes. The store must serve both — a trace-id partition for assembly, plus inverted indexes for discovery — under heavy write load and bounded retention.
Step 8 · The sharp edges
Clock skew, bias & cardinality
Real traces get messy: servers' clocks disagree (so a child looks like it started before its parent), sampling can bias your view, high-cardinality tags blow up storage, and async work (queues, batch) breaks simple parent-child timing.
Handle clock skew by reconstructing order primarily from causal parent-child links (not raw timestamps) and correcting obvious inversions. Remember sampling makes traces not statistically representative — use metrics for rates, traces for exemplars. Cap tag cardinality (no unbounded ids in span tags). Model async flows with span links (a queue consumer links back to the producer's span) rather than pretending it's a synchronous child.
Design for the unhappy path: Skewed clocks → trust causal links over timestamps. Sampled data → don't compute rates from traces. Exploding tags → cardinality caps. Async hops → span links, not fake parents. The clean span tree needs these corrections to match messy reality.
You did it
You just designed a distributed tracing system.
- Metrics aggregate and logs are disconnected — tracing adds a per-request thread across services.
- The model: spans (timed operations) sharing a trace id, linked by parent id into a trace tree.
- Propagate trace context in headers (W3C traceparent) so the id rides in-band across every hop.
- Instrument with auto + manual spans, emitted asynchronously so tracing never slows the request.
- Ship spans fire-and-forget via local agents to a scalable collector — off the critical path.
- Sample (tail-based) to keep errors and slow traces plus a baseline, dropping the boring majority.
- Store indexed by trace id (assemble) + secondary indexes (search), with bounded retention.