The whole design, in writing
Learn system design by building a distributed tracing system like Jaeger, Zipkin or Honeycomb step by step. An interactive guide covering why logs and metrics can't explain a slow request across microservices, trace and span context propagation, in-process instrumentation, an agent + collector ingest path, tail-based vs head-based sampling, a store optimized for trace lookup, the query/waterfall UI, and the unhappy paths (clock skew, sampling bias, cardinality).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is distributed tracing?
A user says "checkout is slow." The request touched the gateway, auth, cart, pricing, inventory, payments and three databases — across a dozen services owned by different teams. Metrics say "p99 is up"; logs are a million disconnected lines. Which hop actually made this request slow? Neither can tell you.
Distributed tracing follows a single request end to end. Each service records a span (its slice of work, with timing), all tagged with one shared trace id, and the system stitches them into a waterfall that shows exactly where the time went. It's the third pillar of observability — metrics say something is wrong, logs say what happened, traces say where.
What the new pieces do
- Servicesclient
- The microservices handling one user request as it hops from service to service. Each does a slice of the work and calls the next.
Step 1 · Why logs & metrics fall short
The needle across a haystack of services
You have per-service metrics and mountains of logs. Yet you can't answer "why was this specific request slow?" What's fundamentally missing?
Metrics + logs exist. Why can't they explain one slow cross-service request?
Retention isn't the issue. The gap is that metrics are aggregates (they lose the individual request) and logs are per-service and disconnected (no thread links one request's log lines across services).
A metric like p99 is a statistic over many requests — it can't point at one. Logs record events per service with no shared identifier tying one request's lines together across a dozen services. Tracing adds exactly that missing thread.
More dashboards still show aggregates. The missing primitive is a per-request identifier that connects every service's work on that one request into a single timeline.
Metrics aggregate — they tell you p99 rose but not which request or where. Logs are per-service and disconnected — no shared thread ties one request's lines together across services. Tracing supplies the missing primitive: a single trace id that follows one request through every service, turning scattered events into one causal timeline.
Why this piece earns its place
The cheap version of this idea already exists, and it’s worth saying why it isn’t enough: stamp a request id on every log line and grep for it across services. That buys you the thread, and it is genuinely better than nothing — but it gives you a list of events, not a shape. Log lines are instants; spans are intervals with a parent, so only the span model can tell you that inventory spent its time waiting on payments rather than working. No nesting, no per-hop duration, no way to see that two calls ran in parallel. And the expensive part is identical either way — you still have to carry that id through every hop in every language, which is the work of step 3. So you pay tracing’s hard cost and collect a fraction of its value. If you are going to thread the request anyway, thread something that has structure.
Step 2 · The unit of work
Spans and traces
To reconstruct a request's journey you need a data model. What's the smallest useful record, and how do many of them add up to one request's story?
What's the right data model for one request across services?
A single summary line can't show the nested timing of each hop or which downstream call was slow. You need a record per operation, linked into a tree.
Counters are metrics — they aggregate and lose the individual request's structure entirely. Tracing needs per-operation timed records that form a causal tree.
Each unit of work is a span with timing, tags and a parent-span id; all spans from one request carry the same trace id, so they assemble into a tree/waterfall showing the full path and where time was spent.
The unit is a span: one operation (an HTTP handler, a DB query) with a name, start time, duration, tags, and a parent span id. Every span from one request shares a single trace id. Assemble all spans with the same trace id by their parent links and you get a trace — a tree/waterfall of the entire request, with each bar's width showing where the time went.
Why this piece earns its place
Notice what this model does not give you: a moment when a trace is done. Spans are emitted independently by different processes, so a trace is not an object anyone writes — it is a bag of spans that agree on an id, assembled by whoever reads them. The root finishes last, because it cannot end until its children do, so the first spans you receive are the leaves, and the parent a span points at may not have arrived yet, or may never arrive at all. Two consequences follow. The read path has to render a forest with dangling parents rather than wait for a well-formed tree, or the incident page shows nothing at the moment you need it. And nothing downstream can truly ask “is this trace complete?” — the best available answer is “no new spans for a while”, which is a timeout, and every later decision that wants a whole trace inherits that uncertainty.
- spanone timed operation
- trace idshared across all spans
- parent idbuilds the tree
Step 3 · Carry the id across the wire
Context propagation
Service A calls B calls C, each in a different process (or language). For all their spans to share one trace id, that id has to travel with the request across every network hop. How does the id get from A to C?
A→B→C are separate processes. How do all their spans share one trace id?
Independent ids can't be reliably stitched after the fact — timing overlaps and shared fields are ambiguous. The id must be propagated with the request so causality is explicit, not guessed.
When A calls B, it injects the trace id and its own span id into standard headers (W3C traceparent); B reads them, makes its span a child, and injects updated context when calling C. The id rides the request through every hop, so all spans link into one trace.
A shared global lookup per hop adds latency and a dependency on the request path, and races under concurrency. Context travels in-band with the request via headers — no central coordination needed.
Propagate the trace context in request headers. When A calls B, it injects the trace id and its current span id into standard headers (the W3C traceparent format). B reads them, starts its span as a child, and injects updated context when it calls C. The identity rides in-band with the request across every hop and language, so every span links into one trace with no central coordination.
Why this piece earns its place
Riding the id in-band has a consequence at the edge: traceparent is just a header, and a public endpoint that trusts it lets any caller choose your trace ids. That is an unauthenticated write into your observability system — someone can pin every request onto one id until that trace is unopenable, or supply ids that collide with real ones. So apply the ordinary rule for input crossing a trust boundary: internal hops extract the context, the public edge does not. At the edge you start a fresh trace and keep the caller’s claimed id as a tag, so a real client-side SDK still correlates without handing strangers your id space. The second edge case is duller and far more common: proxies, load balancers and gateways drop headers they do not recognise, so the hop where a trace fragments is often not a service at all — you find it by hunting for traces whose root has no children.
- traceparentheader format
- inject/extractat each hop
- in-bandno central lookup
What the new pieces do
- Trace Contextservice
- The trace id + parent span id passed in headers from service to service, stitching all the spans into one trace.
Step 4 · Where spans come from
Instrumentation
Someone has to actually create spans, start the timer, attach tags, and read/write the context headers — in every service, ideally without developers hand-writing it everywhere. How do spans get produced?
Use instrumentation libraries (OpenTelemetry SDKs). Auto-instrumentation hooks common frameworks and clients (HTTP servers, gRPC, DB drivers) to create spans and propagate context automatically at the boundaries; developers add manual spans and tags around important business logic. Spans are emitted asynchronously to a local agent so instrumentation adds negligible latency to the request itself.
Why this piece earns its place
The knob this step introduces is granularity, and both ends of it hurt. Instrument too little and the waterfall has gaps: one span covers a service’s whole handler, so all you learn is that the time went somewhere inside it, which is where you started. Instrument every function and a single request emits hundreds of spans, each one multiplying through the buffer, the collector, the sampler and the bill — and the waterfall becomes a wall nobody can read. Where I would set it: a span at every network boundary automatically, because that is where the interesting time usually hides and it costs nothing to decide, plus manual spans only where someone has an actual hypothesis about in-process time. And be honest about the cost you do pay inside the app. Emission is asynchronous, but creating a span is not free — it allocates, reads the clock and copies tags — so per-request overhead scales with span count, not with how far away the backend is.
Step 5 · Get spans out of the app
Agent → collector ingest
Spans are produced inside thousands of short-lived processes. You need to get them to central storage without coupling the app to the pipeline, adding request latency, or losing the app if the tracing backend hiccups.
How do you ship spans off thousands of processes without risking the app?
A synchronous write to the backend on the request path adds latency and couples the app's availability to the tracing store — a tracing outage would slow or break production. Emission must be async and buffered.
That's neither timely nor reliable. You want a local buffering agent that forwards to a scalable collector tier, fully decoupled from the request.
A local agent (sidecar/daemon) buffers spans off the request path and ships them to a horizontally-scaled collector that validates, batches and forwards to sampling/storage. If the backend is down, spans buffer and then drop — the app never notices.
Two decoupling layers. Services emit spans fire-and-forget to a local agent (sidecar/daemon) that buffers off the request path. Agents forward to a horizontally-scaled collector tier that validates, batches, and hands spans to sampling and storage. Because emission is asynchronous and buffered, a tracing-backend outage causes dropped traces at worst — never a slowed or broken request.
Why this piece earns its place
The thing to get right here is the buffer, because an unbounded one turns a tracing outage into an application outage: spans pile up in the process’s own memory until it dies, which is exactly the coupling this step exists to remove. So the buffer is fixed-size and it drops — and then the real question is whether anyone finds out. Drop silently and tracing lies in the most dangerous way. During an incident the degraded service emits more spans and slower ones, so the buffer fills precisely while the traces you want are being produced, and a search that comes back empty reads as “nothing happened here”. Emit a dropped-span counter as an ordinary metric alongside everything else, so “we have no trace” can be told apart from “there was no problem”. The same bound belongs on the agent’s queue: the point of two layers is that neither of them may grow without limit.
- agentlocal buffer
- collectorscalable ingest
- fire-and-forgetno request impact
What the new pieces do
- Agentgateway
- A lightweight process on each host/pod. Buffers spans emitted by local services and forwards them onward, off the request path.
- Collectorservice
- Receives spans from all agents, validates and batches them, and hands them to sampling and storage.
Step 6 · You can't keep it all
Sampling: head vs tail
At scale, tracing every request would multiply your ingest, storage and cost by orders of magnitude — and 99% of traces are boring, fast, successful requests nobody will ever look at. But the 1% you do want are exactly the errors and slow outliers. How do you keep the interesting traces cheaply?
Tracing 100% is far too expensive, but you want the errors/slow ones. How do you sample?
Uniform random head sampling is cheap but blind — it keeps mostly boring traces and drops most errors and slow outliers, exactly the ones you need. You want the decision to favor interesting traces.
Time-window sampling gives a wildly biased, unrepresentative view and misses later incidents entirely. Sampling must be per-trace and biased toward what matters.
Head sampling decides up front (cheap, but can't know if a trace will error). Tail sampling waits until the trace is complete, then keeps it if it errored or was slow, plus a small random sample of normal traces for baseline — capturing the interesting minority while dropping the boring majority.
Sample. Head-based sampling decides at the start (cheap, but you don't yet know if the request will error or be slow). Tail-based sampling buffers each complete trace and then keeps the interesting ones — errors, high latency — plus a small random baseline of normal traces, dropping the boring majority. Tail sampling costs more to run (you buffer everything briefly) but keeps exactly the traces you'll actually investigate.
Why this piece earns its place
Tail sampling buys its fidelity with two costs the one-line description hides. The first is the decision window. You cannot judge a trace until its last span arrives, and a slow trace is by definition the late one — so the window has to outlast the latency you are hunting, or you decide on incomplete traces and systematically discard the outliers you built this for. Make it too long and the collector holds more in-flight traces in memory, which is the real ceiling on this tier. The second is easier to miss and harder to undo: every span of a trace must reach the same collector instance to be buffered together, so this tier can no longer be round-robin. It needs routing by trace id, which turns something you were treating as stateless into a hashed, rebalancing fleet — with hot instances behind heavy traces, and split decisions while the ring moves. That is the price of deciding late.
- headdecide up front, cheap
- taildecide after, keeps errors/slow
- 1%baseline of normals
What the new pieces do
- Samplerservice
- Decides which traces to keep. Tail-based sampling waits for the whole trace, then keeps errors and slow ones, dropping the boring majority.
Step 7 · Store for trace lookup
The trace store
Kept spans must be stored so you can (a) pull an entire trace by its id instantly, and (b) find traces by service, duration, error, or tag. Those are different access patterns. How do you store spans?
Store spans indexed by trace id so all spans of a trace are fetched together to render the waterfall, and maintain secondary indexes (service, operation, duration, status, tags) so you can search "slow checkout traces with error=true in the last hour." Use a scalable backend (Cassandra/Elasticsearch/columnar) with retention — traces are high-volume and short-lived in usefulness, so keep them days, not forever.
Why this piece earns its place
Every secondary index is paid for on the write path, and here the write path is the whole firehose while the read path is a handful of humans during an incident. Let that asymmetry decide the design: index narrowly, filter late. Service, operation, duration and status earn an index because nearly every search starts there. Arbitrary tags do not — and the version of this that quietly fails is the one where someone points the span stream at a search cluster with dynamic mapping and indexes every field anyone has ever attached. Then the index outgrows the data it points at, and the component that falls over in your busiest hour is the search tier rather than the store. Keep tags as payload on the span, narrow to candidate traces on the indexed fields plus a time range, and filter the rest after the fetch. Scanning candidates is cheap when a person is waiting, and it costs nothing on everything you wrote and nobody ever reads.
- by trace idassemble waterfall
- secondary indexsearch by tag/latency
- retentiondays, not forever
What the new pieces do
- Trace Storedata
- Stores spans indexed so an entire trace can be fetched by its id, and traces can be found by service, duration or tags.
Step 8 · The sharp edges
Clock skew, bias & cardinality
Real traces get messy: servers' clocks disagree (so a child looks like it started before its parent), sampling can bias your view, high-cardinality tags blow up storage, and async work (queues, batch) breaks simple parent-child timing.
Handle clock skew by reconstructing order primarily from causal parent-child links (not raw timestamps) and correcting obvious inversions. Remember sampling makes traces not statistically representative — use metrics for rates, traces for exemplars. Cap tag cardinality (no unbounded ids in span tags). Model async flows with span links (a queue consumer links back to the producer's span) rather than pretending it's a synchronous child.
Why this piece earns its place
Ordering from parent-child links is the right call, but be precise about what it cannot recover. Order is not duration. Two spans on hosts whose clocks disagree still carry absolute start times you cannot compare, and the number you most want in a latency hunt is exactly that comparison: the gap between a parent issuing a call and the child beginning work, which is network plus queueing. Trust the raw timestamps and that gap can come out negative. The way around it is to take both measurements from one clock — the parent’s span already spans the entire call, so parent-observed duration minus the child’s self-reported duration is the unaccounted time, obtained without ever subtracting one host’s clock from another’s. Attribute that remainder to the hop, not to either service. And treat any figure derived by subtracting timestamps across hosts as a hint: the UI should say so rather than drawing a confident bar.
What the new pieces do
- Query UIui
- The flame-graph/waterfall view: reconstructs the trace tree so you can see exactly which span made the request slow.
You did it
You just designed a distributed tracing system.
Everything you assembled, in order
- Metrics aggregate and logs are disconnected — tracing adds a per-request thread across services.
- The model: spans (timed operations) sharing a trace id, linked by parent id into a trace tree.
- Propagate trace context in headers (W3C traceparent) so the id rides in-band across every hop.
- Instrument with auto + manual spans, emitted asynchronously so tracing never slows the request.
- Ship spans fire-and-forget via local agents to a scalable collector — off the critical path.
- Sample (tail-based) to keep errors and slow traces plus a baseline, dropping the boring majority.
- Store indexed by trace id (assemble) + secondary indexes (search), with bounded retention.