See inside
the request.

LLM systems are hard to debug for three reasons: they're non-deterministic (the same input can give different outputs), multi-step (a single request may fan out into retrievals, tool calls, and several model calls), and expensive (every step costs tokens). When something goes wrong — a slow response, a bad answer, a surprise bill — the logs of a normal service tell you almost nothing about which step caused it.

Observability is the fix: instrument the system so every step records what it did, how long it took, and what it cost, all linked into one inspectable timeline per request. It borrows distributed tracing from microservices — the same trace-and-span model — and applies it to the AI call graph. With it, "the request was slow" becomes "the retrieval span took 4 seconds," and "the bill doubled" becomes "this tool span is calling the expensive model twice." You stop guessing and start reading.

A span is one timed unit of work — a single model call, a tool execution, a retrieval — with a name, a start time, a duration, and metadata: tokens, cost, inputs, outputs, errors. A trace is the whole request, made of spans arranged in a tree: a root span for the request, child spans for each step it triggers, nested as deep as the call graph goes.

The tree captures both structure (what called what — the parent/child links) and timeline (when each span ran and for how long). That's exactly what you need to debug a complex agent: you can see the shape of the run and the timing at once. Rendered as a waterfall — spans as horizontal bars on a time axis — a trace makes bottlenecks visible at a glance, and the metadata on each span lets you attribute cost and errors to the precise step that caused them.

Here's the mistake that trips up almost everyone reading their first trace: adding up all the span durations to get "total time." That's wrong whenever any steps run in parallel. If two retrievals fire at once, each taking 2 seconds, they don't cost 4 seconds of wall-clock time — they overlap and cost 2. Summing durations overcounts exactly the parallel work.

The true end-to-end latency is the critical path: the time from the request starting to its last span finishing — equivalently, the longest chain of dependent spans. Only the spans on that path determine how fast the request is; work that runs off the critical path, in parallel, is effectively free for latency (though not for cost). This flips your optimization instinct: to make a request faster, shorten the critical path — parallelizing an off-path step, or speeding up a span that isn't the bottleneck, changes nothing. Cost is the opposite story, which the math makes precise.

Give each span a start time s, duration d, and cost c. End-to-end latency is when the last span finishes — the max end time — not the sum of durations:

Parallel spans overlap, so their durations don't add — the wall-clock time is the critical path, always at most the naive sum.

Cost, by contrast, does add up: you pay for every span's tokens whether or not it ran in parallel with another:

Latency = max end time; cost = sum. Two spans that overlap cut your latency in half versus running them in series, but cost the same. The runnable version below computes latency, cost, and the bottleneck from a span list.

Good traces come from logging the right fields on every span:

With these you can do the three things observability is for: debug (replay the exact request and see which span failed), control cost (sum tokens per span and find the expensive step), and improve quality (feed traced inputs/outputs into evals). Aggregate across many traces and you get the operational view: p50/p95 latency, cost per request, error rates by span type. And observability closes the loop with the harness — the same step logging a harness needs for its bounds is what populates the trace.

The headline mistake is summing durations for latency — it makes parallel systems look slower than they are and sends you optimizing spans that aren't on the critical path. Read latency as the critical path and cost as the sum; they are genuinely different quantities. A related trap is averaging latency: users feel the tail, not the mean, so track percentiles (p95, p99). A p50 of 1s hides a p99 of 20s that's making a fraction of users miserable.

Two more. Logging prompts and responses is logging data — traces often contain user PII and secrets, so redact or access-control them; observability must not become a leak. And sampling: full tracing of every request at scale is expensive, so sample — but always keep the errors and the slow tail, which are the traces you actually need. Done right, observability turns an opaque, non-deterministic system into one you can reason about — and it's the prerequisite for every other kind of improvement, because you can't fix what you can't see.