Latency Budget Builder
Assemble a chat request stage by stage — network, gateway, retrieval, rerank, prefill, decode — and watch the waterfall: time to first token, full response time, a stacked bar of where the milliseconds went, and which stage to attack first when the budget blows.
The two numbers users feel
Every chat request has one latency users obsess over and one they barely notice. Time to first token — the blank-box wait before anything appears — is the obsession: everything from the network hop to the last prefill token counts toward it. Decode time is the other 80% of the clock that nobody minds, because streaming turns it into reading. Budgets that treat "response time" as one number optimize the wrong thing; this tool splits them the way users experience them.
Prefill: the silent budget-killer
Slide the prompt to 50K tokens and watch the waterfall. Prefill processes every input token before the first output token can exist, so big RAG contexts translate directly into dead air — often dwarfing retrieval, rerank and network combined. The two rescues: prompt caching (the stable prefix — system prompt, tools, history — prefills once and is reused) and retrieval discipline (five relevant chunks beat thirty maybe-relevant ones on quality AND latency).
Reading the waterfall
The stacked bar answers the only question that matters: which stage do I attack first? Network dominating? Move compute to the edge or cut round trips. Retrieval? Approximate indexes, fewer candidates, parallel keyword+vector. Rerank? Shrink the shortlist or distill the cross-encoder. Prefill? Cache and trim. Decode? Smaller model, speculative decoding — or just accept it, because streaming already hides it.
Budget the tail, not the mean
A budget built on average stage times is a budget you miss daily. Queueing, cold caches and hot shards inflate the tail, and users touch enough requests that someone is always in it. Feed this tool your p99 stage numbers; if the p99 waterfall fits the budget, the median is a rounding error.
How it works
- TTFT = network + gateway + retrieval + rerank + prompt ÷ prefill-rate.
- Total = TTFT + output ÷ decode-rate; streaming hides the decode.
- The stacked bar exposes the stage to attack first.
- Under 500 ms feels instant; over 2 s feels broken.
Frequently asked questions
What counts toward time to first token?
Everything before streaming starts: the network round trip, gateway/auth, retrieval and reranking, and prefill (processing the whole prompt). Decode is excluded — once tokens flow, users read along. TTFT is the blank-box wait, and it is the number users actually feel.
Why does a big prompt hurt TTFT so much?
Prefill processes every prompt token before the first output token can exist. A 50K-token RAG prompt at 4,000 tokens/sec of prefill is 12.5 seconds of dead air before anything else even counts. Prompt caching rescues the stable prefix; trimming retrieval rescues the rest.
What are realistic prefill and decode speeds?
Order of magnitude for hosted frontier models: prefill in the low thousands of tokens/sec (cached prefixes effectively much faster), decode around 30–150 tokens/sec per request depending on model size and load. Small local models can beat both; giant reasoning models undercut both.
Should I budget the average or the p99?
The p99. Users touch several requests per session, so someone hits your tail constantly. Build the waterfall with p99 stage numbers (retrieval and queueing inflate most under load) and the average will take care of itself.