Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
Latency Budget Builder
Assemble a chat request stage by stage — network, gateway, retrieval, rerank, prefill, decode — and watch the waterfall: time to first token, full response time, a stacked bar of where the milliseconds went, and which stage to attack first when the budget blows.
A chat request has two latencies: the wait before the first token appears, which users feel, and the streaming that follows, which they mostly read through. This builder lays out every stage of a retrieval-augmented LLM request as a waterfall, so you can see which stage owns the time to first token (TTFT) and what the whole response costs.
What it computes
prefill_ms = prompt_tokens ÷ prefill_tok_per_s × 1,000
decode_ms = output_tokens ÷ decode_tok_per_s × 1,000
TTFT = network + gateway + retrieval + rerank + prefill_ms
total = TTFT + decode_ms- network is the client-to-edge round trip, gateway covers auth, routing and rate limiting, retrieval is the vector and keyword search, and rerank is a cross-encoder pass over the shortlist. All four are entered in milliseconds.
- prefill is the model reading the whole prompt before it can emit anything, so it scales with prompt size.
- decode is generating the answer one token at a time. It is left out of TTFT because the output streams.
The bottleneck readout names the largest of the five stages before streaming. The verdict under TTFT has four bands: up to 500 ms “feels instant”, up to 1,000 ms “feels responsive”, up to 2,000 ms “noticeably slow to start”, and anything above “users will assume it broke”. The bar shows each stage’s share of the full response.
A worked example
The defaults model a RAG chat turn: a 60 ms round trip, 15 ms of gateway, 80 ms of retrieval, 50 ms of rerank, a 4,000-token prompt prefilled at 4,000 tokens per second, and a 400-token answer decoded at 60 tokens per second.
| Stage | Calculation | Time |
|---|---|---|
| Network | input | 60 ms |
| Gateway | input | 15 ms |
| Retrieval | input | 80 ms |
| Rerank | input | 50 ms |
| Prefill | 4,000 ÷ 4,000 × 1,000 | 1,000 ms |
| Time to first token | sum of the five | 1,205 ms |
| Decode | 400 ÷ 60 × 1,000 | 6,667 ms |
| Full response | TTFT + decode | 7.9 s |
TTFT lands at 1,205 ms, in the “noticeably slow to start” band, and the bottleneck is Prefill, with 1,000 of those 1,205 ms. Decode is 84.7% of the full response, but streaming hides most of it. Halve the prompt to 2,000 tokens, or double prefill speed to 8,000 tokens per second, and prefill drops to 500 ms, putting TTFT at 705 ms: “feels responsive”. Grow the prompt to 20,000 tokens instead and prefill alone takes 5 seconds.
Assumptions and limits
- Stages run in series. The model adds every stage. If your app overlaps work, for example running keyword and vector search in parallel inside retrieval, enter the overlapped time, not the sum.
- Constant token rates. Real prefill slows per token as context grows, because attention cost rises with length, and decode speed varies with load and batch size. Prompt caching makes the cached part of a prompt far cheaper to prefill; the tool can only approximate that through a higher prefill rate.
- No queueing stage. Time spent waiting for capacity at the model provider is not modeled. Fold it into the gateway number, or production will surprise you.
- One round trip. A new connection needs extra round trips for the TCP and TLS handshakes, so a cold first request can cost two or three times the RTT.
- Visible tokens only. A reasoning model generates its thinking before the answer. If you hide that thinking, its decode time belongs in TTFT, which the tool does not add.
How to use the result
- Attack the bottleneck first. Prefill: cache the stable prefix (system prompt, tools, history) and send fewer retrieved chunks. Network: serve closer to users and cut round trips. Retrieval: approximate indexes and fewer candidates. Rerank: a shorter shortlist or a smaller cross-encoder.
- Turn a TTFT target into a prompt budget. Subtract the fixed stages from the target and multiply by the prefill rate. At the defaults, a 1,000 ms target leaves 795 ms for prefill, or 3,180 prompt tokens; a 500 ms target leaves room for 1,180.
- Budget the tail, not the mean. Enter p95 or p99 stage times. Summing tail values is pessimistic, since stages rarely all hit their tail on one request, and that is the safe side to err on.
- Watch the total for non-streaming callers. Agents, tool calls and JSON parsers wait for the whole answer, so for them the 7.9 s total matters more than TTFT, and decode speed or a shorter answer is the lever.
How it works
- TTFT = network + gateway + retrieval + rerank + prompt tokens ÷ prefill rate.
- Total = TTFT + output tokens ÷ decode rate; streaming hides the decode.
- The bottleneck is the largest of the five stages before streaming starts.
- Up to 500 ms feels instant; over 2 s users assume it broke.
Frequently asked questions
What counts toward time to first token?
Everything that happens before the first output token is streamed: the network round trip, gateway work such as auth and rate limiting, retrieval and reranking in a RAG app, and prefill, where the model reads the entire prompt. Decode is excluded, because once tokens flow the user is reading. With this tool’s defaults those stages add up to 60 + 15 + 80 + 50 + 1,000 = 1,205 ms. TTFT is the blank-screen wait, and it is the latency users notice most in a chat interface.
Why does a big prompt hurt TTFT so much?
Prefill processes every prompt token before the first output token can exist, so prefill time grows with prompt length. At 4,000 tokens per second, a 4,000-token prompt costs one second and a 50,000-token RAG prompt costs 12.5 seconds of dead air, more than every other stage combined. Two fixes help most: prompt caching, so the stable prefix is not reprocessed on every turn, and retrieval discipline, sending a few relevant chunks instead of many marginal ones, which tends to help answer quality as well.
How do I measure prefill and decode speed for my model?
Time a few real requests with streaming on. Decode speed is the output tokens divided by the time between the first and the last token. For prefill, take the TTFT measured at the API, subtract the network round trip, and divide the prompt tokens by what remains. Try both a short and a long prompt, because queueing and fixed overheads distort a single small request. Measure at your real traffic level, since both rates fall when the provider or your own GPUs are busy.
Should I budget the average or the p99?
Budget the tail. A user sends many requests in a session, so a slow p99 is something regular users meet, not a rare event. Build the waterfall from p95 or p99 stage times, especially retrieval and any queueing, which inflate most under load. Adding up per-stage tail values is pessimistic, since the stages rarely all hit their worst case on the same request, but that is the safe side to err on. If the tail waterfall fits the target, the median will too.
What is a good time to first token for a chat app?
This tool’s bands are a reasonable guide: up to 500 ms feels instant, up to 1 second feels responsive, up to 2 seconds is noticeably slow to start, and beyond that users start to assume something broke. They line up with long-standing usability guidance that about one second is the limit for keeping a user’s flow of thought uninterrupted. Voice interfaces need to be faster still, because silence in a spoken conversation is far more noticeable than a blank text box.
When does full response time matter more than TTFT?
Whenever nothing is shown until the answer is complete. Agents that call tools, pipelines that parse JSON and batch jobs all wait for the last token, so for them decode dominates: at the defaults, decode is 6,667 of the 7,872 ms total. The levers there are shorter outputs, a faster or smaller model, and techniques such as speculative decoding. In a streaming chat interface the full time matters less, because the user starts reading as soon as the first tokens arrive.