Time to First Token
also: TTFT
How long until the first token of the reply appears — the latency number chat users actually feel.
TTFT spans queueing plus prefill (processing the prompt) — everything before streaming starts. Once tokens flow, users read along happily; before that, they stare at a blank box. Prompt/prefix caching, admission control and shorter contexts buy TTFT; batching efficiency buys tokens/sec. Report both, optimize TTFT first.
Worked example: TTFT (time to first token) is the latency until the first output token appears, dominated by prompt processing (prefill), so a 10k-token prompt has a much higher TTFT than a 100-token one on the same model. Gotcha: TTFT drives perceived responsiveness — users forgive slow typing but not a long stall — so streaming plus prompt-prefix caching (skipping re-encoding a repeated prompt head) are the main levers.