Time to first token (TTFT)
How long you stare at nothing before the first character appears. The wait before the model starts typing, not how long it takes to finish.
See it
What it is
TTFT measures one specific gap: request sent, first output token received. Pick that boundary and write it down, because 'received by my server' and 'painted in the browser' drift apart by hundreds of milliseconds and a dashboard mixing the two is worthless. Everything after the first token is throughput (tokens per second). Users judge the two completely differently. Nobody counts your tokens per second, but everyone feels three seconds of a blank box, and past roughly ten seconds they assume it broke and hit send again.
Prefill, the model reading your whole prompt before it can write a word, is usually a big slice, which is why TTFT climbs with input length. It is rarely the only slice. Queueing behind other traffic, gateway routing, network round trips, your own retrieval and reranking, and a reasoning budget can each dominate on a given request, and guessing which one is a good way to optimise the wrong thing. Time each stage, then fix what is actually slow: trim the context, cache the stable prefix, route simple requests to a smaller model, and run genuinely independent stages in parallel rather than in a serial chain.
Two traps. Reasoning models spend their thinking budget before the visible answer starts, so TTFT can be tens of seconds by design: show the thinking, or show progress. And averages lie here. Track p95 and p99, because the tail is where people rage-refresh.
Ask AI for it
Instrument and reduce time to first token in this AI feature. Define TTFT explicitly as request sent to first output token received, record it separately from total completion time, log p50/p95/p99, and surface them in the dashboard. Add a separate timing for each stage so the blame is attributable: queue wait, retrieval, reranking, moderation, provider prefill, and first token. Only then optimise. Parallelise stages that are genuinely independent of each other, but keep retrieval and every mandatory safety check ahead of generation rather than moving them off the critical path, since an ungrounded or unmoderated answer is not a faster answer. Put the stable system prompt and few-shot examples at the front of the prompt so they can be cached, route short or simple requests to a smaller model, and render a skeleton or status line immediately on submit so the wait is never a blank box.