Wiki / concepts / wiki

Time to first token

The wait before a model starts answering, mostly prefill: the model reading the whole prompt and context. Tokens-per-second figures in benchmarks measure generation after that wait, so at long context they can hide most of the delay.

Kai's example (Sept 2026): one user's software reported 57 tokens per second, but measured end to end at 8,000 tokens of context it was 3 tokens per second, because prefill took 94% of the response time. Switching inference servers, with the same machine, model and quantization, cut prefill from 49 seconds to 1.7, and generation speed didn't change. Another user running agents on an older Ultra-chip machine went from a minute or two to under 10 seconds. Kai's advice is to change the server before buying hardware.

Related: memory-bandwidth-bound-generation, params-vs-context-memory.

Source: report

Linked from

Memory-bandwidth-bound generation