Time to first token
Time to first token, or TTFT, measures the wait between sending a question to an AI model and seeing the first word of the answer come back. For an IBM i shop it is the number that says whether local AI on a Power 11 server with a Spyre Accelerator feels instant or feels broken, because it counts queue time and the whole prompt read, not just the writing.
What does time to first token measure?
Red Hat states it plainly: the time from when a request is sent to when the first token of the response is received. A token is a word or a piece of a word. So this is the silence a person stares at before anything appears on the screen.
Think of ordering food. Time to first token is the gap between handing over your order and the first plate landing on the table. It says nothing about how long the whole meal takes. It only measures the quiet at the start.
That quiet is what people judge. A shop running a model next to IBM i on a Power 11 server with a Spyre Accelerator will hear about this number first, long before anyone asks how many answers the card produces in an hour.
What is inside that wait?
Three things happen before the first word
- The request waits in lineThe serving engine admits a request only when there is room for it. A busy card means a longer wait here, and that waiting time counts toward the score.
- The card reads the promptThis step is called prefill. The card has to read every token of the question before it can write one token of the answer. A long prompt makes this step long.
- The first token comes outThe clock stops the moment the first piece of the answer reaches the caller. Everything after that belongs to the other speed numbers.
Source: the IBM Spyre plugin performance guide, which states that time to first token includes time spent waiting in the queue and the full prefill time up to the last chunked prefill step.
How is it different from the other speed numbers?
Four numbers get quoted together and they are not interchangeable. Time to first token is the only one a person feels while waiting for the screen to change.
The four latency numbers a serving run reports
| Number | What it measures | When it matters |
|---|---|---|
| Time to first token (TTFT) | Request sent, first word received | A person waiting at a screen |
| Inter-token latency (ITL) | The gap between two output words | Whether the answer stutters as it writes |
| Time per output token (TPOT) | The average time for each word after the first | Steady reading speed |
| End-to-end latency (E2EL) | Request sent, last word received | Batch jobs, reports and overnight work |
Sources: the Red Hat AI Inference Server key metrics documentation and the IBM Spyre plugin performance guide. IBM describes inter-token latency as the time gap between each generated token.
How do you make it smaller on Power 11?
Most of the fix is configuration, not more hardware.
- Keep chunked prefill on. IBM requires it for decoder models such as Granite, and it is what cut the long opening wait.
- Ask for fewer output tokens. The serving engine decides whether to admit a request using the prompt length plus the output length you asked for, so a huge output ceiling makes a request wait longer before it even starts.
- Do not over-stack concurrent users. IBM holds one container to 131,072 tokens across the whole batch, so pushing past that budget makes the wait jump instead of producing more answers.
- Use prefix caching. When many prompts start with the same text, the card can reuse that work instead of reading it again.
Prefill delay before the answer starts, on a large request
IBM figures from the IBM Spyre Accelerator for Power documentation. Chunked prefill arrived in the IBM Spyre Enablement Stack for Power 1.1 in March 2026.
Published sample numbers travel badly. Red Hat's own key metrics example reports a mean time to first token of about 194 milliseconds, but that run used a 1 billion parameter model on an NVIDIA GPU, not a Spyre card on Power 11. Measure your own model, your own prompt lengths and your own concurrency before promising anyone a number.
Related Terms
Inference serving
The software layer that takes requests for a model, packs them onto the hardware efficiently, and returns answers. It decides how much work one accelerator card can actually do.
PagedAttention
PagedAttention is the memory manager inside the vLLM serving engine. It hands out an AI model's working memory in small fixed blocks instead of one worst-case reservation, which is what lets one Spyre Accelerator card in a Power 11 server answer many requests at once instead of a few.
IBM Spyre Enablement Stack for Power
The drivers, runtime and models IBM requires before a Spyre Accelerator card runs anything on a Power 11 server. It arrives from Red Hat inside Red Hat AI Inference Server, so the card sits idle until the Linux partition that drives it has the stack installed and entitled.
Red Hat AI Inference Server
Red Hat's model serving product, built on the open source vLLM engine. IBM lists it as one of two supported ways to drive a Spyre Accelerator on IBM Power.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.