Key-value cache
The key-value cache is the working memory an AI model keeps while it writes an answer, holding the attention keys and values for every token it has already read or produced. On an IBM Power 11 server that pile sits in the Spyre Accelerator card memory, and how much of it is free is what decides how many requests one card can answer at the same time.
What is a KV cache?
An AI model writes an answer one token at a time, and each new token depends on every token before it. Instead of re-reading the whole conversation for each new token, the model saves two sets of numbers for every token it has already seen. They are called the key and the value. That saved pile is the key-value cache. The vLLM project describes it as the attention key and value tensors kept in memory to generate the next token. A token is a piece of a word, so a long question and a long answer both add to the pile.
Picture a long phone call where you write a note for every sentence the caller says. The notes make the rest of the call fast, because you never have to ask anyone to repeat. But the notepad keeps growing, and the pad has a fixed number of pages. The KV cache is that notepad, and the accelerator card is the pad.
Why does it decide how many requests one card can answer?
The cache is not small. The vLLM team measured a single sequence in the LLaMA-13B model taking up to 1.7 GB. Every active request holds its own pile at the same time. When the free cache runs out, no new request can start, so free cache is the real ceiling on how many people one card serves at once. Red Hat says it plainly in its vLLM tuning guidance: a larger KV cache lets the server support more concurrent tokens and requests, which raises throughput.
Context length in the two Spyre setups IBM documents
Values from IBM Redbook SG24-8592, Implementing AI on Power11, January 2026. IBM defines the context length setting as the largest sum of prompt length and output tokens, and every token inside it needs cache room.
Where does the cache sit on a Power 11 server?
It sits on the accelerator card, not in the server main memory. One IBM Spyre Accelerator for Power is a PCIe Gen5 x16 adapter, feature code ECSE, with 32 accelerator cores and 128 GB of LPDDR5 memory inside a 75 watt envelope. IBM states that up to eight of those cards cluster in a single expansion drawer for 256 cores, 1 TB of memory and 1.6 TB per second of combined bandwidth, and that the software stack can shard a model across cards. So adding cards adds model room and cache room together.
One Spyre card against a full drawer of eight
| Measure | One card | Eight cards in one drawer |
|---|---|---|
| Accelerator cores | 32 | 256 |
| On-card memory | 128 GB LPDDR5 | 1 TB |
| Combined memory bandwidth | IBM does not publish a per-card figure | 1.6 TB per second |
| Power draw | 75 watts | 600 watts at 75 watts each |
IBM Redbook SG24-8592, January 2026, and the IBM Spyre Accelerator for Power documentation. IBM supports the adapter only in ENZ0 PCIe Gen4 expansion drawers fitted with ENZA fanout modules, and each ENZA module takes up to four cards.
What happens when the cache fills up?
- New requests wait. The serving engine admits a request only when there is cache room for it, and that wait counts against time to first token.
- vLLM can preempt a request that is already running to free cache space for another one. The preempted request is recomputed once space comes back.
- In vLLM version 1 the default preemption mode is recompute rather than swap, so the work is redone on the card instead of moved out to host memory.
- On Power, the fix IBM documents for a container that runs out of memory is to reduce the batch size or the context length.
Preemption is normal housekeeping, not a failure. But a server that preempts often is a server whose cache is too small for the traffic it is being handed, and the repeated recompute work shows up as slower answers for everyone.
How do you give it more room?
Three levers IBM documents for Spyre on Power
- Shorten the contextThe max model length setting is the largest sum of prompt plus output tokens. IBM uses 3,072 tokens in its entity extraction example and 32,768 in its retrieval example. A shorter context means each request holds a smaller pile.
- Lower the batch sizeThe max batch size setting caps how many requests run together. IBM example commands use 16 for entity extraction and 32 for retrieval. Fewer seats at the table leaves more cache for each one.
- Add cardsThe world size setting tells the container how many accelerators to use, and the IBM retrieval example uses four. Up to eight cards fit in one drawer, and the stack can shard one model across them.
Settings and example values from IBM Redbook SG24-8592, Implementing AI on Power11, January 2026, which applies to Spyre GA1, Power11 servers, Red Hat Enterprise Linux 9.6 or later and Red Hat AI Inference Server 3.1.
Related Terms
PagedAttention
PagedAttention is the memory manager inside the vLLM serving engine. It hands out an AI model's working memory in small fixed blocks instead of one worst-case reservation, which is what lets one Spyre Accelerator card in a Power 11 server answer many requests at once instead of a few.
Prefix caching
Prefix caching lets a Spyre Accelerator skip re-reading the opening part of a prompt it has already read. For an IBM i shop whose requests all begin with the same long instruction block, it is the setting that stops the card from paying for that block on every single request.
Continuous batching
Continuous batching lets one Spyre Accelerator serve many requests at once by handing an open seat to a waiting request the moment an earlier one finishes, instead of holding a fixed group together until the slowest one ends. For an IBM i shop it is the setting that decides how many people a single card can answer before anyone waits in line.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
Time to first token
Time to first token, or TTFT, measures the wait between sending a question to an AI model and seeing the first word of the answer come back. For an IBM i shop it is the number that says whether local AI on a Power 11 server with a Spyre Accelerator feels instant or feels broken, because it counts queue time and the whole prompt read, not just the writing.