PagedAttention
PagedAttention is the memory manager inside the vLLM serving engine. It hands out an AI model's working memory in small fixed blocks instead of one worst-case reservation, which is what lets one Spyre Accelerator card in a Power 11 server answer many requests at once instead of a few.
What does PagedAttention do?
While a model writes an answer it keeps notes on every token it has seen so far. Those notes are the key-value cache, or KV cache. The cache grows with each new token and it has to sit in the accelerator's own memory, next to the model. PagedAttention is the part of the vLLM serving engine that decides where each piece of that cache goes.
Picture a storage room. The old way gave every visitor a whole aisle, in case they turned up with a truckload. Most brought three boxes, so most of the room sat empty. PagedAttention hands out one shelf at a time and keeps a list of which shelves belong to whom. Anyone who has worked with paged memory on IBM i will recognize the idea, because that is where it came from.
Why does the key-value cache run out of room?
The cache is big, and nobody knows in advance how big it will get, because nobody knows how long the answer will be. The vLLM project reported that this cache can reach 1.7 GB for a single sequence on a 13 billion parameter Llama model. If the server reserves the longest possible answer for every request, it runs out of memory long before the card runs out of compute, and it has to turn requests away.
Share of key-value cache memory wasted, as measured by the vLLM team
Source: the vLLM project announcement post, 20 June 2023. The older figure is memory lost to fragmentation and over-reservation. With PagedAttention the only waste left is the unfilled tail of a request's last block.
How does it hand out memory?
How vLLM places the key-value cache, in four steps
- Cut the memory into blocksvLLM divides the free accelerator memory into fixed blocks. Each block holds the keys and values for a set number of tokens, 16 by default.
- Give each request a block tableEvery request gets a small table that maps its tokens, in order, to whichever blocks hold them. The blocks do not have to sit next to each other in memory.
- Add a block only when one fillsA new block is handed out when the current one is full. Nothing is held back for a length the request may never reach.
- Share the blocks that matchWhen two requests start with the same text, they can point at the same blocks instead of each storing a copy. vLLM calls this prefix caching and turns it on by default.
The 16 token default and the prefix caching default come from the vLLM cache configuration reference.
What does this mean on IBM Power 11?
IBM Spyre Accelerator for Power is a PCIe-attached inference card for Power 11 systems, carried in the ENZ0 PCIe4 expansion drawer. IBM describes it as built on 5 nm chip technology, running FP8 and FP16 work with continuous batching inside a 75 watt power envelope, and it requires either Red Hat AI Inference Server or Red Hat OpenShift AI. Both of those run the vLLM runtime, so PagedAttention is what manages the cache on the card.
The card's memory is the budget, and IBM publishes the arithmetic. A container starts with a ceiling of 32,768 tokens of context and 32 requests, and the vLLM engine shrinks the batch as the context grows so the two multiplied together never pass 131,072 tokens. That single number is the honest answer to how many people one card can serve at the same time.
How the Spyre container trades batch size against context size
| Requests in one batch | Context for each request | Total tokens in the batch |
|---|---|---|
| 32 requests | 4,096 tokens | 131,072 tokens |
| 16 requests | 8,192 tokens | 131,072 tokens |
| 4 requests | 32,768 tokens | 131,072 tokens |
Source: IBM Spyre Accelerator for Power product documentation. These three rows are IBM's own examples of the same token budget spent three ways.
Two Spyre details differ from stock vLLM. The block size in the Spyre plugin is currently fixed at 64 tokens rather than 16. Prefix caching has to be switched on, and it needs chunked prefill, which is on by default with a 1024 token chunk. Because those chunks are a fixed size, a chunk is reused only when the whole chunk is already cached, so IBM says the hit rate varies with the workload and with how much of a prompt repeats. Encoder-only models such as embeddings and rerankers do not use prefix caching at all.
When did this arrive?
From a research idea to a card in a Power 11 drawer
- 20 June 2023The vLLM project published PagedAttention with its serving engine, reporting cache waste under 4 percent and up to 24 times the throughput of Hugging Face Transformers.
- 12 September 2023Woosuk Kwon and eight co-authors posted the paper, Efficient Memory Management for Large Language Model Serving with PagedAttention, reporting 2 to 4 times the throughput of FasterTransformer and Orca at the same latency.
- December 2025IBM shipped Spyre Enablement Stack for Power 1.0 on Red Hat AI Inference Server 3.3 EA1, the first release that put the vLLM runtime and the Spyre card together on Power 11.
The release named in IBM's current documentation is Spyre Enablement Stack for Power 1.3 on Red Hat AI Inference Server 3.6 EA1.
Related Terms
vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
Red Hat AI Inference Server
Red Hat's model serving product, built on the open source vLLM engine. IBM lists it as one of two supported ways to drive a Spyre Accelerator on IBM Power.
Inference serving
The software layer that takes requests for a model, packs them onto the hardware efficiently, and returns answers. It decides how much work one accelerator card can actually do.