Prefix caching
Prefix caching lets a Spyre Accelerator skip re-reading the opening part of a prompt it has already read. For an IBM i shop whose requests all begin with the same long instruction block, it is the setting that stops the card from paying for that block on every single request.
What does prefix caching do?
Before an AI model writes a single word, it has to read the whole question. That reading step is called prefill, and it is the expensive part. Prefix caching saves the result of that reading so the card never has to read the same opening text twice.
Think of a cashier who has to re-read a long coupon page out loud for every customer in line. Prefix caching is the cashier writing the coupon total on a sticky note once and reading the note after that. Everyone behind the first customer gets served faster, and nothing about the answer changes.
This matters on IBM i because most real jobs repeat themselves. A retrieval job against Db2 for i sends the same instruction block, the same rules and often the same reference text on every call. Only the last line is new. Without prefix caching the Spyre Accelerator pays full price for the repeated part every time.
How does a cache hit happen?
Four things happen on the way to a reused block
- The prompt is cut into blocksA block is a fixed number of tokens of working memory. A token is a word or a piece of a word.
- Each block gets a fingerprintvLLM hashes the block using the tokens inside it, the fingerprint of the block in front of it, and a few extra values. Two prompts that start with the same words therefore produce the same fingerprints.
- Only full blocks are storedA half-filled block is never cached. That is why the saving comes from the opening of a prompt and not from a few matching words in the middle.
- A match skips the workWhen a new request carries a fingerprint already in the cache, the serving engine reuses the stored block and skips computing that part of the prefill.
Source: the vLLM project design note on prefix caching, which states that the block hash combines the parent hash value, the block tokens and extra hashes, and that only full blocks are cached. Since vLLM v0.11 the default hashing algorithm is sha256.
Why are hit rates lower on a Spyre card?
Spyre reads a prompt in fixed-size chunks, and a chunk is much larger than a block. IBM is direct about the consequence: the cache can skip a chunk only when that entire chunk is already stored. A near miss saves nothing.
Two token sizes that decide how much work gets skipped
The 1024 token prefill chunk size is the value IBM lists as supported for the Spyre Accelerator for Power. The 64 token block size is the fixed value in the Spyre vLLM plugin scheduler design note. Blocks carry the fingerprints; chunks are the unit of prefill work that can be skipped.
The plugin maintainers say the same thing from the software side: because prefill chunks are a fixed size, a prefill gets shorter only when a whole chunk is available in cache, so Spyre workloads may show lower hit rates than the same job on another accelerator. Expect a real benefit on repeated long openings and very little on short, varied prompts.
Which workloads use it?
IBM splits this by model type. Decoder models, the ones that write text, use prefix caching. Encoder-only models, the ones that turn text into numbers, do not.
IBM example containers for Red Hat AI Inference Server 3.4 on Spyre
| Workload | Model in the example | Cards | Prefix caching |
|---|---|---|---|
| Entity extraction | Granite-3.3-8b-Instruct | 1 | Enabled |
| Retrieval augmented generation | Granite-3.3-8b-Instruct | 4 | Enabled |
| Reranker | bge-reranker-v2-m3 | 1 | Not used |
| Embedding | Granite-Embedding-125m-English | 1 | Not used |
Source: the example Podman commands in the IBM Spyre Accelerator for Power documentation. The two text-writing examples add the prefix caching flag. The reranker and embedding examples switch chunked prefill off instead, and IBM says prefix caching cannot work without chunked prefill.
What should an IBM i shop watch for?
- Check how the switch behaves in your release. IBM documents prefix caching for the Power stack as something you add with the enable flag. The newer upstream plugin turns it on by default instead, with a separate flag to switch it off.
- Keep the repeated text at the very start of the prompt. The cache reads from the front, so a shared instruction block placed after the customer question saves nothing.
- Read the hit rate as a hint, not a receipt. IBM recorded a known issue in stack versions 1.0 through 1.2 where reported hit rates can come back higher than the real ones under heavy concurrency.
- Watch the two reported counters. Red Hat AI Inference Server reports prefix cache queries and prefix cache hits in tokens, so you can see how much of the reading was genuinely skipped.
IBM notes that the inflated hit rate issue affects metric reporting only. It does not change the answers the model gives or how well the cache actually works. Measure speed on your own prompts rather than trusting the percentage alone.
Related Terms
PagedAttention
PagedAttention is the memory manager inside the vLLM serving engine. It hands out an AI model's working memory in small fixed blocks instead of one worst-case reservation, which is what lets one Spyre Accelerator card in a Power 11 server answer many requests at once instead of a few.
Time to first token
Time to first token, or TTFT, measures the wait between sending a question to an AI model and seeing the first word of the answer come back. For an IBM i shop it is the number that says whether local AI on a Power 11 server with a Spyre Accelerator feels instant or feels broken, because it counts queue time and the whole prompt read, not just the writing.
IBM Spyre Enablement Stack for Power
The drivers, runtime and models IBM requires before a Spyre Accelerator card runs anything on a Power 11 server. It arrives from Red Hat inside Red Hat AI Inference Server, so the card sits idle until the Linux partition that drives it has the stack installed and entitled.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.