Tensor parallelism
Tensor parallelism splits one AI model across several Spyre Accelerator cards so the cards answer a single request together. On IBM Power 11 it is the setting that decides whether a model fits at all and how long a prompt it can take, and IBM publishes verified setups at only one, two or four cards per model.
What does tensor parallelism do?
A model is a very large stack of number tables. Tensor parallelism cuts those tables into slices and gives one slice to each accelerator card. The cards then work on the same request at the same time and pass partial results to each other. Nobody holds the whole model, so a model that is too big for one card can still run.
Think of four people reading one long contract. One person reading alone needs the whole thing on their desk. Four people can each take a quarter, read at once, then compare notes at the end. The reading finishes sooner and no single desk has to be big enough for the whole contract. The comparing notes part is real work, though, and it is why the cards need fast links between them.
In vLLM, the serving engine inside Red Hat AI Inference Server, the dial is --tensor-parallel-size. Its short form is -tp and its default value is 1, which means one card and no splitting.
How many Spyre cards does one model need?
That is not a free choice. The vLLM Spyre plugin publishes a list of verified deployment configurations, and each model on that list comes with the card counts IBM and Red Hat actually tested.
Verified Spyre card counts by model
| Model | Verified card counts | Longest verified context |
|---|---|---|
| Granite 3.3 8b Instruct | 1, 2 or 4 | 32,768 tokens on 4 cards |
| Granite 4.2 8b | 1, 2 or 4 | 32,768 tokens on 4 cards |
| Granite 4.1 30b | 4 only | 32,768 tokens |
| Granite Vision 3.3 2b | 1, 2 or 4 | 32,768 tokens on 4 cards |
| Ministral 3 14B Instruct | 1 or 4 | 32,768 tokens on 4 cards |
| Granite embedding and BGE reranker models | 1 only | No multi-card setup published |
Verified deployment configurations from the vLLM Spyre plugin documentation, read 8 November 2026. The two largest text models on that list, Granite 4.1 30b and Granite SWA 20b, are published at four cards and nothing smaller. A card count that is not on the list is simply not published as verified.
Does adding cards buy a longer prompt?
Often yes, because the prompt and the answer both have to sit in card memory while the model works, and more cards means more pooled memory. But the gain is published model by model, not as a rule.
Granite Vision 3.3 2b: longest verified context by Spyre card count
From the vLLM Spyre plugin verified configurations. For this model each doubling of cards doubles the published context, but that is not true everywhere: Granite 3.3 8b Instruct is verified at 8,192 tokens on one card and again at 8,192 on two, and only reaches 32,768 at four cards.
How do you set it?
Three settings in the container command IBM publishes
- List the cardsAIU_PCIE_IDS is a required container variable. IBM documents it as the space-delimited list of Spyre adapters the Red Hat AI Inference Server container uses. AIU, short for Artificial Intelligence Unit, is the IBM name for the accelerator unit on the card. Every IBM example lists one PCIe address for each card the model will use.
- Say how manyIBM examples set a plain shell variable called AIU_WORLD_SIZE to that same count. Every published example uses 1, 2 or 4, and the number of PCIe addresses in the step above always matches it.
- Pass it to vLLMThe container command ends with -tp followed by that number, alongside --max-model-len and --max-num-seqs. The IBM four-card retrieval example uses -tp 4 with a 32,768 token context and a batch size of 32. Its one-card vision example uses -tp 1 with a 3,072 token context and a batch size of 16.
Commands from the IBM Spyre Accelerator for Power documentation, covering IBM Spyre Enablement Stack for Power 1.3 with Red Hat AI Inference Server 3.6 EA1, September 2026.
What goes wrong if you get it wrong?
- More than 24 cards in one partition is not supported. IBM states the maximum is 24 cards per logical partition, and a system with more cards installed has to spread them across several partitions.
- In a 12-card drawer, 8 cards sit in x16 slots and 4 sit in x8 slots. IBM says to put a four-card retrieval container on the x16 slots, because that is where the cross-card traffic happens when a model is sharded across four cards.
- Two four-card retrieval containers per 12-card drawer is the documented ceiling. IBM reports that running three of them with a large context and a large batch can put Spyre cards into a failed state and force a partition reboot.
- A 12-card drawer runs its cards at a reduced power limit and needs Red Hat AI Inference Server 3.5 or later. An 8-card drawer can be switched to full power, and IBM says the lower limit costs retrieval and entity extraction work more than it costs embedding work.
Card count is not a free dial. Every card you hand to one model is a card no other job can use. The IBM advice for a 12-card drawer is to leave the spare cards for reranker, embedding and entity extraction work rather than giving them all to one model.
Related Terms
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.
Red Hat AI Inference Server
Red Hat's model serving product, built on the open source vLLM engine. IBM lists it as one of two supported ways to drive a Spyre Accelerator on IBM Power.
Key-value cache
The key-value cache is the working memory an AI model keeps while it writes an answer, holding the attention keys and values for every token it has already read or produced. On an IBM Power 11 server that pile sits in the Spyre Accelerator card memory, and how much of it is free is what decides how many requests one card can answer at the same time.
ENZ0 Expansion Drawer
The PCIe4 expansion drawer IBM documents for supported Spyre Accelerator for Power configurations on Power 11 systems.