Glossary

Continuous batching

CB

Continuous batching lets one Spyre Accelerator serve many requests at once by handing an open seat to a waiting request the moment an earlier one finishes, instead of holding a fixed group together until the slowest one ends. For an IBM i shop it is the setting that decides how many people a single card can answer before anyone waits in line.

What it decidesHow many requests share one card
What it replacedStatic batching, one fixed group
IBM batch size examples16 to 32 requests
Selectable sinceStack 1.0, December 2025

What does continuous batching do?

A card that answers AI questions can work on more than one question at a time. Continuous batching is the rule that decides who gets a seat. After every single step of writing, the engine looks again. If a request just finished, a waiting one takes its place right then.

Think of a shuttle bus. The old way filled every seat, drove the whole loop, and would not pick anyone up until the last rider got off. Continuous batching is a bus that stops at every corner. An empty seat gets filled on the spot, so nobody stands at the curb watching that seat drive past.

This matters on IBM i because real jobs are uneven. A short lookup from a green screen, a longer customer question from a web form, and a big nightly extract against Db2 for i all reach the same card with very different lengths. Under static batching the short ones wait for the long one. Under continuous batching they do not.

How does the card decide what to run next?

Four things the scheduler does at every step

  1. Two lines formThe engine keeps a waiting line for prompts it has not started and a running line for requests already writing text.
  2. Requests already writing go firstThe scheduler serves the running line before anything else and works out how many tokens each one needs next. A token is a word or a piece of a word.
  3. New prompts fill the room that is leftEach step has a token budget. Whatever is left after the running line goes to prompts from the waiting line, so reading a new question and writing an older answer can happen in the same step.
  4. If memory runs short, someone loses a seatWhen there are not enough working memory blocks free, the engine can push a lower priority request back out and compute it again later.

Source: the vLLM project description of its scheduler, which keeps a waiting queue and a running queue, serves requests already decoding first, then admits new prompts from the waiting queue inside a token budget, and can preempt a low priority request by freeing its key-value blocks.

What was the old way?

Static batch mode and continuous batching mode on a Spyre card

BehaviorStatic batch modeContinuous batching mode
When a request can joinOnly when a new batch startsAfter any generation step
What the card does while waitingHolds the group until the slowest request endsGives the open seat away at once
How the size is setWarmup batch size, default 1 requestThe max-num-seqs flag, 16 or 32 in IBM examples
How IBM picks itThe default, when the continuous batching variable stays 0Set that variable to 1

Source: the container environment variable table in the IBM Spyre Accelerator for Power documentation. The continuous batching variable accepts 0 or 1 and defaults to 0. IBM describes 1 as continuous batching mode and anything else as static batch mode. The static mode warmup batch size defaults to 1 request.

Upstream, the choice is going away. The Spyre plugin documentation now states that support for static batching with text writing models has been removed, and that those models run in chunked prefill mode instead. IBM example commands show the same drift. The two oldest Red Hat AI Inference Server example sets switch continuous batching on by hand, and the two newest do not set the variable at all.

How many requests should share one card?

Requests allowed in one batch, from IBM configuration values

Static mode default1 request
IBM example, vision and retrieval16 requests
IBM example, entity extraction32 requests

All three numbers come from the IBM Spyre Accelerator for Power documentation. The 1 is the default warmup batch size for static batch mode. The 16 and 32 are the batch size values IBM exports into the max-num-seqs flag in its own example Podman commands for Red Hat AI Inference Server 3.6.

  • Bigger is not automatically better. One card holds a fixed amount of working memory, so every request you add leaves less room for each of the others.
  • IBM recorded a real failure at the top end. Running three four-card retrieval augmented generation containers in one twelve-card drawer, with large input context and a large batch size, can leave Spyre cards in a failed state, and the logical partition needs a reboot to recover.
  • IBM guidance for that case is to run no more than two of those four-card containers in a single drawer, and to give the cards left over to other work such as reranker, embedding or entity extraction.
  • Match the number to the job. IBM uses 32 requests for entity extraction and 16 for its vision and retrieval examples, so model size and prompt length decide the right setting rather than one best value.

When did it arrive on Power?

IBM Spyre Enablement Stack for Power releases

  1. December 2025Stack 1.0 ships alongside Red Hat AI Inference Server 3.3. Its example commands already switch continuous batching on.
  2. March 2026Stack 1.1 adds chunked prefill, prefix caching, dynamic logical partitioning of Spyre, and Red Hat OpenShift AI support.
  3. June 2026Stack 1.2 adds twelve card configurations, custom fine tuned model deployment, and enhanced error handling recovery.
  4. September 2026Stack 1.3 adds more models, a new metric, and updated containers. Its example commands no longer set the continuous batching variable.

Dates and contents come from the What is new section of the IBM Spyre Accelerator for Power documentation. Each stack release pairs with an early access release of Red Hat AI Inference Server, from 3.3 through 3.6.

Continuous batching changes how many people a card can serve at once. It does not change the answer any one of them gets. Measure it with throughput and with the wait before the first word appears, on your own prompts, rather than from a batch size number alone.

Related Terms

vLLM

The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.

PagedAttention

PagedAttention is the memory manager inside the vLLM serving engine. It hands out an AI model's working memory in small fixed blocks instead of one worst-case reservation, which is what lets one Spyre Accelerator card in a Power 11 server answer many requests at once instead of a few.

Prefix caching

Prefix caching lets a Spyre Accelerator skip re-reading the opening part of a prompt it has already read. For an IBM i shop whose requests all begin with the same long instruction block, it is the setting that stops the card from paying for that block on every single request.

Inference serving

The software layer that takes requests for a model, packs them onto the hardware efficiently, and returns answers. It decides how much work one accelerator card can actually do.

Spyre Accelerator

IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
Back to Glossary