Inference serving
The software layer that takes requests for a model, packs them onto the hardware efficiently, and returns answers. It decides how much work one accelerator card can actually do.
Inference is a model answering a question it was trained to answer. Inference serving is the layer that manages that at scale: accepting many requests at once, grouping them so the accelerator stays busy, managing the working memory each conversation needs, and handing back results. On IBM Power with a Spyre Accelerator, IBM names Red Hat AI Inference Server or Red Hat OpenShift AI as the supported serving layer. The configuration of this layer, rather than the card itself, usually determines how many concurrent users a single card supports.
Related Terms
vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.
Red Hat AI Inference Server
Red Hat's model serving product, built on the open source vLLM engine. IBM lists it as one of two supported ways to drive a Spyre Accelerator on IBM Power.
AI Inference Partition
A separate supported operating-system partition used to execute an AI model while an IBM i partition may remain the production application and data source.