Red Hat AI Inference Server
Red Hat's model serving product, built on the open source vLLM engine. IBM lists it as one of two supported ways to drive a Spyre Accelerator on IBM Power.
Red Hat AI Inference Server accepts requests for a model, packs them onto the accelerator efficiently, manages the memory each conversation needs while it runs, and returns the answers. It is built on vLLM with optimisation work from Neural Magic, which Red Hat acquired. IBM's Spyre documentation names it, or Red Hat OpenShift AI, as a requirement for the accelerator on Power 11. How this layer is configured has a direct effect on how many concurrent users a single card supports, and therefore on how many cards a project needs.
Related Terms
vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
Red Hat OpenShift AI
Red Hat's larger AI platform, and the second option IBM accepts for driving a Spyre Accelerator on IBM Power.