vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.
vLLM serves large language models with high throughput and careful memory handling. PagedAttention hands out small blocks of memory as a conversation actually needs them, rather than reserving a worst case for every user, which is the same idea an operating system uses for paged memory. Continuous batching lets new requests join a batch already in progress so the accelerator does not idle waiting for a tidy queue. vLLM supports more than 500 model architectures and more than 200 accelerator types. You can run it yourself, but for a supported Spyre configuration on IBM Power, IBM names the Red Hat products rather than a self-assembled install.
Related Terms
Red Hat AI Inference Server
Red Hat's model serving product, built on the open source vLLM engine. IBM lists it as one of two supported ways to drive a Spyre Accelerator on IBM Power.
Inference serving
The software layer that takes requests for a model, packs them onto the hardware efficiently, and returns answers. It decides how much work one accelerator card can actually do.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.