bfloat16
bfloat16 is the 16-bit number format IBM Power10 and Power11 cores use to run an AI model in half the memory of full precision, without giving up the range of values a model needs. On a Power server running Linux, vLLM now reaches for bfloat16 first when the precision setting is left on auto, so the on-chip Matrix-Multiply Assist hardware does the math.
What does bfloat16 do for an AI workload on Power?
A model is a huge pile of numbers. Every number takes room in memory, and memory is usually what runs out first. bfloat16 writes each number in 16 bits instead of 32, so the same model takes half the space.
Think of a number as having two jobs: saying how big it is, and saying how exact it is. bfloat16 keeps the full budget for how big, and spends less on how exact. That trade suits AI models, because they care more about reaching very large and very small values than about the last few decimal places.
Bits each format spends on the exponent, the part that sets how big and how small a number can get
Eight exponent bits is the whole point of bfloat16. It reaches the same size of number as 32-bit FP32, stopping near 3.4e38, which is a number with 39 digits. FP16 keeps only five exponent bits, so it stops at 65,504. Figures from Table 1 of the 2019 paper A Study of BFLOAT16 for Deep Learning Training.
How is it different from FP16?
Both are 16 bits wide. They split those bits differently, and on IBM Power they are separate instructions, not a switch on the same instruction.
FP16 spends more bits on being exact and fewer on being big. That makes it finer inside a narrow band and more likely to run out of room at the top and the bottom. The 2019 bfloat16 study found that training in bfloat16 matched full FP32 results in the same number of passes with no change to the model settings, while FP16 needed tuning to get there.
The three floating point formats IBM Power Matrix-Multiply Assist takes as input
| Format | Total bits | Sign, exponent, mantissa | Largest normal number | Power instruction family |
|---|---|---|---|---|
| FP32, single precision | 32 | 1, 8, 23 | 3.40e38 | xvf32ger |
| bfloat16 | 16 | 1, 8, 7 | 3.38e38 | xvbf16ger2 |
| FP16, half precision | 16 | 1, 5, 10 | 65,504 | xvf16ger2 |
Matrix-Multiply Assist also takes FP64 double precision and the integer formats INT16, INT8 and INT4. Both 16-bit float families read two 4 by 2 input matrices and add the product into a 4 by 4 matrix of FP32 numbers, so the running total is kept at full precision. Sources: the IBM Redpaper Matrix-Multiply Assist Best Practices Guide and the IBM paper A matrix math facility for Power ISA processors.
Which IBM AI silicon runs bfloat16?
The processor cores do. The add-in card does not, going by what IBM has published.
Matrix-Multiply Assist arrived with Power ISA v3.1 and Power10, and bfloat16 is one of its native input formats. The IBM Power11 E1150 Redbook says those accelerators were refined again for Power11 and that the chip supports many AI data types.
- Power10 and Power11 cores: bfloat16 is a native Matrix-Multiply Assist input format, with every result added into FP32. IBM counts four of these engines per Power10 core.
- IBM Spyre Accelerator: IBM Research lists low precision FP16, FP8, INT8 and INT4 arithmetic on the card, plus FP32 for work such as activations. bfloat16 is not in that list.
- So a model served in bfloat16 is a job for the cores. If you plan to move that model onto a Spyre card later, plan on converting it to one of the formats Spyre does list.
Missing from a published list is not the same as impossible in hardware. IBM has not published a full format reference for the Spyre Accelerator on Power, so confirm the formats your model needs with IBM or your reseller before you order a card.
What do I set in the runtime?
Three checks before you serve a model in bfloat16 on Power
- 1. Leave the precision setting on autoOn PowerPC, the automatic choice in vLLM now prefers bfloat16 over FP16 and FP32. The change was merged into the vLLM project on 30 September 2026 so that models use the Power Matrix-Multiply Assist hardware. You can still ask for FP16 by name if a model needs it.
- 2. Check which format the model was published inMany open models ship with bfloat16 weights already. If a model ships in FP16 or in a quantized integer format, the runtime has to convert it, and conversion costs time and memory while the model loads.
- 3. Size memory at 2 bytes per numberbfloat16 is 2 bytes per number. A model with 7 billion parameters, meaning 7 billion stored numbers, needs roughly 14 GB just for its weights, before the working space each request needs. Memory, not the accelerator, usually decides whether the model fits.
The vLLM change applies to the automatic setting only. Naming a precision yourself still overrides it.
The IBM Power11 E1150 Redbook claims a 10 to 20 times advantage over Power9 for AI inferencing workloads, crediting more memory bandwidth and new instructions. That is a vendor figure for a workload mix IBM chose, not a bfloat16 benchmark. The IBM Power10 tutorial demonstrated 5 times the throughput of Power9 on a full precision FP32 model and said only that lower precision types such as bfloat16 and INT8 were expected to do better.
Related Terms
Matrix Math Acceleration (MMA)
An on-chip acceleration feature in IBM Power10 and Power11 processors that speeds up the matrix math operations used in AI inferencing, without requiring a separate GPU.
Quantization
Storing a model's numbers with less precision so it fits in less memory and runs faster, at some cost to accuracy.
TOPS
Trillions of operations per second. A rough headline rating for AI hardware, useful for comparison and misleading on its own.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
vLLM
The open source model serving engine inside Red Hat AI Inference Server, known for PagedAttention and continuous batching.