Quantization
Storing a model's numbers with less precision so it fits in less memory and runs faster, at some cost to accuracy.
A model is a very large collection of numbers. Quantization stores those numbers with fewer digits, which shrinks the model and speeds it up. The Spyre Accelerator supports INT4, INT8, FP8, and FP16, and IBM's NorthPole research chip natively supports 2-bit, 4-bit, and 8-bit precision with quantization-aware training to protect accuracy at the lowest depths. This matters to an infrastructure buyer because a Spyre card has a fixed 128 GB of memory: quantization is one of the levers that decides whether a given model fits on one card or needs several.
Related Terms
TOPS
Trillions of operations per second. A rough headline rating for AI hardware, useful for comparison and misleading on its own.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
IBM Granite
IBM's own family of open language models, published under the Apache 2.0 licence so they can be downloaded, tuned, and run on hardware you own.