IBM Granite: The Open Model That Comes With the Hardware

Buying an AI accelerator without picking a model is like buying a lorry and never deciding what to put in it. The card is the vehicle. The model is the cargo. IBM sells the vehicle and gives away one of the better cargoes for free, which is a strange enough arrangement that it is worth understanding properly.

Granite is IBM's own family of language models. It is what Red Hat bundles into its Linux AI image, it is what IBM Research ran on its experimental NorthPole chip, and it is the model most likely to end up on a Spyre card sitting in a Power 11 server. For an AS/400 shop, it is the one piece of this whole stack you can download and try tonight without asking anyone for a budget.

What the licence actually buys you

Granite ships under Apache 2.0. That is the plain, permissive open source licence, the same kind of arrangement that covers a great deal of the software already running in your machine room. In practice it means four things.

1. You can download it

   No sales conversation, no evaluation licence, no expiry. The model weights are published on Hugging Face, Ollama, GitHub, Docker Hub, and inside IBM watsonx.

2. You can change it

   Tuning the model on your own material is allowed and expected. That is what the InstructLab tooling exists to do.

3. You can run it in production

   There is no per-user fee, no per-token meter, and no renewal date that can be used as leverage later.

4. You can keep it on your own floor

   Nothing has to leave the building. For a shop whose data is the reason it still runs IBM i, this is usually the deciding argument.

IBM also had the Granite 4.0 models certified to ISO 42001 and cryptographically signed them, which is IBM answering the audit question before it gets asked. For a regulated shop that is not a marketing detail, it is the difference between a pilot that survives a compliance review and one that does not.

The memory trick that decides how many cards you buy

Here is the part that belongs in an infrastructure conversation rather than a software one. A Spyre Accelerator card has 128 GB of memory on it. That number does not move. Everything you want the card to do has to fit inside it.

A language model holds a running memory of the conversation it is having. The longer the conversation, and the more people talking to it at once, the more of the card gets eaten by that running memory rather than by the model itself. Granite 4.0, released in October 2025, changed the internal design to attack exactly this problem.

The technical reason is that Granite 4 mixes two different designs. Roughly nine parts in ten use a newer approach called Mamba-2, and the remaining part uses the conventional transformer attention everyone else uses. The conventional approach gets slower and hungrier as a conversation grows. The newer one does not, in the same way.

Translated into money: more than seventy percent less memory per request means several times more concurrent users on the same card. A project that would have needed four cards might need one. That is a hardware decision made by a software choice, and it is exactly the kind of thing that gets missed when the model is treated as somebody else's department.

Granite 4.0 and Granite 4.2, side by side

The Granite 4.2 addition worth understanding is the reasoning switch. Turned on, the model works through a problem in steps before answering, which costs more time and more memory but produces better results on anything with several moving parts. Turned off, it answers directly and cheaply. Having both in one model means one deployment can serve a cheap classification job and an expensive planning job without running two different models.

Three places Granite runs on IBM Power

1. On the processor you already own

   A smaller Granite model can run using the Matrix Math Acceleration built into every Power 10 and Power 11 core. No card, no expansion drawer, no extra power circuit. This is where almost every honest pilot should start.

2. On a Spyre Accelerator card

   A 75 watt card with 128 GB of memory and more than 300 TOPS, driven by Red Hat AI Inference Server or Red Hat OpenShift AI. This is where steady production load belongs.

3. As a service through IBM watsonx

   The same model family, consumed over the network with someone else running the hardware. The trade is convenience against your data leaving the building.

All three run the same open model. That is the quiet strength of the arrangement. A pilot tuned on the processor you already own does not have to be rebuilt when it moves onto a card, because the cargo never changed, only the vehicle.

What Granite is not

It is not an IBM i product. It has no awareness of your libraries, your journals, or your Db2 for i tables. Connecting a model to that data is a separate build, and it is genuinely the hard part of most projects. Our software sibling covers the tooling for that side of the work.

It is also not a finished application. A model answers questions put to it. Deciding which questions get asked, checking the answers, logging what happened, and deciding what a wrong answer costs are all still yours to design.