NVMe U.2
NVMe U.2 is the 2.5 inch flash drive bay that holds every bit of internal storage on a Power 11 server, wired as native PCIe straight to the processor. For an IBM i shop staging data for AI inference it settles two things: how much flash fits inside the box, and whether your one expansion drawer goes to drive bays or to Spyre Accelerator cards.
What is an NVMe U.2 drive?
It is a flash drive about the size of a deck of cards that slides into the front of the server. NVMe stands for Non-Volatile Memory Express. That is the language the drive and the processor speak to each other. U.2 is the shape of the plug. IBM carries each drive in a 15 mm sled, which is just a tray that clicks into the bay.
The old way put a traffic cop between the drive and the processor. A disk controller took every request, lined it up, and passed it along. NVMe takes the cop out of the road. The drive plugs into the same PCIe wiring the processor already uses for adapter cards, so a request goes straight there and back.
On Power 11 this is not one storage choice among several. IBM's Redbook for the scale-out servers says NVMe drives are the exclusive internal storage option on the Power S1122 and the Power S1124. The same book names the reason it cares about the direct wiring: databases, analytics, and AI inferencing. The older SAS drive enclosure, the EXP24SX, is not supported on Power 11 at all. IBM points you at the NED24 NVMe drawer instead.
How many drives fit in each Power 11 server?
NVMe U.2 drive bays IBM documents, by machine
Counts are internal bays, except the NED24, which is a separate drawer you cable to the server. The Power E1150 has ten physical bays, but IBM supports only six of them when two or three sockets are populated. All ten work with four sockets.
How the internal bays are wired on the Power 11 scale-out servers
| Machine | Base backplane | Bays per backplane | Backplanes allowed | Drives maximum |
|---|---|---|---|---|
| Power S1122, a 2U box | Feature Code EJ1X | 4 | One or two | 8 |
| Power S1124, a 4U box | Feature Code EJ1Y | 8 | One or two | 16 |
Each backplane cables to its own NVMe JBOF card, which is a plug-in adapter holding a 52 lane PCIe Gen4 switch. JBOF stands for Just a Bunch of Flash. One card per backplane, two cards maximum, and each card takes up a PCIe slot in the server. IBM allows the JBOF card in slots C8, C10 and C11, but only two of those three at once.
Which drive do you order for IBM i?
The one with the IBM i feature code. This is the part that bites people. The same physical capacity ships under two different numbers, one for IBM i and one for AIX and Linux, because IBM formats the drive differently for each. Ordering the wrong number does not get you a drive you can slow down and live with. It gets you a drive your operating system will not use.
Enterprise NVMe U.2 feature codes IBM lists for the NED24 drawer
| Capacity | Feature code for IBM i | Feature code for AIX and Linux |
|---|---|---|
| 800 GB | ES5B | ES5A |
| 1.6 TB | ES5D | ES5C |
| 3.2 TB | ES5F | ES5E |
| 6.4 TB | ES5H | ES5G |
| 15.3 TB, mainstream grade | ECTB | ECT9 |
From IBM Redbook SG24-8590. IBM also lists an older ES4 series at 1.6, 3.2 and 6.4 TB with the same IBM i and AIX split. Enterprise drives are built for heavy writing. Mainstream drives are cheaper and built for reading, so IBM aims them at boot drives and general storage.
Running the drive through the Virtual I/O Server changes the order again. IBM's IBM i input and output support page lists a third feature code for the VIOS case and states that support is for the VIOS VSCSI LVM client only, so block size is 4096. Confirm the feature code against your own machine type and operating system release before you order.
Does a drawer of drives cost you your Spyre cards?
On a 2U Power S1122, yes. This is the sharpest planning decision on the whole page, and it is easy to miss. IBM supports the Spyre Accelerator only inside an accelerator expansion drawer, never in the server's own slots. Each of those drawers holds up to eight Spyre cards. A NED24 full of drive bays is also a drawer. The S1122 allows exactly one drawer of any kind, so the drive bays and the accelerator cards are competing for the same spot.
Expansion drawers each Power 11 scale-out server allows
| Machine | Drawers in total | NED24 drive drawers | ENZ0 PCIe drawers |
|---|---|---|---|
| Power S1122, a 2U box | 1 | 1 | 1 |
| Power S1124, a 4U box | 2 | 1 | 2 |
Read the rows together. The S1124 allows two drawers in total, so one NED24 and one ENZ0 is a real configuration. The S1122 allows one drawer in total, so you pick. Attaching any drawer also costs two PCIe slots in the server, because IBM requires two PCIe x16 to CXP converter cards per drawer.
What should you check before you buy?
Five checks on an NVMe order for AI work
- Count the slots, not just the baysInternal NVMe eats PCIe slots. Each of the two JBOF cards takes one, and each expansion drawer takes two more for its converter cards. Add it up before you assume a Spyre card has somewhere to go.
- Match the feature code to the operating systemIBM i and AIX or Linux take different numbers for the same capacity, and a VIOS setup takes a third. Check your release, not a reseller listing.
- Know what is hot-pluggable and what is notIBM supports concurrent maintenance on the NVMe drives, so a drive can be replaced without taking the machine down. The JBOF card is not concurrently maintainable, because of the cabling to the backplane.
- Decide how the drives get dividedEach NVMe device is individually addressable and can be handed to one logical partition on its own. IBM also documents splitting drives across two redundant Virtual I/O Server instances so no single VIOS failure takes the storage with it.
- Check the socket count on an E1150Ten bays are in the machine, but only six are supported until the fourth socket is populated. A two socket E1150 is not a ten drive machine.
All five come from IBM Redbooks SG24-8590 and SG24-8589 and from IBM's IBM i input and output support page.
Two smaller limits worth knowing. The Power E1150 runs single-port NVMe only, and a drive that supports both U.3 and U.2 runs in U.2 mode on it, because the E1150 carries no SAS protocol at all. And the 15 mm carrier accepts a 7 mm drive, with IBM's conversion kit EC7X on the E1150, so a thinner drive is not automatically ruled out.
Related Terms
PCIe Gen5
PCIe Gen5 is the slot generation that sets how fast an adapter card can talk to an IBM Power processor, at 32 GT/s per lane. For an IBM i shop sizing AI work it matters for one narrow reason: every Gen5 slot IBM documents on Power 11 is eight lanes wide, so it carries the same 64 GB/s as a Gen4 sixteen lane slot.
ENZ0 Expansion Drawer
The PCIe4 expansion drawer IBM documents for supported Spyre Accelerator for Power configurations on Power 11 systems.
Spyre Accelerator
IBM's PCIe AI inference card. On IBM Power 11 it carries 32 cores, 128 GB of memory, and more than 300 TOPS inside a 75 watt envelope.
Virtual I/O Server
VIOS is a dedicated IBM Power partition that shares one set of physical disk, Fibre Channel and Ethernet adapters out to every other partition on the server. For an IBM i shop that means a Linux partition running an AI model can get its storage and its network connection from VIOS instead of buying its own physical cards.
IBM Power S1124
The Power S1124 is the largest scale-out IBM Power 11 server, machine type 9824-42A, and IBM lists it for 8, 12, 16 or 24 Spyre AI accelerator cards ... but only when both processor modules are installed. An IBM i shop can keep production on IBM i 7.4 TR12 or later and run model inference in a Red Hat Enterprise Linux partition on the same 4U machine.