Nvidia offers 120kW liquid cooled Blackwell rack as industry standard

Nvidia is donating its 120kW, 1400A liquid cooled rack design to the Open Computer Project (OCP) for AI running on Blackwell GPUs.

Blackwell is now in mass production and is shipping to board partners says Shar Marasimhan, director of product marketing for data centre GPU and AI training at Nvidia.

“We are submitting the GB200 NVL72 rack design as an official contribution to OCP with the reinforcement, the NVLink, spine, plumbing and cooling quick release as well as the direct liquid cooling manifolds to the trays which we will make available to the entire community,” he said.

“Blackwell is in full production with deliveries a week ago.”

• Rubin successor to Blackwell GPU
• Rubin to use HBM4 memory

The rack combines 36 Grace CPUs with 72 Blackwell GPUs and 18 NVLink switches with a combined bandwidth of 1.8Tbyte/s. The reinforced steel rack can handle AI models with 27tn parameters and supports 1.4 exaflops of performance with 5000 copper cables, 120kW of power at 1400A, double today’s rack designs

“This has to be in a single rack with copper cabling for lower cost and far less power than fibre optics,” said Marasimhan. “We did a lot of simulation and modelling on liquid cooling and we have our own manifold designs with direct to chip cooling.”

Meta took the GB200 NVL72 rack modified for their specific data centre needs and released as the Catalina reference design back to the open source community. “This is what we love to see,” he said.

“Opening up the reference designs should help increase the adoption. We are making it possible to use fewer racks and as you require fewer servers we are looking at the power into the data centre so that’s a win for the customer. The most time consuming task is the high voltage lines to the data centres, so we make it easier for the adoption