A modern AI accelerator can be built to perform trillions of calculations per second, but none of that performance is useful if the hardware cannot get rid of the heat it creates.
AMD’s Instinct MI300X, for example, has a maximum total board power of 750 watts while carrying up to 192 GB of HBM3 memory. Newer accelerator generations are already pushing higher power envelopes, with AMD’s current documentation listing up to 1,400 watts for the MI355X.
That changes the design problem. Engineers are no longer deciding only how many transistors, tensor cores or memory stacks they can fit into a package. They also have to decide whether the resulting heat can be moved out quickly enough, without creating hotspots that throttle performance, damage components or shorten the useful life of the system.
Thermal engineering has therefore moved from the cooling department into the architecture itself.
AI hardware is becoming a heat-density problem
It is tempting to think about power in simple totals: this chip uses 1,000 watts, so the cooling system needs to remove 1,000 watts. The real problem is more local.
Two processors can consume the same amount of electricity while presenting very different thermal challenges. One might spread its heat relatively evenly. Another might concentrate a large share of its power in small regions, creating intense hotspots that are much harder to cool.
That distinction is becoming especially important as AI packages combine multiple compute dies, high-bandwidth memory and dense interconnects in very small spaces. A recent IEEE thermal test vehicle was designed to reproduce those conditions, combining six emulated logic dies and 12 emulated HBM stacks. The test package could dissipate more than 2.6 kW, with localized hotspots reaching 314 W/cm².
HBM makes the package harder to cool
High-bandwidth memory is one of the clearest examples of performance and thermal engineering colliding.
HBM puts memory physically close to the accelerator, allowing enormous data bandwidth without forcing the processor to constantly reach across a conventional memory hierarchy. AMD’s MI300X, for instance, pairs its 750-watt accelerator with 192 GB of HBM3 and more than 5 TB/s of theoretical memory bandwidth.
The downside is physical density. HBM uses vertically stacked memory dies connected with through-silicon vias, and researchers have found that predicting both junction temperature and hotspot position becomes increasingly important as those stacks and their surrounding thermal conditions change.
That creates a delicate design balance. Put compute and memory closer together and communication gets faster. Pack more memory into the package and capacity and bandwidth increase. But the same package now contains more heat sources in a confined area.
The problem can also travel in both directions: a hot compute die can raise the temperature of nearby memory, while thermal constraints inside the memory stack can limit how aggressively the overall system can run.
The package is now part of the cooling system
Older server designs made it relatively easy to draw a mental line between the processor and the cooling system: silicon, package, heatsink, fan.
Advanced AI hardware is different. Multi-die packages introduce interposers, microbumps, substrates, thermal interface materials and multiple HBM stacks. Each layer adds another path through which heat must travel.
Researchers are even investigating new materials for that path. A 2025 study of a 3D GPU-memory architecture found that a thermally conductive boron-nitride interposer reduced simulated hotspot temperatures by about 20°C compared with a silicon interposer in the modeled design.
The implication is easy to miss: the material beneath a chip can influence how much useful performance engineers can extract from it.
Why air cooling is becoming harder to justify
Air cooling is simple, familiar and cheap enough for ordinary servers. But there is a point at which moving enough air through a rack becomes an inefficient way to deal with the heat.
ASHRAE now describes AI data centers as having power densities that can routinely exceed 50–120 kW per rack, with higher densities expected. Its guidance recommends integrated liquid-cooling infrastructure for environments where conventional air cooling is no longer sufficient.
This isn’t just about adding bigger fans. Fans consume power themselves, require physical space and still depend on moving heat from the chip into air before the air can carry it away.
Liquid cooling attacks the problem closer to the source.
Liquid cooling changes what AI servers can be
Direct-to-chip liquid cooling places a cold plate directly over the high-power components. Coolant passes through channels in the plate, absorbs heat and carries it away through a closed loop.
The performance advantage can be substantial. In one recent study comparing air and liquid cooling for an H100 GPU, researchers reported GPU temperatures of roughly 41–50°C with liquid cooling versus 54–72°C with air cooling under the tested workloads, alongside a reported performance improvement of about 17% in the liquid-cooled configuration. These results come from that particular experimental setup rather than being a universal benchmark for every server design, but they illustrate the connection between temperature and sustained performance.
At the extreme end, engineers are also testing two-phase cooling, where a refrigerant changes phase as it absorbs heat. In the IEEE test vehicle, this approach handled heat fluxes of up to 314 W/cm² at localized hotspots and reduced peak chip temperatures by as much as 25°C compared with the tested baseline conditions.
Rack-scale systems show where this is going
Once accelerator power gets high enough, the cooling problem stops being a server problem.
NVIDIA’s GB200 NVL72 is a useful example. The rack-scale system connects 72 Blackwell GPUs and 36 Grace CPUs in a liquid-cooled architecture. NVIDIA describes the rack as a single 72-GPU NVLink domain rather than a collection of unrelated servers.
The newer GB300 NVL72 takes that approach further with 72 Blackwell Ultra GPUs and 36 Grace CPUs in a fully liquid-cooled rack-scale design.
That is an important architectural clue. The cooling system has been designed alongside the compute system because the physical and thermal constraints are now tightly linked.
| Design layer | Thermal question | Typical engineering response |
|---|---|---|
| Compute die | Where are the hotspots? | Power mapping, frequency control, better heat spreading |
| HBM/package | Can memory and compute remain within safe temperatures? | Package layout, thermal materials, optimized interfaces |
| Server | Can the heat be removed without throttling? | Cold plates, liquid loops, intelligent monitoring |
| Rack | Can hundreds of components share one thermal envelope? | Manifolds, coolant distribution and rack-level design |
| Data center | Can the facility reject all that heat efficiently? | Warm-water loops, dry coolers, heat reuse and facility controls |
The table is useful because it shows how the thermal constraint propagates outward. A decision that starts inside a package can eventually influence the design of an entire building.
Warmer coolant can actually improve efficiency
One of the more counterintuitive developments is that the best liquid-cooling system does not necessarily use extremely cold water.
NVIDIA says its Rubin-generation AI infrastructure can operate with coolant temperatures as high as 45°C. The idea is simple: when the coolant is warm enough, facilities in suitable climates can reject heat through dry coolers without relying as heavily on mechanical chillers.
This turns a thermal limitation into an infrastructure design opportunity.
The goal is no longer simply to make the chip as cold as possible. The goal is to keep the silicon and other components inside their allowable operating ranges while moving heat through the facility with as little additional energy as possible.
Thermal limits are starting to influence architecture
This is where the subject gets more interesting than a discussion about server fans.
If adding more compute creates disproportionately difficult thermal problems, engineers have several ways to respond. They can improve performance per watt, change the memory hierarchy, alter chiplet layouts, reduce precision, spread workloads across more dies or redesign the cooling path itself.
In other words, thermal constraints can influence the architecture before the first chip is manufactured.
That idea is already showing up in research. A 2026 study used a physics-based model and generative design to produce alternative direct-to-chip cooling-channel geometries for the NVIDIA GB200. The resulting design reduced average temperature by more than 5°C and maximum temperature by more than 35°C relative to the study’s baseline parallel-channel design.
Temperature also affects reliability
Heat creates another problem that is easy to overlook: mechanical stress.
Packages contain materials that expand and contract at different rates. Repeated temperature changes can contribute to warpage, interconnect fatigue, delamination and other reliability problems. In a high-density AI package, thermal cycling is not an abstract materials-science issue; it can become a manufacturing and lifetime constraint.
That is one reason thermal design has to happen early. Waiting until the end of a chip program to ask how the package will be cooled is increasingly impractical when the package itself determines so much of the thermal behavior.
The next performance metric may look different
There is a subtle change happening in how AI hardware should be evaluated.
Raw FLOPS are still useful. So are memory bandwidth and model throughput. But as power density climbs, those numbers need context.
A faster accelerator that constantly runs into thermal limits can lose to a slightly slower design that sustains its workload efficiently. The useful question becomes something closer to: how much AI work can this system perform per unit of energy while staying inside its thermal envelope?
ASHRAE’s AI data-center framework reflects this broader view. Its recommendations connect compute density, cooling architecture, monitoring, workload management and facility efficiency rather than treating them as separate engineering domains.
What comes next
The thermal problem will not disappear simply because processors become more efficient.
Efficiency gains can be consumed by higher model sizes, more memory, larger clusters and more inference demand. That means thermal design will remain tied to how quickly the industry can scale AI infrastructure.
The interesting part is where the engineering response is heading. We are seeing tighter integration between compute dies and memory, increasingly sophisticated packaging, direct-to-chip liquid cooling, warmer facility loops, rack-scale architectures and software that can respond to changing thermal conditions.
There is a useful lesson here for anyone following the AI hardware race: the next big accelerator is not designed in isolation.
Its package, memory, power delivery, coolant path, rack and data-center environment increasingly form one system.
That is why thermal limits are no longer just a cooling problem. They are becoming one of the boundaries around what AI hardware can physically be.
References for further reading
- AMD Accelerator Specifications — current power and memory specifications for Instinct accelerators.
- NVIDIA GB200 NVL72 — rack-scale liquid-cooled architecture and system specifications.
- ASHRAE AI Data Center Energy Performance Framework — thermal and power-density guidance for AI facilities.
- IEEE: Experimental Demonstration of High-Power Thermal Test Vehicle Using Two-Phase Cooling — extreme heat-flux testing for AI-style packages.
- Generative Design for Direct-to-Chip Liquid Cooling for Data Centers — research on optimizing cooling channels for heterogeneous AI packages.
- Neural Network Surrogate Model for Junction Temperature and Hotspot Position in 3D HBM Chiplets — research on thermal behavior in stacked HBM.
Discover more from Aree Blog
Subscribe now to keep reading and get access to the full archive.
