The Facility Is in Your Training Loop
0
0
Thermal, power and failure effects on large-scale training throughput, and why the building is a variable in your results
Most machine learning practitioners treat infrastructure as an abstraction that either works or does not. You request a node, the node appears, and what remains is a matter of batch sizes, parallelism strategy and learning rate schedules. The physical facility is somebody else’s problem, one level below the layer you are paid to think about.
That abstraction leaks. It leaks through your throughput numbers, your failure logs, your wall-clock time and, eventually, your budget. The building your GPUs sit in is a variable in your training run, and unlike most variables in your run, it is one you cannot tune after the fact. This piece sets out the mechanisms by which facility decisions reach into the training loop, what the measured effects actually are, and, importantly, where those effects are smaller than the marketing suggests.
The metric problem comes first
Any discussion of infrastructure effects on training performance has to start with measurement, because the metric most people watch is close to useless for this purpose.
GPU utilisation, as reported by nvidia-smi and most dashboards, tells you the fraction of time during which at least one kernel was resident on the device. It does not tell you whether the silicon was doing arithmetic. A GPU stalled on a memory fetch, waiting on a NCCL collective, or spinning on a straggler in a synchronous all-reduce reports high utilisation while doing nothing useful. Utilisation of 95% is compatible with a badly bottlenecked run.
Model FLOPs Utilisation is the metric that survives contact with reality. Introduced in the PaLM paper, MFU is the ratio of the useful floating-point operations your model actually executes per second to the theoretical peak of the hardware. It is a hardware-independent statement about how much of the silicon you paid for is doing work, and the honest headline is that the figure is usually about half. SemiAnalysis, benchmarking GPT-3 175B on H100s in BF16, recorded MFU rising from 34% to 54% over twelve months, a 57% improvement in training throughput from software changes across the CUDA stack alone, with no hardware change at all. That single data point makes the case for watching the metric: a fifth of your theoretical compute moved without anyone touching a GPU.
Two cautions before using it as evidence. First, the denominator matters enormously: an H100 SXM is rated at roughly 989 TFLOPS for BF16 tensor operations and around 1,979 TFLOPS for FP8. Quote MFU against the wrong precision, and you have moved the number by a factor of two without touching the hardware. Second, MFU conflates every source of inefficiency. A poor MFU may be your data loader, your parallelism layout, your kernel choice, or the facility. The number tells you that throughput is being lost, not where.
That said, MFU has one property that makes it the right instrument here: it is sensitive to infrastructure in a way that loss curves are not. A sudden drop of more than a few per cent from an established baseline is a system event. The usual suspects are a thermally throttling node, a degraded network link, or a straggler GPU dragging a collective. Which brings us to the first channel.
Channel one: thermal, and why the honest answer is smaller than the headline
The mechanism is dynamic voltage and frequency scaling. An H100 SXM carries a 700W thermal design power and adjusts its clock frequency automatically based on temperature and workload. This is worth stating precisely, because it is commonly described wrongly. DVFS is not a cliff that the GPU falls off at some published threshold. Hard limits do exist, and NVIDIA specifies distinct slowdown and shutdown temperatures that vary by product, but they sit well above normal operating range. The effect that matters in practice is continuous: cooler silicon sustains higher boost clocks, and warmer silicon sustains lower ones, across the entire operating band. A GPU never touching a throttle threshold can still be leaving performance on the table simply by running hot.
The interesting question is how much it matters, and here the available evidence is more nuanced than most vendor material admits. A 2025 benchmarking study by researchers from Kansas State, Florida Atlantic, Lawrence Berkeley National Laboratory, Brookhaven and Johnson Controls did the controlled experiment: two nodes, each with 8 H100 80GB GPUs, one direct-to-chip liquid-cooled and one air-cooled, running identical workloads. The full paper is on arXiv.
The thermal result is unambiguous. Under the GPU Burn stress test, the liquid-cooled node held its GPUs between 41 and 50°C while the air-cooled node ranged from 54 to 72°C, both hotter and considerably less stable. On throughput, the liquid-cooled node delivered an average of 54 TFLOPS per GPU against 46 for the air-cooled node. That is the 17% figure that has been quoted widely since.
Now the part that rarely gets quoted. When the same rigs ran actual training workloads- five LLMs from 7B to 32B parameters fine-tuned with QLoRA, the throughput advantage almost vanished. Across all eight workloads tested, the average improvement in achieved FLOPs was 0.3%, and the average reduction in training duration was 1.34%. Llama-3.1-8B, for instance, ran at 35.12 TFLOPS per GPU liquid-cooled against 34.76 air-cooled, a difference of about 1%.
So the 17% is real, and it is also a stress-test upper bound rather than a production expectation. The gap between the two tells you something worth understanding: GPU Burn saturates the thermal envelope in a way that a parameter-efficient fine-tuning workload does not. Whether you see the thermal benefit in throughput depends entirely on whether your workload is thermally bound. Many are not. Meta’s own account of the Llama 3 405B run makes the same point from the other direction: they observed DVFS being affected by temperature variation across the cluster and explicitly noted it was not a significant problem for them.
If you are being sold liquid cooling on a promise of double-digit training speedups, the evidence does not support it for typical workloads. What the evidence does support is a different and arguably better claim, which is the second channel.
Channel two: power, where the effect is large and reliable
In the same study, the production-workload difference that was large was not throughput. It was power. Averaged across the workloads, the liquid-cooled node drew about 5.2 kW against 6.2 kW for the air-cooled node. On the LLM fine-tuning runs, the gap was roughly 1.2 kW per node, and on the highest-utilisation workload it reached 1.5 kW. On those averages, that is about a sixth of node power removed for identical work, an effect that dwarfs the throughput difference and holds far more consistently across workloads.
The mechanism is mundane. An air-cooled node carries its own fans, and under heavy load those fans work hard. The test rigs make this concrete: the air-cooled node ran 8 fans, the liquid-cooled node 4. As computational intensity rises, fan power rises with it, and the fans are simultaneously less effective at removing heat than a cold plate. The efficiency gap compounds precisely when the machine is busiest.
There is a measurement artefact buried in this that deserves more attention than it gets, because it distorts a metric the entire industry quotes. Node fans are powered at the node. Facility metering therefore counts them as IT load, not as cooling overhead. A data hall full of air-cooled servers is thus reporting a slice of its cooling energy in the denominator of PUE rather than the numerator, which flatters the number. Move that thermal work to a central chilled water loop, and it reappears, correctly, as facility overhead. The paper’s authors make the point directly: liquid cooling both reduces total energy and moves cooling load out of the IT bucket where it was hiding, producing a more honest PUE and a less flattering one at the same time. Any comparison of PUE figures across cooling architectures that does not account for this is comparing different things.
Channel three: volatility, the load your facility has to survive
The power your cluster draws is not just larger than a conventional workload’s. It moves differently, and this is a facility problem that no amount of code tuning solves.
A training cluster is a single synchronous machine. Thousands of GPUs move through compute and communication phases together, so they draw power together and idle together. Research published in 2025 by NVIDIA, Microsoft and OpenAI documents GPU racks transitioning from roughly 30% of full draw to 100% and back within milliseconds. Google’s engineering team, in a paper on power and thermal fluctuations in ML infrastructure, describe AI workloads swinging by around 15 megawatts against roughly 1.5 megawatts for the non-AI workloads that preceded them, and makes the underlying point directly: traditional data centres host tens of thousands of workloads with uncorrelated power profiles, whereas batch-synchronised ML training does not. Meta encountered the same phenomenon during Llama 3 training, where simultaneous power changes across tens of thousands of GPUs produced fluctuations in the tens of megawatts and stressed the limits of the data centre’s power delivery.
For a practitioner, this matters in two ways. First, it constrains what a facility will let you do: a site that cannot absorb your transients will impose ramp limits, power caps, or simply decline the deployment, and each of those shows up in your effective throughput. Second, the mitigations are now landing in the hardware you are about to rent. NVIDIA’s GB300 NVL72 places electrolytic capacitors in the power shelf, roughly 65 joules per GPU, occupying around half the volume of the power supply, charging during the quiet phases and discharging into the peaks. It also implements a startup power cap and a deliberate power taper at job end. NVIDIA reports the combination cutting grid-facing peak power by up to 30% for the same GPU output.
That 30% is a provisioning number as much as a stability number. Facilities are sized for worst-case instantaneous draw. Flatten the peak and the same connected megawatts support more compute. When capacity is rationed, and in the UK it currently is, this is the difference between getting your cluster and joining a queue.
Channel four: failures, which scale against you
The most under-modelled facility effect is hardware failure, and the Llama 3 405B run gave the field its best public dataset.
Over a 54-day pre-training snapshot on 16,384 H100s, Meta recorded 466 job interruptions. Of these, 47 were planned maintenance and 419 were unexpected. GPU-related issues accounted for 58.7% of the unexpected interruptions: 148 (30.1%) from GPU failures including NVLink, and 72 (17.2%) from HBM3 memory failures. Network switch and cable problems caused 35 (8.4%). Two CPU failures were recorded in the entire period. That works out to roughly one unexpected failure every three hours, or about 50,000 GPU-hours between failures.
The arithmetic that follows is what matters for anyone planning a larger run. Mean time between failures for the cluster falls roughly linearly as you add GPUs. Double the cluster, and you halve the interval between interruptions. Since a synchronous training job typically has to stop and restart from a checkpoint when any participating GPU fails, this failure rate is a direct tax on throughput. Epoch AI’s modelling of the Llama 3.1 405B run estimates that around 2.1% of total training time went on checkpointing and failure recovery, equating to roughly 1% of MFU lost across the run, with optimal checkpointing every 4 minutes. That is a manageable overhead at 16,384 GPUs. It is manageable because checkpoint writes are fast relative to the failure interval, and that relationship degrades as clusters grow.
The facility’s contribution here is thermal. GPU and HBM3 failures dominate the failure distribution, and both are temperature-sensitive components under sustained high power. Thermal cycling, the repeated expansion and contraction that comes with a load swinging between idle and full draw, is a recognised driver of solder and interconnect fatigue. This is where the stability result from the cooling study reconnects: a component held between 41 and 50°C is not experiencing the same cycling as one oscillating between 54 and 72°C. It is worth being precise about the strength of this claim, because it is the weakest link in the chain. Nobody has published a controlled study attributing GPU failure rates to cooling architecture at cluster scale over a full hardware lifetime. The mechanism is well established in electronics reliability generally, and the thermal difference is measured, but the specific quantitative link from cooling architecture to GPU MTBF remains an inference rather than a measurement. Treat it as a plausible mechanism with strong priors, not a proven number.
What this means when you are specifying compute
Pull the four channels together, and a short list of questions falls out, aimed at whoever is actually procuring your infrastructure.
On thermal, ask for measured sustained GPU temperatures under load rather than a design specification, and be realistic about what you are buying: stability and power efficiency, and probably longevity, rather than a large throughput gain. On power, ask what happens when your cluster’s draw swings by a large fraction of its rating in milliseconds. Is there a contractual ramp limit? Is rack-level energy storage present, permitted, or prohibited? Who owns the buffering when several tenants’ loads move in phase? On efficiency, insist on knowing whether quoted PUE figures were measured on air-cooled hardware with fan power counted as IT load, because otherwise the comparison is meaningless. On failures, ask what the facility’s own contribution to interruption rates has been, and what the thermal history of the hardware you are being offered actually looks like.
The reason this list points towards purpose-built data centre colocation rather than a general-purpose facility is not that colocation is inherently virtuous. It is that these are engineering questions with engineering answers, and only a facility designed around high-density liquid-cooled loads can answer them with numbers rather than adjectives. A hall built for 5 kW racks and retrofitted for 130 kW ones will have to improvise, and improvisation shows up in your logs.
The limits of the evidence
Intellectual honesty requires flagging what the studies above do not establish. The cooling benchmark compared two nodes that differed in CPU as well as cooling architecture, which is a genuine confound the authors acknowledge. It tested single nodes, not clusters, so it captures none of the multi-node communication overhead that dominates real training economics, and it used QLoRA fine-tuning rather than full pre-training, which have different power characteristics. Its throughput findings are, as discussed, modest in production and dramatic only under synthetic stress. The Llama 3 failure data is a single cluster of one hardware generation observed over 54 days. The reliability inference connecting cooling to failure rates is exactly that, an inference.
What survives all of those caveats is the core claim, which is narrower than the marketing version but more defensible: the physical facility is not neutral with respect to your training run. It shows up in your power bill with certainty, in your failure rate with high probability, and in your throughput under some conditions and not others. The practitioners who know which is which are better placed than the ones who have been told the building does not matter, and rather better placed than the ones who have been told it fixes everything.
Special thanks to Gavin Lester for his input in this article.
Gavin Lester is COO at Carbon-Z, a UK colocation provider running high-density and liquid-cooled data centres for AI and HPC workloads.
Artificial Intelligence – The Data Scientist