Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
A new open dataset of 5,000+ energy experiments across 50 LLMs and 10 GPUs shows hardware choice alone can cut inference energy costs by up to 70%.
The Thesis
Running a large language model is not just a software problem — it is a hardware problem, and the wrong GPU can waste most of your energy budget. This paper introduces Watt Counts, the largest publicly available benchmark dataset for LLM energy consumption, covering 50 models tested across 10 NVIDIA GPU types in both batch and real-time server configurations. The central finding is that no single GPU is optimal for all models or workloads: the best hardware choice shifts depending on what model you run and how you serve it. For organizations running inference at scale, this means that hardware procurement decisions — not just model compression or quantization — are a first-class lever for sustainability and cost. The catch is that this study covers only NVIDIA GPUs and focuses on measurement rather than prescribing automated selection strategies.
Catalyst
LLM inference has scaled from a research curiosity to a significant share of data center power draw, making energy efficiency a genuine operational and regulatory concern. Simultaneously, cloud and colocation providers now offer a wide menu of GPU SKUs — from the H100 to older A10G and T4 chips — creating real optionality that operators lack the data to exploit. This paper fills that empirical gap at a moment when the decision actually matters financially.
What's New
Prior LLM benchmarks — such as MLPerf and various throughput-focused leaderboards — optimize for tokens-per-second or latency and treat energy as an afterthought, if they measure it at all. Some academic work has studied GPU power for training, but inference energy across heterogeneous hardware at this scale had no public, reproducible dataset. Watt Counts is the first open-access benchmark to systematically cross 50 models with 10 GPU types and publish the raw data alongside a community-extensible framework, letting practitioners reproduce and extend results rather than trust a single vendor's marketing.
The Counter
The 70% energy reduction headline is real but narrow: it applies to server scenarios where you can choose the optimal GPU for a specific model, which assumes procurement flexibility that most organizations simply don't have — you use what your cloud provider offers or what you already bought. The study covers only NVIDIA GPUs, so it tells you nothing about AMD MI300X, Intel Gaudi, or custom silicon, which are increasingly relevant for inference. All experiments were run on single-GPU setups, meaning multi-GPU and multi-node deployments — the norm for large models — are uncharacterized. The benchmark also does not model dynamic workloads, mixed-model serving, or the thermal and power constraints of shared data center environments. And since the dataset is static at publication, its value depends entirely on whether the community actually submits new results, which is uncertain.
Longs
- AMD (AMD) — alternative GPU vendor whose efficiency profile could look attractive if benchmark expands beyond NVIDIA
- SMCI (Super Micro Computer) — server integrators that configure heterogeneous GPU racks benefit from clearer hardware selection data
- GRID (Grindr is wrong — use) VRT (Vertiv Holdings) — data center power and cooling infrastructure directly tied to inference energy budgets
- ENPH or FSLR — renewable energy suppliers to hyperscale data centers face growing demand as inference power draw is quantified
- BOTZ (Global Robotics & AI ETF) — broad AI infrastructure exposure including inference hardware
Shorts
- Cloud providers (AWS, GCP, Azure) selling fixed GPU instance types — if customers can optimize hardware selection, they'll demand more granular options or move to bare-metal, pressuring high-margin managed inference products
- Vendors of inference-optimization software (quantization, pruning tools) whose value proposition competes with simply picking a more efficient GPU for a given model
- H100-only inference startups — the paper implies that the most expensive GPU is not always the most energy-efficient choice, undermining premium-GPU-centric business models
Enablers (Picks & Shovels)
- NVIDIA Management Library (NVML) — the low-level power measurement API that makes per-GPU watt readings possible
- vLLM — open-source LLM inference engine used widely for server-scenario benchmarking
- Hugging Face model hub — source for the 50 open-weight models tested in this dataset
- MLflow / Weights & Biases — experiment tracking infrastructure needed to manage 5,000+ reproducible runs
Private Watchlist
- Electricity Maps — real-time grid carbon intensity data that could be layered onto this benchmark for carbon-aware scheduling
- Baseten — inference deployment platform where hardware-aware routing is a direct product feature
- Modal Labs — serverless GPU infrastructure startup where energy-per-token cost is a pricing input
- Trainy — GPU training and inference optimization startup focused on efficiency
Resources
The Paper
While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.