Skip to content

Advertisement

DevOps Society

Tokens per Watt Sounds Precise Until You Try to Budget With It

Microsoft is scaling Maia to multi-gigawatt inference from 2027 and selling it on tokens per watt. Here is what that metric tells infra and FinOps leaders, and what it hides.

Published your local timeupdated

Tokens per Watt Sounds Precise Until You Try to Budget With It

Microsoft says its next inference chip will produce 40% more tokens per megawatt than leading GPU-based systems. Fine. Whose tokens, at what context length, at what latency, and for whom?

In mid-October Microsoft committed to scaling its Azure Maia chips to multi-gigawatt inference deployments from 2027, with Maia 300 entering production that year. The headline efficiency metric, tokens per watt, is a fair signal of how cheaply Microsoft can run its own AI services. It says very little about what an Azure customer will pay or get. If you run infrastructure or FinOps, treat it as a vendor claim to test against your own workload, not a number to plan with.

What did Microsoft actually commit to?

The announcement came in a blog post from Scott Guthrie, executive vice president of cloud and AI, on 12 October. Microsoft called Maia the primary inference system for its own products and services.

Maia 200 is already in production in the United States, Europe and Australia, serving Copilot and Teams. Microsoft said GitHub Copilot and Microsoft Foundry will run on it by the end of 2026, and claimed up to 1.7 times more tokens per watt than leading GPU infrastructure.

Maia 300 is the bigger story. According to InfoQ's write-up, it uses compute chiplets on TSMC N2P with eight HBM4 stacks per accelerator, for 288 GB of memory and 18 TB/s of bandwidth. Each accelerator is rated at 950 W. A rack holds 96 of them and draws 167 kW. Microsoft claimed up to seven times the rack-scale token throughput of Maia 200.

Note where the spec sheet spends its words. Memory. That tells you what Microsoft thinks the bottleneck is.

Why are practitioners skeptical of tokens per watt?

InfoQ's Steef-Jan Wiggers rounded up the pushback, much of it from comments on Satya Nadella's LinkedIn post.

Rick Bullotta called the metric absurd, on the grounds that there is no standard definition of a token and that counting them rewards verbose models. Michel Floyd asked whether the unit should be tokens per second per watt or tokens per kilowatt-hour. Vinay Bansal, who is building a competing inference accelerator, asked what context length sat behind the 40% figure. Phillip Pham of Pexon Consulting asked whether any of this reaches customers as cheaper reserved throughput.

Microsoft also says Maia 300 maximizes useful tokens per watt without defining useful, and it does not name the GPU systems it compared against.

None of these are cheap shots. They are the questions a capacity planner would ask before signing anything.

What does tokens per watt actually tell you?

Something real, to be fair. Inference at this scale is limited by power. A 167 kW rack is a data center planning problem before it is a software problem, and when a cloud provider is talking in gigawatts, the number of useful outputs per unit of electricity is how its own finance team thinks.

If you operate your own GPU fleet, you think this way too. Power and cooling in a colocation contract are fixed. More output from the same feed is free capacity.

What doesn't it tell you?

Quite a lot. Floyd's point is the easy one. A watt is a rate. A token count is a count. Somewhere you need time, and once you add it you are really measuring tokens per joule, which nobody puts on a slide because it sounds less exciting.

The harder gaps are these:

  • Tokenizers differ. The same paragraph can be a different number of tokens on different models, so tokens per watt does not compare across model families.

  • Input and output tokens are not the same work. Prefill on a long prompt and decode of a long answer stress the hardware differently. A blended number hides which one you are buying.

  • Throughput trades against latency. Bigger batches raise tokens per watt and also raise time to first token. A figure measured at a batch size your users would never tolerate is not your figure.

  • Context length changes everything. Bansal's point. Long-running agents are memory-bound, and the 40% may look different at the context lengths your agents actually use.

  • Quality is missing. More tokens is not more value if a model needs more of them to get to the same answer.

Can you even buy it?

Not directly. Neither Microsoft post says customers can provision Maia capacity. Matt Kimball of Moor Insights & Strategy told Data Center Knowledge that Maia mainly serves internal workloads, with the payoff showing up as better service and lower costs for Microsoft. AWS and Google sell their own accelerators as instances. Microsoft, so far, does not.

That is the catch. For a FinOps lead, any Maia efficiency gain arrives second-hand, through Copilot pricing or Foundry pricing, or it stays in Microsoft's margin.

What should you measure instead?

Measure the thing you pay for. That is cost per million output tokens, at your p95 latency target, at the context lengths your traffic actually has. If you own hardware, add energy per completed request, because a request is what your business counts. If you hold committed capacity, Pham had it right: price per guaranteed token is the number that decides whether you benefit.

And when a vendor gives you a tokens-per-watt figure, ask for the model, the batch size, the context length and the latency it was measured at. If they cannot tell you, the number is marketing.

Microsoft said it will share more at Ignite 2026 in San Francisco in November. Bring the questions above. The answers will tell you more than the headline did.

Advertisement

Follow DevOps Society on LinkedIn

Practical infrastructure engineering in your feed.

Follow

Written by

DevOpsSociety Editorial Team

Editorial Team

The DevOpsSociety Editorial Team covers DevOps, cloud infrastructure, Kubernetes, AI infrastructure, platform engineering, cybersecurity, FinOps, and modern engineering practices. We publish practical insights, technical guides, architecture analysis, and research for engineers and technology leaders.

More from DevOpsSociety →
The Infrastructure Briefing

Get the infrastructure briefing.

Practical DevOps, cloud, AI infrastructure and engineering insights, delivered weekly. Read by engineers and engineering leaders.

No spam. Unsubscribe anytime.

Tokens per Watt Sounds Precise Until You Try to Budget With It, DevOps Society