Jovan Stojkovic of UT Austin and three Microsoft Azure engineers built a model of one AI datacenter and let it age for fifteen years. Demand grows 15% a year. Accelerators refresh every two years. Power and cooling get sized once, then lived with.
The facility starts at roughly $0.2M a year in total cost. It ends near $0.3B a year. Same building, same operating team, different hardware generations.
That is the refresh trap. Token prices fall on one curve. The infrastructure bill falls on another, and the second one is set by decisions made years earlier.
The capital markets have noticed. The four largest hyperscalers are expected to spend $1.8 trillion through 2028, about $600 billion more than their businesses generate in cash over the same period, Arete Research analyst Jim Fontanelli said at the AI Infra Summit in Santa Clara in September.
Where the token price goes
Cheaper inference came from the software stack first. McKinsey's June analysis of 13 technology levers puts model optimization at 85–95% lower cost per token, 3D advanced packaging at 80–90%, and custom inference silicon at 70–80%. Combined over time, the firm projects a decline of up to two orders of magnitude.
List prices tell the same story from the buyer's side. TechTarget's October guide prices budget models at $0.14–1 per million tokens, midrange models at $2–15, and frontier models at $20–75. A workload of 20 million tokens a day at $5 per million runs about $3,000 a month before output tokens.
Those numbers reward patience. If compute gets cheaper every quarter, the rational move is to wait.
What a fifteen-year model sees
The Microsoft paper, published at ISCA 2026, splits datacenter life into three clocks. The building runs 15–30 years. IT hardware is amortized over 3–5 years for GPUs. Workload scheduling changes with every model release.
The clocks interact. A 10 MW facility holding about 500 H100 servers at 75% utilization consumes 70 GWh a year. IT equipment is 60–70% of annual cost. Scale the fleet to 25,000 servers and annual TCO reaches about $0.3B, with hardware, expanded power delivery and higher operating expenses arriving together.
For AI fleets, the hardware refresh dominates long-term cost because GPU performance, power, and costs evolve rapidly.— Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri and Ricardo Bianchini, ISCA 2026
Refresh policy is where the interaction becomes a bill. Fixed four-to-six-year cycles are close to optimal for general-purpose servers. They are a poor fit for accelerators whose performance per watt changes every generation.
The paper's simulation finds alternative refresh policies cutting TCO by 15–20%. Some configurations favor early replacement. Others favor extending the life of the existing fleet. The answer depends on the workload growth rate, the hardware roadmap and the cost of power at that site.
Power topology and cooling chosen at build time decide how cheaply the next three hardware generations can be installed.
Three decisions that set the floor
Power distribution comes first. Flatter domains, pooled across more equipment, cost more at build time and reduce stranded capacity later. The lifecycle benefit is 4.2% of TCO. The trade-off is fault isolation: a per-PDU design contains failures better, a per-datacenter design strands less capacity.
Cooling is the second. A hybrid design of 75% cold plates and 25% air reaches a PUE of 1.15 and cuts lifecycle TCO by 9%. Networking is the third. NVLink inside servers, InfiniBand inside racks and Ethernet across racks gives a 6% reduction by matching each tier to the traffic it carries.
None of these choices is cheap to reverse. A building shell, a power topology and a cooling loop are fifteen-year commitments. That is why the framework treats them as part of the refresh decision rather than as fixed inputs.
The operating layer
Software is the part that can move after the concrete sets. Model migration, quantization, KV-cache reuse, prefill/decode disaggregation and heterogeneity-aware scheduling each reduce TCO by 12–39% in the paper's model.
Smooth model migration is the largest single lever. It spreads demand across old and new accelerators instead of triggering a purchase order the week a model ships. Disaggregation stretches the same logic across hardware generations: run compute-heavy prefill on one class of chip, memory-bound decode on another.
Those percentages are per strategy, not additive. The value comes from the sequence. Migration first, then quantization and cache reuse, then scheduling across whatever mix of hardware the site already owns.
Put the stages together and the paper reports up to 40% lower TCO than siloed management. Build contributes 15%, IT provisioning 23%, operation 19%. The compounding is the point. Each stage widens or narrows the options available to the other two.
The utilization gap
Enterprise buyers rarely run at hyperscaler utilization. TechTarget's guide recommends at least 50% GPU utilization for smaller models and 80% or more for complex ones. Below that line, the cost per useful token rises even when the tariff falls.
Burst traffic makes the gap worse. A team that provisions for the daily peak pays for idle capacity the rest of the day. The guide's mitigation list is unglamorous: right-size instances, cache repeated prompts, batch non-real-time jobs and route simple queries to cheaper models.
The same logic appears in the lifecycle model. At 75% average utilization, a 10 MW facility supports about 500 H100 servers. Drop utilization and the fleet either shrinks, stranding facility capacity, or stays and strands compute. Both outcomes raise the cost per token that the invoice never shows.
The power multiplier
Facility overhead explains why refresh timing carries so much money. American Compute traces every watt from the GPU to the meter. Non-GPU server components add about 53% to GPU load. Shared networking adds another 14%. Cooling and power delivery add 40%.
Every 1 W of GPU load becomes 2.44 W at the meter. Air cooling takes 25–40% of total facility power. Direct-to-chip liquid cooling holds it below 15% and a PUE near 1.15. Above roughly 100 kW per rack, air stops being practical at all.
This is the lifecycle case for 800 VDC. As we wrote in October, the shift to 800 volts is usually framed as a rack-level efficiency story. The TCO framework makes it a refresh story: a facility wired for 120 kW racks can absorb two or three accelerator generations, and one wired for 30 kW cannot.
The hardware price curve reinforces the point. The paper prices a P100 at about $9,000 per GPU, roughly $90,000 per server. An H100 is about $30,000 per GPU and more than $350,000 per server. Capex per server has grown faster than CPU budgets, while the useful life of the accelerator has not.
What to ask a TCO model
Most enterprise cases still compare a GPU-hour price with an amortized server price. The framework suggests four questions that change the answer.
- What share of annual cost is IT equipment, not energy?
- What refresh cadence does the model assume, and what happens at two, four and six years?
- Which decisions are reversible after the shell is poured?
- What utilization does the model assume, and what does cost per useful token look like at half of it?
None of the four is about token price. Each changes the number by more than a typical model release.
The bill is the strategy
Cheap tokens expand the set of viable workloads. More workloads justify more capacity. More capacity makes refresh timing, power topology and cooling the dominant costs. The loop is rational, and it produces a bill that rises while unit prices fall.
Arete's $600 billion funding gap is the macro version. A hyperscaler that refreshes on a fixed cadence pays for hardware it cannot fully use. One that coordinates build, provisioning and operation buys the option to wait for a better generation without stranding capital.
The 40% is not a discount. It is the price of treating a datacenter as one system for fifteen years instead of three procurement cycles.