What's Inside the Box: AI Data Centers

Inside one server

An NVIDIA DGX H100: the standard building block of an AI cluster

8 × NVIDIA H100 GPUs do the math. Each has 80 GB of high-bandwidth memory (HBM) right beside it.
NVLink switches join every GPU to every other at 900 GB/s, so the 8 act like one.
2 × Intel Xeon 8480C CPUs (56 cores each, 2 TB of memory) run the system, feed in data and schedule work.
NVMe flash (about 30 TB): training data and checkpoints while training; model weights and cached prompts while serving.
8 × 400 Gb/s network cards link it to the rest of the cluster.

Xeon CPU
56 cores, 1 TB

NVMe flash
30 TB

Network
8 × 400 Gb/s

PCIe switches: CPUs, storage and network reach the GPUs

H100
80 GB
HBM

H100
80 GB
HBM

H100
80 GB
HBM

H100
80 GB
HBM

H100
80 GB
HBM

H100
80 GB
HBM

H100
80 GB
HBM

H100
80 GB
HBM

NVLink switches: every GPU to every other at 900 GB/s

Whole server: up to 10.2 kW, from six 3.3 kW power supplies

The GPUs do the work; the CPUs coordinate. Everything else in the box exists to keep the GPUs fed with data.

Source: NVIDIA, “Introduction to NVIDIA DGX H100/H200 Systems,” DGX H100/H200 User Guide (docs.nvidia.com): 8 × H100 (640 GB total), 2 × Xeon 8480C, 2 TB RAM, 8 × 3.84 TB NVMe, 8 × ConnectX-7 at 400 Gb/s, 900 GB/s NVLink, 10.2 kW max. Prompt caches kept on disk: e.g. DeepSeek API, “Context caching on disk,” 2024.

HON 4013 · AI, FINANCE, AND HUMAN FUTURES

The technology of an AI data center: the chips, the power, the heat, the water, and the models that run on them

Movie still of four boys huddled on a school porch, one wide-eyed, with the caption: The datacenters are coming.

GPU

Server

Rack

Cluster

Campus

Dr. Jesse Roberts  ·  Tennessee Technological University  ·  Fall 2026

All slides, one per page

Large models are mostly matrix math

Every layer multiplies a matrix of numbers by a matrix of learned weights

One blue output = one row × one column, multiplied pair by pair and added up. In Kimi K2 that is 7,168 multiply-adds.

Attention: matrix multiplies turn each word into queries, keys and values, then mix them.
Feed-forward experts: more matrix multiplies, in every layer.
The rest (softmax, activations, normalization) is a sliver.

~74 billionmath operations for every word Kimi K2 writes (with a 4,096-word context)

99.9%of them are matrix multiplications: the same simple operation, repeated billions of times

Running a model is multiplying huge matrices, over and over, so the question is which chip does that best.

Source: Moonshot AI, Kimi K2 model card, 2025 (hidden size 7,168; 61 layers; 64 attention heads; 32B active parameters). Operation count and 99.9% share: our estimate at a 4,096-token context (2 operations per active weight plus attention; non-matrix work counted at ~5 operations per value).

GPUs are efficient and fast at math

NVIDIA H100 vs. Intel Xeon 8480C, the CPU that sits beside it in the same server

CPU: a few big cores

GPU: thousands of small units

H100 GPUXeon 8480C CPU

Power700 W350 W

Parallel FP32 lanes16,8961,792

Peak math, FP3267 TFLOPS7.2 TFLOPS

Peak math, BF16 tensor cores989 TFLOPS—

Memory bandwidth3,350 GB/s307 GB/s

Power ≈ switches per second × energy per switch. Each transistor that flips charges a tiny capacitor, and that energy ends up as heat.
This CPU spends its transistors on making one thread fast: branch prediction, reordering, big caches. Most are idle in any given cycle.
This GPU spends them on arithmetic and keeps thousands of units busy every cycle.
Per operation, the GPU is more efficient: 4.7× more math per joule in FP32, 69× with its tensor cores. It draws more because it does 9× to 138× more math every second.

A GPU isn't wasteful, it's busy: more operations every second, packed 8 to a server and 72 to a rack.

Sources: NVIDIA H100 Tensor Core GPU datasheet (SXM: 700 W, 16,896 FP32 cores, 67 TFLOPS FP32, 989 TFLOPS dense BF16, 3.35 TB/s). Intel Xeon 8480 (56 cores, 2.0 GHz base, 350 W, 8 × DDR5-4800). CPU FP32 peak is our calculation: 56 cores × 2.0 GHz × 64 FLOPs per cycle (AVX-512); its AMX matrix units are not counted.

From chip to campus

An AI data center is the same few parts, nested five levels deep

GPU: does the math. The common one is the NVIDIA H100, about $25,000–40,000 new, or about $3.50–4.30 an hour to rent.
Server: 8 GPUs, 2 CPUs, memory, flash and network cards in one box: about $300,000–400,000.
Rack: a cabinet of servers sharing power and cooling. One NVIDIA GB200 NVL72 rack holds 72 GPUs and draws about 120–132 kW.
Cluster: thousands of racks wired into one network so they work as one computer.
Data center / campus: one or more buildings with power and cooling plants, sized in megawatts to gigawatts.

Data center building · 100s of MW

AI cluster · thousands of racks

Rack · 72 GPUs · ~130 kW

Server · 8 GPUs + 2 CPUs

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

CPU

CPU

Other compute

CPU and storage racks

What these are: next slide

Electrical room · power in

Cooling plant · heat out

Every layer is sized in watts: how much power it can feed in, and how much heat it can carry out.

Sources: H100 and DGX H100 prices: IntuitionLabs, “Data center GPU prices,” Sept. 2026 (citing Compute.exchange and Thunder Compute); rental: Jarvislabs, “H100 price guide,” Oct. 2026 (on-demand list prices). NVIDIA, DGX H100/H200 User Guide. GB200 NVL72: 120–132 kW (HPE, via Spheron 2026).

Not every rack is a GPU rack

A data hall seen from above: GPU, CPU and storage racks share one network

GPU racks train and serve models. About 120–132 kW each, so they must be liquid cooled.
CPU racks run the ordinary cloud: websites, databases, email. The average rack in the industry draws under 8 kW and is air cooled.
Storage racks hold flash and disks: data sets, model checkpoints, everyone's files.
The network links every server to every other, so a job can spread across the hall.

Data hall · top view

GPU racks
liquid cooled · ~130 kW each

network switches and fiber

CPU racks
air cooled · under 8 kW on average

Storage racks
flash and disk

About half of today's data centers are more than 11 years old: built for rows of CPU and storage racks, not 130 kW GPU racks.

One GPU rack draws about 15× the average rack, which is why older halls can't simply be filled with GPUs.

Sources: Uptime Institute, Global Data Center Survey 2024 (average rack under 8 kW; about half of sites over 11 years old), via Upsite, “Data Center Trends,” 2024. GB200 NVL72: 120–132 kW (HPE, via Spheron 2026). 15× = 120 kW ÷ 8 kW.

How much of it is GPU today?

Buildings are mixed, so analysts count electricity by type of server instead

Data-center electricity worldwide, by segment (TWh)

■ AI-optimized servers ■ conventional servers ■ cooling and other · 2027 is a forecast

+80 TWhadded by AI-optimized servers, 2025 → 2026

+2 TWhadded by conventional servers, 2025 → 2026

Counting buildings instead

About 11,000 commercial data centers worldwide; more than 50,000 if company server rooms count.
1,189 are hyperscale (early 2025), and they hold 44% of all capacity.

Different methods, different totalsGartner counts electricity by server type: 565 TWh in 2026. The IEA uses its own model: 415 TWh in 2024, with accelerated servers growing 30% a year and conventional ones 9% to 2030. Synergy counts buildings and capacity. Compare their directions, not their totals.

Conventional servers still draw a bit more than AI servers, but nearly all the growth is AI; they cross in 2027.

Sources: Gartner, “Data Center Electricity Consumption to Grow 26% in 2026,” press release, June 10, 2026, Figure 1 (segments may not sum to totals due to rounding). IEA, Energy and AI, 2025 (base case). Synergy Research, via TechInsights, June 2025 (hyperscale count and capacity share). Datacentres.com FAQ, 2026 (facility counts, an estimate).

What causes the power draw?

Talk it over with a neighbour before we move on

What a chip does all day, and the energy each operation takes

picojoules per operation, log scale · figures for a 45 nm chip

8-bit integer add

0.03

8-bit integer multiply

0.2

32-bit float add

0.9

32-bit float multiply

3.7

32-bit read, on-chip SRAM

5

32-bit read, off-chip DRAM

640

A GPU runs these operations trillions of times a second. What causes its power draw?

Source: M. Horowitz, “Computing's energy problem (and what we can do about it),” ISSCC 2014 (45 nm), as tabulated in S. Han et al., “Learning both weights and connections for efficient neural networks,” NeurIPS 2015. 170× = 640 pJ ÷ 3.7 pJ.

What causes the power draw?

Talk it over with a neighbour before we move on

What a chip does all day, and the energy each operation takes

picojoules per operation, log scale · figures for a 45 nm chip

8-bit integer add

0.03

8-bit integer multiply

0.2

32-bit float add

0.9

32-bit float multiply

3.7

32-bit read, on-chip SRAM

5

32-bit read, off-chip DRAM

640

A GPU runs these operations trillions of times a second. What causes its power draw?

■ doing math ■ moving data

Moving data costs far more than math. One read from off-chip memory ≈ 170× a 32-bit multiply.
It's all transistors switching; a longer wire is more charge to move, so distance is what costs.
Caching reuses each number, so training spends mostly on math. Serving one user at a time is memory-bound; the fix is batching.

It's all switching; distance makes it expensive. Good chips and good software keep numbers close to the math.

Every watt in becomes heat

A computer does no physical work: nothing moves, nothing is lifted, so the energy leaves as heat

Grid power
electricity in

Substation, UPS
small losses

IT equipment
GPUs, CPUs, memory, network

Heat
same watts that went in

Cooling system
pumps, fans, chillers

Outside
warm air or evaporated water

Power usage effectiveness (PUE) = total facility power ÷ power reaching the IT equipment

Google fleet, 2024

IT 100 kW

PUE 1.09

Industry average, 2024

IT 100 kW

overhead 56

PUE 1.56

Gray = cooling and power-conversion overhead for each 100 kW of computing

One rack at full load ≈ 87 space heaters

87 space heaters × 1.5 kW

1 GB200 NVL72 rack
~130 kW

Air can't carry that much heat away; it takes liquid on the chips.

Every watt in is a watt of heat out, so the cooling system is sized to the computers, watt for watt.

Sources: Google, 2025 Environmental Report (fleet PUE 1.09 in 2024), via DCD, 2025. Uptime Institute, Global Data Center Survey 2024 (average PUE 1.56). GB200 NVL72: 130–132 kW at full load, direct liquid cooling required (HPE, via Spheron 2026). 87 heaters = 130 kW ÷ 1.5 kW.

How data centers get rid of heat

A power plant cools its steam; a data center cools its chips. Two steps, each trading power for water

1
Off the chips

AirFans push cool air through the servers; hot and cold aisles kept apart. Fine for CPU racks.

Rear-door coilA liquid coil on the back of an air-cooled rack catches the hot exhaust. A retrofit for denser racks.

Direct-to-chip liquidCold plates on each GPU and CPU, in a closed loop. What 130 kW GPU racks need.

ImmersionWhole servers sit in a tank of non-conducting fluid; no server fans. Still uncommon.

2
Out of the building

ChillersRefrigeration, like a giant air conditioner. Most electricity.

Free coolingCold outside air or water does the work; chillers rest. Needs a cool climate.

Evaporative towerEvaporating water carries heat off. Little electricity, uses water.

Dry coolerFans blow heat into the air. No water, more electricity on hot days.

Sea or lake waterLike a power plant's once-through cooling: cold water in, heat out.

Why Google runs at a PUE of 1.09

83% lessoverhead than the industry average: 0.09 vs. 0.54 per unit of computing (2025)

40% lesscooling energy when DeepMind's AI tuned the pumps, chillers and towers (2016)

SeawaterHamina, Finland, cools with the Gulf of Finland through heat exchangers, mixing the warm water before it goes back

Cooling is a menu, not a fixed cost: site, climate and design decide how much power and water it takes.

Sources: Data Center Frontier, “Tech explainer: data center cooling,” July 20, 2026 (methods and trade-offs). Google, data center efficiency page (fleet PUE 1.09 vs. 1.54 industry average, Uptime Institute 2025). DeepMind, “DeepMind AI reduces Google data centre cooling bill by 40%,” 2016. DCD, “Google's Finland data center pioneers new seawater cooling.”

Where the water goes

Water is used for cooling twice: the power plant cools its steam, and the data center cools its chips

Power plant: cooling the steam

power lines

Data center: cooling the chips

A · Tower

B · Dry cooler

hot coolant back to the heat exchanger

Cooling the plant · indirect use, set by the grid's mix

litres evaporated per kWh, by how the plant is cooled (medians)

Wind, solar, hydro

≈ 0

Gas comb. cycle

≈0 dry · 0.4 once-thru · 0.8 tower

Nuclear

1.0 once-thru · 2.3 pond · 2.5 tower

Coal

0.9 once-thru · 2.6 tower (max 4.2)

Once-through cooling evaporates less but withdraws far more: ~170 L per kWh pulled from a river or lake for nuclear, most of it returned warmer.

A · Cooling tower (open loop)Evaporates water to dump heat: about 1.9 L per kWh on average; ~80% of the water drawn in is lost to the air.

B · Dry cooler (closed loop)Fans blow the heat into the air: almost no water, more electricity on hot days. Microsoft says this avoids 125+ million L a year per site.

For scale, a 100 MW site for one day: ~4.6 million L evaporated on site with towers (A), plus ~6.1 million L at a nuclear plant with towers, or ~0 L from wind.

On site, water is a design choice; off site, it's a grid choice. Towers save power, dry coolers save water.

Sources: Macknick et al., “Operational water consumption and withdrawal factors for electricity generating technologies,” Environ. Res. Lett. 7, 045802 (2012), NREL (gal/MWh, converted). EESI, “Data centers and water consumption,” 2025 (average WUE 1.9 L/kWh; ~80% evaporates). Microsoft, “Sustainable by design: next-generation datacenters consume zero water for cooling,” Dec. 9, 2024. 100 MW example: our calculation.

Training vs. inference

The same chips do two very different jobs

Training: build the model once

Weeks of work on tens of thousands of GPUs, all in lockstep.
Every GPU waits for the slowest one, so it needs one site and a very fast network.
Done once per model, then again for the next model.

Inference: use the model, every time anyone asks

Millions of small, independent requests.
Can be spread over many sites, closer to the users.
Grows with every user, every day the model is in service.
Judged per answer: time to first word, words per second.

Training is one huge synchronized job; inference is millions of small ones. Next: how to make the small ones cheaper.

Cheaper inference: batching

Every new word needs every weight read from memory. Make each read do more work.

Batch of 1

Weights in memory
read once per word

Math units
0.3% busy

1 user

~161 pJ per operation: almost all of it spent fetching the weight

Batch of 256

Weights in memory
the same one read

Math units
87% busy

~1.4 pJ per operation: one fetch now feeds 256 users' math

Why we want the GPU fullFetching a number from memory costs far more than the math done with it, and an idle GPU still has to be paid for and powered. Fewer fetches per operation means less energy per operation and more answers per second from the same chip.

Math units busy (H100, writing text)

Energy per operation (picojoules, log scale)

A full GPU is an efficient GPU: ~150× less energy per operation at batch 512 than at batch 1.

Sources: NVIDIA H100 datasheet (989 TFLOPS dense BF16, 3.35 TB/s). Energy: Horowitz, ISSCC 2014 (45 nm): 640 pJ per 32-bit DRAM read, 1.5 pJ per 16-bit multiply-add; newer chips are lower, the ratio is what matters. Our calculation: energy per op = 320 pJ ÷ (2 × batch) + 0.75 pJ; ignores attention and KV-cache reads.

Cheaper inference: caching

Keep the results of old calculations and reuse them, so each new word only adds its own math

Without a cache

With a KV cache

■ recomputed ■ read from the cache ■ new work · a 4-word prompt, then 6 new words

Prompt caching goes one step further: a shared beginning (a system prompt, a long document) is processed once and reused by every request that starts with it.

1,000-word prompt, 1,000-word answer750× less work1,500,500 token passes without a cache vs. 2,000 with one

The price: memory, for every user, all the time9.2 GBKimi K2 keeps ~70 KB per token in FP16, so one 128k-token conversation fills 9.2 GB of GPU memory

Caching reuses old calculations, so there is far less work to do; the price is memory.

Source: Moonshot AI, Kimi K2 model card, 2025 (61 layers, multi-head latent attention as in DeepSeek-V3: 512 latent + 64 position dimensions per layer). Our calculations: KV size = (512 + 64) × 2 bytes (FP16) × 61 layers = 70,272 bytes per token; token passes = 1,001 + 1,002 + … + 2,000 = 1,500,500.

Smaller models, 1: quantization

Same model, fewer bits per weight: it takes less memory, leaving room to serve more users

One weight, in bits

Math per second, NVIDIA B200

Memory for 1 trillion weights

BF16
16 bits

FP8
8 bits

FP4
4 bits

2,250 TFLOPS

4,500 TFLOPS

9,000 TFLOPS

2 TB

1 TB

0.5 TB

A smaller model leaves room for more users: freed memory holds more conversations at once, so the GPU stays full instead of waiting on data.
The catch is rounding error: test each model at each precision.

Kimi K2 on one 8 × B200 server (1,440 GB of GPU memory)BF162,000 GB weightsdoesn't fitFP81,000 GB weightsroom for ~47 usersFP4500 GB weightsroom for ~102 usersusers with a 128k-token conversation each (9.2 GB of FP16 KV cache)

A smaller model lets one GPU do more for the same power, if the model can tolerate the rounding.

Source: NVIDIA B200 specifications (HGX, 180 GB per GPU).

Smaller models, 2: distillation

A different, much smaller model, trained to imitate a big one

Teacher
DeepSeek-R1
671B weights,
37B used per word

“The capital of France is …” (illustrative)

Student
R1-Distill-Qwen-7B
7B weights

~5× less mathper word: 37B → 7B weights used

~96× less memoryto hold the model: 671B → 7B weights

Two ways to imitate: match the teacher's probabilities for every next word (more signal than a single right answer), or train on answers the teacher writes. DeepSeek made its distilled models the second way, from about 800,000 teacher-written examples.
The result: a model that fits on one GPU with most of its memory left for users' conversations, so many more people are served on the same hardware, while keeping much of the teacher's skill on the tasks it was shown.
The catch: the big teacher still has to be trained first, and the student only learns what the teacher demonstrates.

A smaller model lets one GPU do more for the same power: more users, less waiting on data.

Source: DeepSeek-AI, “DeepSeek-R1,” arXiv:2501.12948, 2025. Probabilities shown are illustrative.

Smaller models, 3: mixture of experts

A huge model whose effective size, for each word, is small

One word
(a token)

Router
scores all 384 experts, keeps the top 8

384 experts, each a small feed-forward network · 8 used for this word

Output
weighted sum

32B of 1Tweights used per word in Kimi K2 (3.2%)

~31× lessmath per word than a dense model of the same size

Kimi K2 has 384 experts in each layer; the router sends each word to 8 of them, plus 1 shared expert that every word uses.
The catch: all 1 trillion weights still sit in GPU memory, so mixture of experts saves math, not memory.

Each word runs a 32B model, not a 1T one, so one GPU does more for the same power.

Source: Moonshot AI, Kimi K2 README (8 of 384 experts active per token; 32B of 1T parameters).

Research changes the curve: linear attention

Optimization makes today's method cheaper. A new method changes how the cost grows.

full attention (n²)Kimi Linear hybrid (3:1)linear attention (n)rest of the model

FLOPs to read the whole context (y axis, log scale)

Reading 1 million tokens, one H100 at peak

Full attention1.4 × 10¹⁸≈ 23 min

Kimi Linear hybrid3.4 × 10¹⁷≈ 6 min

Linear attention4.7 × 10¹⁴≈ 0.5 s

Rest of the model6.7 × 10¹⁶≈ 68 s

Kimi Linear (Moonshot AI, Oct. 2025)A new attention, Kimi Delta Attention, keeps a fixed-size memory instead of looking back at every token. Three of these layers for every full-attention layer gave up to 75% less KV cache and up to 6× faster decoding at 1M tokens, while beating full attention in the authors' tests.

Optimization moves you along the curve; research changes the curve. At long contexts, n² → n beats every other trick.

Source: Kimi Team, “Kimi Linear: An expressive, efficient attention architecture,” arXiv:2510.26692, 2025. FLOP counts are our estimates for a Kimi K2-sized model (61 layers, 64 heads, head sizes 192/128, 128 × 128 linear state per head, 32B active), causal prefill; time at the H100's 989 TFLOPS dense BF16 peak.

What to take away

Talk it over with a neighbour before we move on

An AI data center is GPUs, CPUs, storage and network, nested chip → server → rack → cluster → campus, and every layer is sized in watts.
GPUs draw more power because they do far more math per second, not because they are wasteful.
Every watt becomes heat, and moving data over distance is the expensive part.
Water use depends on the cooling design on site and the grid's power sources off site.
Inference gets cheaper through engineering (batching, caching, quantization, distillation, experts), and fastest through new ideas like linear attention.

Five years from now, which will have cut the cost of AI the most: better chips, better engineering of today's models, or new ideas in the models themselves? Why?

Check the facts

Chips and racks

~74 billion operations per word (Kimi K2): 2 × 32B active weights + attention 61 layers × 64 heads × 2 × (192 + 128) × 4,096 words; 99.9% are matrix multiplies

7,168 multiply-adds per output: Kimi K2's hidden size

CPU peak 7.2 TFLOPS: 56 cores × 2.0 GHz × 64 operations per cycle

GPU 9× to 138× faster, 4.7× to 69× per joule: 67 or 989 TFLOPS vs. 7.2; then ÷ 700 W vs. ÷ 350 W

GPU rack ≈ 15× average rack: 120 kW ÷ 8 kW

+80 vs. +2 TWh: Gartner 2026 minus 2025 (175 − 95; 195 − 193)

Power, heat, water

Memory read ≈ 170× a multiply: 640 pJ ÷ 3.7 pJ

PUE overhead: 100 kW × (1.09 − 1) = 9 kW; × (1.56 − 1) = 56 kW

87 space heaters: 130 kW ÷ 1.5 kW

Litres per kWh: NREL gal/MWh × 3.785 ÷ 1,000 (e.g. nuclear tower 672 → 2.5)

100 MW site for a day: 100,000 kW × 24 h = 2.4 million kWh; × 1.9 L = 4.6 million L; × 2.54 L = 6.1 million L

Once-through withdrawal ~170 L/kWh: 44,350 gal/MWh × 3.785 ÷ 1,000

Cheaper inference

295 operations per byte: 989 TFLOPS ÷ 3.35 TB/s

Math units busy: (2 × batch operations ÷ 2 bytes) ÷ 295, capped at 100%

Energy per operation: 320 pJ ÷ (2 × batch) + 0.75 pJ → 161 pJ at batch 1, 1.1 at 512 (~150×)

Caching, 750× less work: 1,001 + 1,002 + … + 2,000 = 1,500,500 passes vs. 2,000

KV cache 70 KB per token: (512 + 64) × 2 bytes × 61 layers; × 131,072 tokens = 9.2 GB

Quantized weights: 10¹² × bits ÷ 8 = 2 / 1 / 0.5 TB; users = (1,440 GB − weights) ÷ 9.2 GB → 47 and 102

Distillation: 37 ÷ 7 ≈ 5× less math; 671 ÷ 7 ≈ 96× less memory

Mixture of experts: 1T ÷ 32B ≈ 31× less math per word

Linear attention: full = 61 × 64 × 320 × n²; linear = 61 × 64 × 7 × 128² × n; rest = 2 × 32B × n; time = operations ÷ 989 TFLOPS; crossover at 2 × 32B ÷ (61 × 64 × 320) ≈ 51k tokens

Debatable or uncertain

Hydro ≈ 0 water: NREL counts reservoir evaporation (~17 L/kWh). We don't charge it: the fair comparison, the same land with no dam, isn't a real option. Others do charge it.

Energy figures are from a 2014, 45 nm chip; today's are lower. The 640 pJ read is only confirmed as "over 500."

Estimates simplify: batching ignores attention reads; linear-attention cost is a rough model; server overhead is ignored.

Industry totals (Gartner, IEA, Synergy) use different methods; prices change monthly.

Sources are in the slides. Numbers are estimates.