Why training needs datacenters

What this lesson shows

Cut the model into slices, one slice per GPU, and wire them together as one machine.

The question it opens with

A frontier model won't fit on one GPU, and needs enormous compute.

The walkthrough, in the lesson's own words

• 280 GB model, 80 GB chip. Tap the model to slice it until it fits.
• Here's the fix, worked out: split the model so each slice fits.
• Now you drive. Add GPUs until every slice fits.
• Scale is a physical problem: memory, wiring, and watts.
• Right. Cut into 4 slices, one per GPU, then wired together so they act as one machine. That's how every frontier model is trained.
• Still overflowing. Each slice is bigger than 80 GB, keep slicing, one more piece per GPU, until every slice fits.
• Round numbers: 280 GB is the weights alone, 140 billion parameters at two bytes each. 80 GB is an H100-class chip, the workhorse GPU of 2023-2025. Newer chips hold more, but frontier models still outgrow any single one.
• Why not just wait for a bigger chip, or shrink the model?
• No chip is close to big enough, and models grow faster than memory does, so waiting won't help. Shrinking (quantization, 4.6) is a serving trick; training already runs much of its math in low precision (BF16/FP8) and still needs extra room, the weights plus gradients plus optimizer state, several times the copy you'd serve. Bigger models learn better (the scaling laws from 4.4), so a frontier model must be this big while it's still learning, and the only fix is to slice it across GPUs.
• 280 GB ÷ 4 GPUs = 70 GB each, which fits in 80. So we split the model into 4 slices, one per GPU. Next, try it yourself.
• Split across GPUs and linked over NVLink, the chips work in lockstep as one giant machine.
• Honest counts: NVIDIA's GB200 NVL72 rack links 72 GPUs; frontier runs span tens of thousands across many racks for weeks.
• Once training ends, the datacenter's main job becomes serving answers, running the finished model for millions of users. In 2026, roughly two thirds of all AI compute goes to that serving work (called inference), not to training.
• That's why capacity, batching (grouping many people's requests into one pass through the weights), and speed shape every AI product you use.
• Racks like these draw serious power. Datacenters already use about 2% of the world's electricity, AI is roughly a third of that and the fastest-growing slice, and in 2026 companies can often buy chips faster than the grid can power them.
• So the physical problem isn't just memory and wiring: new AI datacenters are planned around power plants and grid hookups as much as around GPUs.
• Headlines say training a frontier model takes a whole datacenter and tens of thousands of GPUs. Why can't it just run on one very powerful machine?
• The model is far bigger than any single chip's memory, so it has to be sliced across many GPUs and linked with fast cables so they act as one. Scale is a physical problem of memory, wiring, and watts, which is why it takes a datacenter, not just a faster computer.

Key takeaway

Scale is a physical, not just mathematical, problem.

What you can do after this lesson

You can explain why frontier training needs datacenters: models are split across many GPUs.

Where it leads: The datacenter trained it once. But most AI compute now goes to answering, so what does one answer cost?

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.