Test-time compute: pay at answer-time

What this lesson shows

A third scaling axis: spend more compute per question at inference to buy accuracy.

The question it opens with

Bigger models cost a fortune to train. Is training the only way to buy more capability?

The walkthrough, in the lesson's own words

• Training money is already spent. Drag the effort dial: buy a better answer for this one question, right now.
• Predict: as we spend more compute per question, what shape does accuracy trace?
• You just spent compute at answer-time. Name the contrast: train-time vs test-time.
• A third scaling axis, paid per question, on demand.
• The climb-then-flatten shape is the real, documented effect; the accuracy and cost numbers are invented, not benchmarks.
• 1× = one quick answer · 16× = think long, or sample 16 and pick the best. When you sample several tries instead of thinking longer, the tries can run side by side, so you pay more in compute, but not always more wall-clock time.
• Accuracy has nearly flattened up here, but cost hasn't: every extra try costs as much as the first. More thinking buys accuracy, at a price.
• Same weights, same question. The two train-time dials (more parameters, more data) were spent before this question was ever asked, this dial you turn at answer-time.
• What happens to accuracy as we spend more compute per question?
• Same weights, same question, we just let the model think longer, or sample several answers and keep the one most tries agree on.
• It climbs, then flattens. The first extra tries help a lot; later ones barely move the needle, even endless tokens can't push a model past its own ceiling.
• You guessed flat. Close call, but no: thinking out loud genuinely catches mistakes early on, so accuracy does rise at first.
• You guessed it climbs forever. A reasonable hope, but each extra try has less left to fix, so the gains shrink and the curve bends flat.
• How does more compute become a better answer?
• Two ways. The model can write a longer chain of thought, generating more reasoning tokens before it commits, or it can sample several independent answers and keep the one most of the tries agree on (a majority vote). Either way it's just more generated tokens (lesson 4.8), so the gains shrink once the easy mistakes are already caught.
• Both train-time dials are paid once, upfront, a fortune in GPUs, frozen into the weights. The dial you just dragged is different: it spends compute at answer-time, per question, on demand.
• A tricky logic puzzle just stumped your model. The textbook fix is to train a bigger one, but in lesson 4.4 we saw that both train-time dials (more parameters, more data) are spent before any question is ever asked. This lesson is about a dial you can turn at answer-time instead.
• Train-time scaling is fixed and bought in advance; test-time scaling spends more per question at inference (answer-time, as opposed to during training). It buys real accuracy, but cost and latency keep climbing with every extra unit of effort. That trade-off is exactly why reasoning models give you an effort dial.
• The same chatbot answers a trivia question instantly but takes 30 seconds on a hard math proof, and gets it right. What is it doing differently?
• It is spending more test-time compute on the hard question, thinking longer or sampling several tries and keeping the best. Practically: you can dial effort up only for the hard requests instead of paying for a bigger model on every one.
• More thinking buys accuracy, at a price. But where do you SET the dial for a given task? Next lesson.
• Accuracy rises steeply at first then flattens, while cost climbs in a straight line.

Key takeaway

You can buy capability at answer-time, not just at training-time.

What you can do after this lesson

You can explain test-time (inference) compute as a third way to scale, and its cost/latency trade-off.

Where it leads: But more thinking isn't always better, there's a dial, and a sweet spot.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.