Inference economics: what an answer costs

What this lesson shows

Answers are served token by token: each token is a pass through the model, the KV cache holds GPU memory for your whole reply, and batching, output length, and thinking budgets set the bill. Quantization and routing are the levers that lower it.

The question it opens with

Training happened once, long ago. So why does every answer you ask for still cost the provider real time, memory, and money?

The walkthrough, in the lesson's own words

• The datacenter trained the model once. So why does each answer still cost the provider money?
• Build one answer's cost. Drag the length, flip on reasoning, switch the model tier.
• Same hard request. Flip a lever and watch the bill drop, without dropping quality.
• One answer costs time, memory, and money, every single time. Two levers cut it.
• Training finished months ago and the weights never change. So why does every answer you ask for still cost the provider real time, memory, and money?
• Training happened once. But an answer is generated token by token, and each token is a fresh pass through the whole model. Every reply reruns billions of multiplications, holds GPU memory the whole time, and the provider pays for all of it, every time.
• Toy mid-2026 numbers: output tokens billed a few dollars per million, GPU memory in the low tens of GB. Real prices vary by model and provider.
• Every output token is a separate pass, so length sets both the bill and the wait.
• Let the model think before it answers. Those hidden reasoning tokens are billed like output, so the count roughly triples.
• A bigger model costs more per token and decodes slower (writes the reply more slowly).
• Three dials set the cost of one answer: how many tokens it generates, whether it reasons first, and which model runs it. The KV cache, the scratch memory the model holds on the GPU while writing your reply (4.2), stays full the whole time, and that memory is the real scarce resource.
• A GPU serves many requests at once (batching, S.1/S.2): it runs one big matmul for the whole group instead of one per user. That spreads the fixed cost of loading the weights across everyone, which is what makes per-token prices this low. The catch: the KV cache for every request in the batch has to fit in GPU memory at the same time, so memory, not raw math, is usually what caps how many you can serve.
• Flip a lever below to cut the bill on this hard request.
• Run the model at lower numeric precision. Cheaper per token and roughly half the KV-cache memory, with little quality loss on most tasks.
• Send the request to a right-sized model instead of the biggest one. Most answers don't need the flagship.
• Illustrative savings. The direction is real, the exact multiplier is invented.
• The big cost is training the model. Once that's paid, answers are basically free.
• Training is a one-time bill. Serving is forever: every answer reruns the model token by token and holds GPU memory the whole time. Serving is now about two-thirds of all AI compute.
• An answer's cost is set by how many tokens it generates, whether it reasons first, and which model runs it. Quantization and routing are the levers that cut the bill without cutting quality.
• The scarce resource in serving is usually GPU memory, not math. The KV cache (4.2) grows with every token in the context and has to stay resident for the whole reply, so a long answer or a long conversation eats the memory that would otherwise serve other users. That's why batching, shorter outputs, quantization, and routing all matter: they let one GPU serve more answers per second.
• Your chatbot subscription caps how many 'thinking' answers you get each month, and long replies burn your quota faster. Why would the provider meter it like that?
• Every token you see costs GPU time and memory, and hidden thinking tokens are billed the same way. Long replies and long thinking multiply that cost, so the caps and the routing to smaller models are the provider passing that arithmetic on to you.
• KV cache: the scratch memory the model holds on the GPU while writing your reply (4.2)

Key takeaway

You traced where one answer's cost comes from, and found the lever that cuts the bill without cutting quality.

What you can do after this lesson

You can explain why an AI answer costs time and money, per-token serving, KV-cache memory, batching, output length, and name the levers that cut the cost.

Prefer reading first? This lesson has a companion explainer: Inference economics

Where it leads: Beyond the core arc: how models are aligned, tuned, and probed.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.