Does the model learn from my conversation?

Not during the conversation. Weights do not change at inference. Providers may later use collected conversations to train a future version, which is a separate decision governed by their data settings, not something happening live in your chat.

Why does training need so much more hardware?

Training processes vast batches and has to keep gradients and optimizer state for every weight, several times the size of the model itself. Inference only reads the weights forward. That is why training runs on large clusters for months and one answer runs on a fraction of that.

What is fine-tuning, then?

A small, late round of training on a specific dataset, done after the expensive pretraining. It changes weights, so it is training, but at a scale a small team can afford. Once it finishes, using the result is inference again.

What does inference compute actually mean?

Inference compute is the processing spent answering requests with a finished model, as opposed to training compute, the processing spent creating it. When analysts say inference now dominates AI compute, they mean the world's chips increasingly spend their cycles serving answers rather than learning weights. Reasoning models push further in that direction, because they burn extra tokens thinking before they reply.

Is inference cheaper than training?

Per request, enormously cheaper: fractions of a cent against tens of millions of dollars. In total, usually not. Training is paid once, while inference is paid on every request and grows with users, so for a popular product the accumulated inference bill overtakes training and is commonly estimated at 80 to 90 percent of lifetime compute cost.

How much energy does one AI answer use?

The two public figures are both self-reported from 2025. Google measured a median Gemini text prompt at about 0.24 watt-hours, and OpenAI's Sam Altman put an average ChatGPT query at about 0.34 watt-hours, roughly what an oven draws in a second. Long reasoning answers cost more, and neither figure includes the energy spent training the model.

Why are reasoning models so much more expensive to run?

They spend compute thinking before they answer: the model generates a hidden chain of reasoning tokens, sometimes thousands of them, and API providers bill those as output tokens. That buys accuracy at inference time instead of training time, and it is a big part of why the industry's compute bill keeps shifting toward inference.

Plain-language explainer

Training and inference, the two halves of AI

Training is the one-time process that sets a model's weights by showing it data and adjusting its numbers after every miss. Inference is what happens every time the finished, frozen model answers a request. Training costs a fortune once; inference costs a little, millions of times, and now dominates the total compute bill.

What is the difference between training and inference?

Training is where the numbers are learned: the model reads enormous amounts of text, gets things wrong, and every weight is nudged to be less wrong. It happens once, costs a fortune, and then stops. Inference is every time you use the result: the weights are frozen and read, not changed, to turn your prompt into an answer. Nothing you type during inference is learned. That single fact explains why a model does not know today's news, why it forgets your last conversation, and why answers are billed per token.

Last reviewed September 6, 2026

Training and inference side by side.

	Training	Inference
When it happens	Once, before release, plus the occasional fine-tune	Every request, for the life of the product
What changes	The weights, after every batch	Nothing; the weights are frozen
Hardware	Thousands of GPUs wired together for weeks or months	One to a few GPUs per replica, many replicas
Cost shape	A huge one-time bill	A small bill, paid millions of times
Who does it	A handful of labs	Every product that calls a model

Do not just read it. Operate the mechanism yourself in a short interactive lesson.

See it work: From the internet to your answer →

Free, no code, no signup.

Ten seconds, no lesson: play with the standalone widget, and embed it in your own page.

What actually changes during training?

A language model is, physically, a giant list of numbers called weights, billions of them. At the start of training they are random and the model's guesses are garbage. Training shows the model a scrap of text with the next token hidden, lets it guess, measures how wrong it was, then nudges every weight a tiny step in the direction that would have made the error smaller. That loop is gradient descent, and it repeats trillions of times across a mountain of text.

The defining mechanical fact is that the weights change. Every training step is a write. Nothing is stored as sentences or facts you could point to. What accumulates is a slow shaping of the same billions of numbers, until the patterns of language, and a lot of knowledge about the world with them, are pressed into their values.

Training also carries heavy bookkeeping. To know which direction to nudge each weight, the system keeps gradients and optimizer state alongside the weights, which in standard setups takes several times the memory of the model itself. That is a big part of why training needs a cluster and answering you does not. When the run ends, the bookkeeping is thrown away and the weights are written to disk. That file of frozen numbers is the model.

What happens during inference, if the weights never change?

Inference is everything after: each time you send a prompt, the weights are read, never written. Your text becomes tokens, the tokens flow forward through the layers, and out comes a score for every possible next token. One gets picked, appended to the text, and the whole pass repeats for the token after that, until the answer ends. It is pure reading, one forward pass per generated token.

The consequences follow directly. The model cannot absorb new facts from your conversation, because learning means changing weights and inference never touches them. Its knowledge stops at the training cutoff date. It forgets you between sessions unless the product saves notes and pastes them back into the prompt. When a chatbot seems to remember your preferences, that is a product feature wrapped around a frozen model, not the model learning.

The border between the phases is sharp, but it can be crossed on purpose. Fine-tuning is a deliberate return to training: a short, comparatively cheap run on a specific dataset that really does change the weights, then freezes them again. What does not happen in mainstream products, as of mid-2026, is live learning from your chats. The interactive lesson on this site lets you run both halves yourself: nudge weights downhill to fix a wrong prediction, then freeze them and watch the same numbers answer a prompt.

How different is the hardware for training and inference?

Training a frontier model is one enormous, synchronized job. The model and its bookkeeping are sliced across tens of thousands of accelerator chips that must exchange gradient updates every step, so they sit in one physical cluster wired with very fast links, running for weeks or months. Meta reports about 31 million GPU-hours on H100 chips to train Llama 3.1 405B, on up to roughly 16,000 GPUs at once, and the biggest training clusters reported in 2026 pass 100,000 chips.

Inference has the opposite shape. One reply needs only enough hardware to hold the weights and stream them, typically a single machine with a handful of chips. A provider serves millions of users by running thousands of independent copies of the model, spread across data centers near those users. Nothing synchronizes with anything. Training is one giant computation. Inference is a swarm of small ones.

The bottlenecks differ too. Training wants raw arithmetic and fast chip-to-chip links. Generating text is mostly limited by memory bandwidth: to produce each token, the chip streams the model's active weights out of memory, so how fast memory can feed the processor matters more than how fast it multiplies. That is why inference-focused chips exist at all, from Google's TPUs to AWS's Inferentia to Groq's processors, tuned to serve answers cheaply rather than to train anything.

Which costs more, training or inference?

Per event, no contest. A frontier training run is among the most expensive single computations ever performed. The Stanford AI Index, using Epoch AI's estimates, put the training compute for GPT-4 at roughly $80 million and Gemini Ultra at roughly $190 million, and lab executives have talked openly about billion-dollar runs. One inference request costs the provider far less: fractions of a cent for a short answer from a small model, cents, sometimes dollars, for a long reasoning answer from a frontier one.

But training is paid roughly once, and inference is paid on every request, forever, scaling with your users. A hit product answers billions of prompts a day, a figure OpenAI reported crossing in 2025 for ChatGPT alone. Industry estimates commonly put inference at 80 to 90 percent of the lifetime compute cost of a production AI system. The one-time fortune loses to the trickle that never stops.

The table shows the asymmetry in rounded numbers as of mid-2026. The exact figures drift constantly. The shape does not.

Illustrative orders of magnitude as of mid-2026, from the Stanford AI Index, provider papers, and list prices. Rounded on purpose: the shape is the point.

Cost item	Rough size	When you pay
GPT-4 training run, estimated compute	about $80 million	once
Gemini Ultra training run, estimated compute	about $190 million	once
Llama 3.1 405B training, reported	about 31 million GPU-hours	once
Short answer, small model	fractions of a cent	every request
Long reasoning answer, frontier model	cents to a few dollars	every request
ChatGPT-scale traffic	billions of prompts a day	every day

Why is inference taking over the total compute bill?

For years the AI compute story was training: bigger runs on bigger clusters. Then the products got users, and the balance flipped. Deloitte has estimated that inference was about a third of AI compute in 2023, about half in 2025, and around two thirds in 2026, and Gartner projects that most AI infrastructure spending supports inference from 2026 onward. The reason is mechanical. Training grows with the number of new models. Inference grows with the number of uses, and uses grow much faster.

Reasoning models accelerated the flip. A reasoning model spends extra compute at answer time, generating a hidden chain of thought before it replies, so one hard question can burn thousands of tokens instead of a few hundred. Researchers call this test-time compute: capability bought at inference instead of in training. At the extreme, one widely reported benchmark run of an OpenAI reasoning model at its highest setting spent compute worth thousands of dollars on a single task. All of it inference.

Agents multiply it again. An agent handling one request may call the model dozens of times while it reads files, browses, and retries, and every step is more inference. Accounting-wise, the industry's bet on reasoning and agents moves even more of the total bill from a one-time training line to a per-use inference line. That is why new data centers are increasingly described as inference capacity rather than training capacity.

What makes serving answers fast its own engineering problem?

Training has no user waiting on it. If a step takes a second longer, nobody notices. Inference has a human staring at a blank screen, so it is judged on two clocks: time to first token, and tokens per second after that. Serving teams live inside a tradeoff between latency for one user and throughput for all of them, and two techniques dominate the field.

Batching. Producing one token means streaming the model's active weights from memory, and doing that for a single user leaves most of the chip's arithmetic idle. So serving systems run many users' requests through the model together, sharing each read of the weights across the whole batch, and continuous batching adds and removes requests mid-flight to keep the chip full. It is the main reason serving millions of users does not take millions of GPUs.

The KV cache. Attention needs the Key and Value vectors of every earlier token in the conversation. Recomputing them for each new token would make long chats crawl, so the server caches them, and each new token computes only against the cache. The price is memory: the KV cache grows with context length and with every concurrent user, and at long contexts it can rival the weights themselves. Prompt caching, reusing that stored work across requests, is why providers bill repeated prompt prefixes at a discount. None of this machinery exists in training. Serving is a discipline of its own.

What people get wrong

The model keeps learning from your chats. Weights are frozen at inference. Anything it seems to remember was put back into the prompt by the product.
Training is the expensive part. It is a huge one-off, and for a popular product the accumulated cost of serving answers passes it.
A model can just be updated with new facts. Changing weights means a training run; that is why current information arrives through retrieval and tools instead.
Inference is the trivial half. Serving fast, cheap answers to millions of users is its own engineering field, with dedicated chips, caches, and batching tricks, and it is where most of the money now goes.

Where you see it in real products

A knowledge cutoff date in the model's own description.
Per-token API pricing, which is a price on inference only.
A chat that forgets you between sessions unless the product saves notes and pastes them back.
The thinking indicator on reasoning models, which is extra inference compute burning while you wait.

Common questions

Does the model learn from my conversation?: Not during the conversation. Weights do not change at inference. Providers may later use collected conversations to train a future version, which is a separate decision governed by their data settings, not something happening live in your chat.
Why does training need so much more hardware?: Training processes vast batches and has to keep gradients and optimizer state for every weight, several times the size of the model itself. Inference only reads the weights forward. That is why training runs on large clusters for months and one answer runs on a fraction of that.
What is fine-tuning, then?: A small, late round of training on a specific dataset, done after the expensive pretraining. It changes weights, so it is training, but at a scale a small team can afford. Once it finishes, using the result is inference again.
What does inference compute actually mean?: Inference compute is the processing spent answering requests with a finished model, as opposed to training compute, the processing spent creating it. When analysts say inference now dominates AI compute, they mean the world's chips increasingly spend their cycles serving answers rather than learning weights. Reasoning models push further in that direction, because they burn extra tokens thinking before they reply.
Is inference cheaper than training?: Per request, enormously cheaper: fractions of a cent against tens of millions of dollars. In total, usually not. Training is paid once, while inference is paid on every request and grows with users, so for a popular product the accumulated inference bill overtakes training and is commonly estimated at 80 to 90 percent of lifetime compute cost.
How much energy does one AI answer use?: The two public figures are both self-reported from 2025. Google measured a median Gemini text prompt at about 0.24 watt-hours, and OpenAI's Sam Altman put an average ChatGPT query at about 0.34 watt-hours, roughly what an oven draws in a second. Long reasoning answers cost more, and neither figure includes the energy spent training the model.
Why are reasoning models so much more expensive to run?: They spend compute thinking before they answer: the model generates a hidden chain of reasoning tokens, sometimes thousands of them, and API providers bill those as output tokens. That buys accuracy at inference time instead of training time, and it is a big part of why the industry's compute bill keeps shifting toward inference.

Related explainers

How LLMs work Gradient descent Inference economics GPUs for AI

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.