Why not just pick the highest score?

Sometimes you do, that is greedy decoding. But training needs a smooth output it can learn from, and sampling needs real probabilities to draw from. Softmax turns arbitrary scores into a distribution that supports both.

What does temperature do to softmax?

It divides the scores before the exponential. Low temperature exaggerates the gaps, so the top option dominates and the model looks decisive. High temperature flattens them, so unlikely options get a real chance. It changes the shape of the distribution, never the underlying ranking.

Where is softmax used inside a transformer?

Twice, for different jobs. Inside attention it turns relevance scores into blending weights. At the very end it turns scores over the whole vocabulary into next-token probabilities. Same function, one mixes context and the other picks a word.

Plain-language explainer

Softmax, explained

What does softmax do, and where does an LLM use it?

Softmax turns a list of raw scores into percentages that add up to 100. It exaggerates gaps: a score slightly ahead becomes a share far ahead, and low scores shrink toward zero without ever quite reaching it. Language models use it in the two places that matter most: at the output, to turn word scores into the next-word probabilities you sample from, and inside attention, to decide how much each earlier word contributes to the current one.

Last reviewed July 25, 2026

Do not just read it. Operate the mechanism yourself in a short interactive lesson.

See it work: How attention blends meaning →

Free, no code, no signup.

What people get wrong

Softmax picks the winner. It only converts scores into shares. The picking, sampling or taking the top one, happens after.
The percentages are the model's confidence about facts. They describe likely text, not truth, which is why a wrong answer can carry a high share.
It is an obscure detail. The temperature dial you see in products works by reshaping scores right before this exact step.

Where you see it in real products

APIs that return logprobs are showing you the shares softmax produced.
The temperature setting is arithmetic applied just before softmax.
Attention heatmaps in AI visualizations are softmax shares drawn as color.

Common questions

Why not just pick the highest score?: Sometimes you do, that is greedy decoding. But training needs a smooth output it can learn from, and sampling needs real probabilities to draw from. Softmax turns arbitrary scores into a distribution that supports both.
What does temperature do to softmax?: It divides the scores before the exponential. Low temperature exaggerates the gaps, so the top option dominates and the model looks decisive. High temperature flattens them, so unlikely options get a real chance. It changes the shape of the distribution, never the underlying ranking.
Where is softmax used inside a transformer?: Twice, for different jobs. Inside attention it turns relevance scores into blending weights. At the very end it turns scores over the whole vocabulary into next-token probabilities. Same function, one mixes context and the other picks a word.

Related explainers

Temperature Attention How LLMs work

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.