Plain-language explainer
Softmax, explained
Softmax is the function that turns a list of raw scores into probabilities that add up to one. It exponentiates each score and divides by the total, which exaggerates gaps: a slightly higher score becomes a much larger share. Language models use it to turn next-token scores into probabilities and, inside attention, to turn match scores into weights.
What does softmax do, and where does an LLM use it?
Softmax turns a list of raw scores into percentages that add up to 100. It exaggerates gaps: a score slightly ahead becomes a share far ahead, and low scores shrink toward zero without ever quite reaching it. Language models use it in the two places that matter most: at the output, to turn word scores into the next-word probabilities you sample from, and inside attention, to decide how much each earlier word contributes to the current one.
Last reviewed
Do not just read it. Operate the mechanism yourself in a short interactive lesson.
See it work: How attention blends meaning →Free, no code, no signup.
Ten seconds, no lesson: play with the standalone widget, and embed it in your own page.
The softmax formula, in three steps
Written out, softmax of a score z is e to the power of z, divided by the sum of e to the power of every score in the list. In symbols: softmax(z_i) = e^z_i / Σ e^z_j. The raw scores going in are usually called logits.
In plain steps: raise e (about 2.718) to each score, add those results up, then divide each result by the total. The exponential makes every value positive, even when a score is negative, and it stretches the gaps between scores. Dividing by the total makes the outputs add up to exactly one, so you can read them as probabilities.
One useful property: adding the same number to every score changes nothing. Only the differences between scores matter. That is why code subtracts the largest score before exponentiating. The answer stays the same, and e to the power of 1,000 never gets computed and overflows.
A worked softmax example with real numbers
Say a model has read 'The cat sat on the' and gives three candidate next words these scores: mat 2.0, sofa 1.0, moon 0.1. Here is softmax, step by step.
Mat ends up with about two thirds of the probability. Look at the ratio: mat's score is 1.0 higher than sofa's, and its probability is 2.7 times higher. That is no accident. A gap of d in score always becomes a ratio of e^d in probability. A gap of 2 would be a ratio of about 7.4, and a gap of 5 about 148.
During training, this same number is the loss. If 'mat' was the real next word, the loss for this step is minus the log of 0.659, about 0.42. Training nudges the weights so the correct word's share grows and that number shrinks.
| Word | Score (logit) | e^score | Divided by 11.21 | Probability |
|---|---|---|---|---|
| mat | 2.0 | 7.389 | 7.389 / 11.21 | 0.659, about 66% |
| sofa | 1.0 | 2.718 | 2.718 / 11.21 | 0.242, about 24% |
| moon | 0.1 | 1.105 | 1.105 / 11.21 | 0.099, about 10% |
What temperature does to softmax, in numbers
Temperature divides every score before softmax runs. Below 1 it stretches the gaps, so the top word takes over. Above 1 it squeezes them, so the long shots get a real chance. The ranking never changes, only how sharp the distribution is.
With the same three scores, dropping the temperature to 0.5 gives mat 86% of the probability and leaves moon with 2%. Raising it to 2 takes mat down to half, and moon climbs to nearly a fifth. That is the whole mechanism behind the temperature slider in AI products.
| Temperature | mat | sofa | moon |
|---|---|---|---|
| 0.5, focused | 86.4% | 11.7% | 1.9% |
| 1.0, default | 65.9% | 24.2% | 9.9% |
| 2.0, loose | 50.2% | 30.4% | 19.4% |
Why softmax matters in real models
Every word a chatbot writes passes through one softmax over its whole vocabulary. GPT-2 scored 50,257 tokens at each step. Llama 3 scores 128,256. The model produces one score per token, softmax turns them into probabilities, and the sampler draws the next token from those.
Inside attention, softmax runs far more often. Each layer and each attention head runs one softmax per position, over every earlier position. Llama 3 8B has 32 layers with 32 heads each, so reading a 1,000-token prompt means about a million softmax rows before the first word comes out. That cost is why FlashAttention computes softmax in tiles with a running maximum, so the full score table never has to sit in memory at once.
It also shows up wherever a product exposes probabilities. The logprobs an API returns are the logarithms of these softmax outputs. Confidence scores in classifiers, token highlighting in playgrounds and the colors in attention heatmaps are all softmax shares.
Softmax vs sigmoid vs argmax
The three are often confused. Argmax picks the single top score and ignores the rest. Sigmoid squashes each score on its own into a value between 0 and 1, so several options can all be likely at once. Softmax looks at the whole list together and splits one unit of probability across it, so raising one option lowers the others.
| Function | Output | Adds up to 1? | Typical use |
|---|---|---|---|
| Argmax | The index of the top score | No, it is one choice | Greedy decoding, picking a final label |
| Sigmoid | Each score squashed to 0 to 1, independently | No | Yes or no questions, multi-label tags |
| Softmax | A share for every option | Yes | Next-token choice, attention weights, one-of-many classes |
What people get wrong
- Softmax picks the winner. It only converts scores into shares. The picking, sampling or taking the top one, happens after.
- The percentages are the model's confidence about facts. They describe likely text, not truth, which is why a wrong answer can carry a high share.
- It is an obscure detail. The temperature dial you see in products works by reshaping scores right before this exact step.
Where you see it in real products
- APIs that return logprobs are showing you the shares softmax produced.
- The temperature setting is arithmetic applied just before softmax.
- Attention heatmaps in AI visualizations are softmax shares drawn as color.
Common questions
- Why not just pick the highest score?
- Sometimes you do, that is greedy decoding. But training needs a smooth output it can learn from, and sampling needs real probabilities to draw from. Softmax turns arbitrary scores into a distribution that supports both.
- What does temperature do to softmax?
- It divides the scores before the exponential. Low temperature exaggerates the gaps, so the top option dominates and the model looks decisive. High temperature flattens them, so unlikely options get a real chance. It changes the shape of the distribution, never the underlying ranking.
- Where is softmax used inside a transformer?
- Twice, for different jobs. Inside attention it turns relevance scores into blending weights. At the very end it turns scores over the whole vocabulary into next-token probabilities. Same function, one mixes context and the other picks a word.
- What is the formula for softmax?
- For a list of scores, raise e to each score, then divide each result by the sum of all of them. Every output lands between zero and one and the outputs add up to one. The exponential is what turns a small lead in score into a large lead in probability.
- Why is softmax called soft?
- Because it is a smooth version of picking the maximum. A hard max gives all the weight to the single top score. Softmax gives most of the weight to the top score and a little to the rest, and it is differentiable, which is what training by gradient descent needs.
- Why not just divide each score by the sum of the scores?
- Raw scores can be negative or zero, so dividing by their sum can give negative or undefined shares. The exponential makes every value positive first, and it rewards a lead: a score 1 point higher gets 2.7 times the probability. Plain division would treat a small lead as a small difference.
Related explainers
More in Inside the transformer
One idea at a time, in your inbox
New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.
We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.