Softmax Probabilities
Turn raw scores into probabilities that sum to one.
๐ก The intuition
Think of an election. Raw vote counts can be any size (even negative in the logit world). Softmax converts them into percentages of the whole โ and because it exponentiates first, a small lead in votes turns into a big lead in share.
raw logit (score)
raw logit (score)
raw logit (score)
Divides logits before softmax โ low = sharper, high = flatter.
Probabilities (always sum to 100%)
ฮฃ = 100%p_i = exp(logit_i / T) / ฮฃ exp(logit_j / T)
Each score is exponentiated (always positive, amplifies gaps) then divided by the total so everything sums to 1. Subtracting the max logit first keeps it numerically stable โ same answer, no overflow.
A tiny worked example
softmax([2.0, 1.0, 0.1])
exp โ 7.389, 2.718, 1.105 ย (sum = 11.212)
divide โ [0.659, 0.242, 0.099] ย (sums to 1.0)
A logit gap of 1.0 becomes about a 2.7ร probability ratio.
Free lesson complete
Continue with the complete Core AI Intuitions path.
Pro unlocks the surrounding lessons, runnable practice, and the full build-ready sequence.
Why it matters in AI
- โธNext-token generation turns the model's logits into a probability per token.
- โธAttention weights are a softmax, so they always sum to 1.
- โธClassifiers output a probability per class via softmax.
- โธTemperature scales logits before softmax to control randomness.
Things people get wrong
โ Logits are already probabilities.
โ No โ logits are raw scores that can be negative and don't sum to 1. Softmax fixes both.
โ Temperature changes which option is on top.
โ It only changes peakedness. The ranking is preserved; the winner stays the winner.
โ You can skip subtracting the max.
โ Real logits can overflow exp(). Subtracting the max is the standard, safe trick โ same result.
Related reading ยท Knowledge Lab