Attention as a Weighted Average
Softmax the scores, then blend the values.
💡 The intuition
You're in a noisy room deciding how much to listen to each speaker. Attention scores are how relevant each voice is. Softmax turns those scores into listening percentages that add to 100%. The output is a blend: mostly the person you trust most, with a little from the others. Attention is just a smart weighted average that decides what to focus on.
weight 66.5%
weight 24.5%
weight 9.0%
Blended output
x: 0.665·8 + 0.245·6 + 0.090·2 = 6.97
y: 0.665·3 + 0.245·2 + 0.090·5 = 2.94
output = [6.97, 2.94]
The output is a new blended vector — pulled toward whichever value has the highest score. Every value still contributes something; nothing is fully ignored.
A tiny worked example
scores [2, 1, 0] → softmax weights [0.665, 0.245, 0.090]
values v1=[8,3], v2=[6,2], v3=[2,5]
x: 0.665·8 + 0.245·6 + 0.090·2 = 6.97
y: 0.665·3 + 0.245·2 + 0.090·5 = 2.935 → output ≈ [6.97, 2.935]
Free lesson complete
Continue with the complete Core AI Intuitions path.
Pro unlocks the surrounding lessons, runnable practice, and the full build-ready sequence.
Why it matters in AI
- ▸This weighted average is the core of the transformer attention block.
- ▸Scores come from query·key similarity; values are what actually gets blended.
- ▸Higher score → more weight → that token contributes more to the result.
- ▸Every token attends to every other this way, which is how context flows through an LLM.
Things people get wrong
✗ Attention picks one token and ignores the rest.
✓ It's a soft blend; every value contributes something (0.090 here).
✗ The weights are the raw scores.
✓ Softmax converts scores into weights that sum to 1.
✗ The output is one of the input values.
✓ It's a new mixed vector, [6.97, 2.935], not any single v.
Related reading · Knowledge Lab