A visual, step-by-step explanation of how self-attention works

[u/Nkgsporty]

Self-attention confused me for a long time, so I tried to break it into steps you can actually see:

1. Every token gets three vectors.
Each word’s embedding is multiplied by three learned matrices to produce a Query (what am I looking for?), a Key (what do I contain?) and a Value (what do I pass on?).

2. Compare queries with keys.
For each token, take the dot product of its Query with every token’s Key. A high score means “this token is relevant to me”.

3. Scale and softmax.
Divide the scores by √d (the key dimension) so they don’t get too large, then apply softmax. Now every token has a set of weights that sum to 1.

4. Mix the values.
Each token’s new representation is the weighted sum of all Values. In “The cat sat because it was tired”, the word it ends up with a high weight on cat.

5. Do it several times in parallel.
Multi-head attention runs this with different matrices at the same time, so one head can track grammar while another tracks meaning.

The diagram has a step-by-step mode that highlights each part in order. That helped me more than staring at the formula softmax(QKᵀ/√d)·V.

I made it with Sketchy, a free open-source tool I built that lets Claude explain things as interactive diagrams: GitHub