Why are the attention scores divided by sqrt(d_k)?
scores = Q @ K.T / math.sqrt(d_k) weights = softmax(scores, dim=-1)
- Keep softmax from saturating
- Make each row sum exactly to one
- Turn scores into token ids
- Mask out future positions
Dot products grow with dimension; scaling keeps them moderate so softmax gradients don't vanish.
Play AI & ML ▸