$ ./guess_the_languageAI & ML
AI & ML · hard · #44

Why are the attention scores divided by sqrt(d_k)?

scores = Q @ K.T / math.sqrt(d_k)
weights = softmax(scores, dim=-1)
  1. Keep softmax from saturating
  2. Make each row sum exactly to one
  3. Turn scores into token ids
  4. Mask out future positions
Dot products grow with dimension; scaling keeps them moderate so softmax gradients don't vanish.
Play AI & ML ▸