$ ./guess_the_languageAI & ML
AI & ML · hard · #47

What does this mask do in a decoder?

mask = torch.triu(torch.ones(4, 4), diagonal=1).bool()
scores = scores.masked_fill(mask, float('-inf'))
  1. Hides future tokens
  2. Hides padding tokens
  3. Hides earlier tokens
  4. Drops random heads
triu with diagonal=1 marks positions j > i, so each token can only attend to itself and earlier tokens.
Play AI & ML ▸