What does this mask do in a decoder?
mask = torch.triu(torch.ones(4, 4), diagonal=1).bool()
scores = scores.masked_fill(mask, float('-inf'))
- Hides future tokens
- Hides padding tokens
- Hides earlier tokens
- Drops random heads
triu with diagonal=1 marks positions j > i, so each token can only attend to itself and earlier tokens.
Play AI & ML ▸