$ ./guess_the_languageAI & ML
AI & ML · hard · #33

What bug does this PyTorch training loop have?

for x, y in loader:
    loss = loss_fn(model(x), y)
    loss.backward()
    optimizer.step()
  1. Gradients pile up across steps
  2. Weights are never updated
  3. Loss is computed on the test data
  4. The model stays in eval mode
Without optimizer.zero_grad(), .backward() adds to the existing .grad values every iteration.
Play AI & ML ▸