What bug does this PyTorch training loop have?
for x, y in loader:
loss = loss_fn(model(x), y)
loss.backward()
optimizer.step()
- Gradients pile up across steps
- Weights are never updated
- Loss is computed on the test data
- The model stays in eval mode
Without optimizer.zero_grad(), .backward() adds to the existing .grad values every iteration.
Play AI & ML ▸