PyTorch: Simple feedforward neural network not running without retain_graph=True

Question

Below is my code for training a Feedforward neural network (FFNN).

The labels are numbers between 0 and 50. The FFNN comprises of a single hidden layer with 50 neurons and an output layer with 51 neurons. Furthermore, I have used negative log likelihood loss.

I am very new to PyTorch so I used a couple of websites for guidance. The strange thing is that none of them required retain_graph to be set to True (they dont pass any arguments when calling backward()). Furthermore, it runs very slowly and the accuracy seems to be fluctuating around a fixed value instead of reducing.

Assuming that the input's format is correct, can someone please explain to me why the network is performing so badly and why the network requires retain_graph to be set to True?

Thank you very much!

n_epochs = 2
batch_size = 100
for epoch in range(n_epochs):
    permutation = torch.randperm(training_set.size()[0])
    for i in range(0, training_set.size()[0], batch_size):
        opt.zero_grad()
        indices = permutation[i:i + batch_size]
        batch_features = training_set[indices]
        batch_labels = torch.LongTensor([label for label, sent in train[indices]])
        batch_outputs = model(batch_features)
        loss = loss_function(batch_outputs, batch_labels)
        loss.backward(retain_graph=True)
        opt.step()

Answer 1

You are missing .zero_grad() operation. Add that to the loop and your code will work fine without retain_graph= True.

loss.backward()
opt.step()
opt.zero_grad()