Run-time error for (allegedly) empty batches in Pytorch
(Updated with further insights on the problem)
I have a dataset of 3000 images which gets into a DataLoader with the following lines:
training_left_eyes = torch.utils.data.DataLoader(train_dataset, batch_size=2,shuffle=True, drop_last=True)
print(len(training_left_eyes)) #Outputs 1500
My training loop looks like this:
for i,(data,output) in enumerate(training_left_eyes):
data,output = data.to(device),output.to(device)
prediction = net(data)
loss = costFunc(prediction,output)
closs = loss.item()
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("batch #{}".format(i))
if i%100 == 0:
print('[%d %d] loss: %.8f' % (epoch+1,i+1,closs/1000))
closs = 0
The information inside the tensors "data" and "output" (labels) are correct and the system works fine until it reaches the batch number 1500. All my batches are full as 3000/2=1500 with no remainder. Once it reaches that last batch there is a RunTimeError stating that there is a 0-dimensional input size. But I don't know why this is supposed to happen as enumerate(training_left_eyes) should be iterating over the values of the DataLoader which are full.
I searched on the web how to solve this and some people were mentioning the "drop_last=True" attribute on the DataLoader, althought this was done in order to prevent semi-empty batches to enter the model I tried it anyway without no avail.
I am starting to tunnel vision too much and can't seem to solve the problem on my own. I could simply insert an if statement but I think that would be very bad practice and I want to learn the proper solution.
If it helps, here is my custom DataSet:
class LeftEyeDataset(torch.utils.data.Dataset):
"""Left eye retinography dataset. Normal/Not-normal"""
def __init__(self, csv_file, root_dir, transform=None):
"""
Args:
csv_file (string): Path to the csv file with annotations.
root_dir (string): Directory with all the images.
transform (callable, optional): Optional transform to be applied
on a sample.
"""
self.labels = label_mapping(csv_file)
self.root_dir = root_dir
self.transform = transform
self.names = name_mapping(csv_file)
def __len__(self):
return len(self.labels)
def __getitem__(self, idx):
if torch.is_tensor(idx):
idx = idx.tolist()
img_name = self.root_dir +'/'+ self.names[idx]
image = io.imread(img_name)
label = self.labels[idx]
if self.transform:
image = self.transform(image)
return image,label
def label_mapping(csv_file) -> np.array:
df = read_excel(excel_file, 'Sheet1')
x= []
for key,value in df['Left-Diagnostic Keywords'].iteritems():
if value=='normal fundus':
x.append(1)
else:
x.append(0)
x_tensor = torch.LongTensor(x)
return x_tensor
def name_mapping(csv_file) -> list:
#Reads the names of the excel file
df = read_excel(excel_file, 'Sheet1')
names= list()
serie = df['Left-Fundus']
for i in range(serie.size):
names.append(df['Left-Fundus'][i])
return names
I can provide any additional code if needed.
Update: after some time trying to solve the problem I manage to pin point what is happening. For some reason on the last batch, the data entering the network is fine but right before the first layer something happens and it dissapears. On this next image you can see a print I do before entering the forward(self,x) and one right after. The sizes are aligned until the batch number 61 (I reduced it from 1500 for this example) in which it somehow goes through the print twice. After that line, the error mentioned above appears.
After rubber duck debugging for a bit I realized that the problem did not relied in the training but on the validation set. The code read
for i,(data,output) in validate_left_eyes:
data,output = data.to(device),output.to(device)
prediction = net(data)
Validate_left_eyes_ was not wrapped by an enumerate() and thus the first batch of data was empty. The problem was then solved.
I apologize as this part of the code was not featured on my question and thus the answer was not so straight forward.