I am using neural network to classify sequence of length 340 to 8 classes, I am using cross entropy as loss. I am getting very high number for the loss . I am wondering if I did mistake in calculating the loss for each epoch. Or should i use other loss function .
criterion = nn.CrossEntropyLoss()
if CUDA:
criterion = criterion.cuda()
optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9)
loss_list = []
for epoch in range(N_EPOCHES):
tot_loss=0
running_loss =0
model.train()
loss_values = []
acc_list = []
acc_list = torch.FloatTensor(acc_list)
sum_acc = 0
# Training
for i, (seq_batch, stat_batch) in enumerate(training_generator):
# Transfer to GPU
seq_batch, stat_batch = seq_batch.to(device), stat_batch.to(device)
optimizer.zero_grad()
# Model computation
seq_batch = seq_batch.unsqueeze(-1)
outputs = model(seq_batch)
loss = criterion(outputs.argmax(1), stat_batch.argmax(1))
loss.backward()
optimizer.step()
# print statistics
running_loss += loss.item()*seq_batch.size(0)
loss_values.append(running_loss/len(training_set))
if i % 2000 == 1999: # print every 2000 mini-batches
print('[%d, %5d] loss: %.3f' %
(epoch + 1, i + 1, running_loss / 50000),"acc",(outputs.argmax(1) == stat_batch.argmax(1)).float().mean())
running_loss = 0.0
sum_acc += (outputs.argmax(1) == stat_batch.argmax(1)).float().sum()
print("epoch" , epoch, "acc", sum_acc/len(training_generator))
print('Finished Training')
[1, 2000] loss: 14.205 acc tensor(0.5312, device='cuda:0')
[1, 4000] loss: 13.377 acc tensor(0.4922, device='cuda:0')
[1, 6000] loss: 13.159 acc tensor(0.5508, device='cuda:0')
[1, 8000] loss: 13.050 acc tensor(0.5547, device='cuda:0')
[1, 10000] loss: 12.974 acc tensor(0.4883, device='cuda:0')
epoch 1 acc tensor(133.6352, device='cuda:0')
[2, 2000] loss: 12.833 acc tensor(0.5781, device='cuda:0')
[2, 4000] loss: 12.834 acc tensor(0.5391, device='cuda:0')
[2, 6000] loss: 12.782 acc tensor(0.5195, device='cuda:0')
[2, 8000] loss: 12.774 acc tensor(0.5508, device='cuda:0')
[2, 10000] loss: 12.762 acc tensor(0.5156, device='cuda:0')
epoch 2 acc tensor(139.2496, device='cuda:0')
[3, 2000] loss: 12.636 acc tensor(0.5469, device='cuda:0')
[3, 4000] loss: 12.640 acc tensor(0.5469, device='cuda:0')
[3, 6000] loss: 12.648 acc tensor(0.5508, device='cuda:0')
[3, 8000] loss: 12.637 acc tensor(0.5586, device='cuda:0')
[3, 10000] loss: 12.620 acc tensor(0.6016, device='cuda:0')
epoch 3 acc tensor(140.6962, device='cuda:0')
[4, 2000] loss: 12.520 acc tensor(0.5547, device='cuda:0')
[4, 4000] loss: 12.541 acc tensor(0.5664, device='cuda:0')
[4, 6000] loss: 12.538 acc tensor(0.5430, device='cuda:0')
[4, 8000] loss: 12.535 acc tensor(0.5547, device='cuda:0')
[4, 10000] loss: 12.548 acc tensor(0.5820, device='cuda:0')
epoch 4 acc tensor(141.6522, device='cuda:0')
I am getting very high number for the loss
What makes you think this is high? What do you compare this to?
Yes, you should use nn.CrossEntropyLoss for multi-class classification tasks. And your training loss seems perfectly fine to me. At initialization, you should have loss = -log(1/8) = ~2.