I'm facing the following error, and I don't know why: This code is on GitHub, I ran it correctly on Collab, but it gives me the following error here:
device="cpu"
lr=3e-5#1e-3
num_training_steps=int(len(dataset) / TRAIN_BATCH_SIZE * EPOCH)
model=Bert_Classification_Model().to(device)
optimizer=AdamW(model.parameters(), lr=lr)
scheduler = get_linear_schedule_with_warmup(optimizer,
num_warmup_steps = 0,
num_training_steps = num_training_steps)
val_losses=[]
batches_losses=[]
val_acc=[]
for epoch in range(EPOCH):
t0 = time.time()
print(f"\n=============== EPOCH {epoch+1} / {EPOCH} ===============\n")
batches_losses_tmp=train_loop_fun1(train_data_loader, model, optimizer, device)
epoch_loss=np.mean(batches_losses_tmp)
print(f"\n*** avg_loss : {epoch_loss:.2f}, time : ~{(time.time()-t0)//60} min ({time.time()-t0:.2f} sec) ***\n")
t1=time.time()
output, target, val_losses_tmp=eval_loop_fun1(valid_data_loader, model, device)
print(f"==> evaluation : avg_loss = {np.mean(val_losses_tmp):.2f}, time : {time.time()-t1:.2f} sec\n")
tmp_evaluate=evaluate(target.reshape(-1), output)
print(f"=====>\t{tmp_evaluate}")
val_acc.append(tmp_evaluate['accuracy'])
val_losses.append(val_losses_tmp)
batches_losses.append(batches_losses_tmp)
print("\t§§ model has been saved §§")
torch.save(model, f"model1/model_epoch{epoch+1}.pt")
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
=============== EPOCH 1 / 3 ===============
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-33-aa98faac385e> in <module>()
14 t0 = time.time()
15 print(f"\n=============== EPOCH {epoch+1} / {EPOCH} ===============\n")
---> 16 batches_losses_tmp=train_loop_fun1(train_data_loader, model, optimizer, device)
17 epoch_loss=np.mean(batches_losses_tmp)
18 print(f"\n*** avg_loss : {epoch_loss:.2f}, time : ~{(time.time()-t0)//60} min ({time.time()-t0:.2f} sec) ***\n")
6 frames
/content/RoBERT_Recurrence_over_BERT/Custom_Dataset_Class.py in long_terms_tokenizer(self, data_tokenize, targets)
158 targets_list.append(targets)
159
--> 160 if remain and self.approach != 'head':
161 remain = torch.tensor(remain, dtype=torch.long)
162 idxs = range(len(remain)+self.chunk_len)
RuntimeError: Boolean value of Tensor with more than one value is ambiguous
This is the file link: https://github.com/helmy-elrais/RoBERT_Recurrence_over_BERT/blob/master/train.ipynb
Your tensor remain (in your Dataset class) is a boolean tensor and not a boolean variable. Therefore, the condition if remain is not well-defined.