I'm working on a sarcasm detector with the BERT model (binary classification). Currently, I'm having trouble with the model evaluation as I don't really understand the predictions array. The model should output 1 for sarcastic and 0 for not, but the predictions don't output that. Please let me know if more code is needed. Thank you!
model:
from transformers import BertForSequenceClassification, AdamW, BertConfig
# Load BertForSequenceClassification, the pretrained BERT model with a single
# linear classification layer on top.
model = BertForSequenceClassification.from_pretrained(
"bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
num_labels = 2, # The number of output labels--2 for binary classification.
# You can increase this for multi-class tasks.
output_attentions = False, # Whether the model returns attentions weights.
output_hidden_states = False, # Whether the model returns all hidden-states.
attention_probs_dropout_prob=0.25,
hidden_dropout_prob=0.25
)
# Tell pytorch to run this model on the GPU.
model.cuda()
evaluation:
from sklearn.metrics import confusion_matrix
import seaborn as sn
import pandas as pd
print('Predicting labels for {:,} test sentences...'.format(len(eval_input_ids)))
# Put model in evaluation mode
model.eval()
predictions , true_labels = [], []
# iterate over test data
for batch in eval_dataloader:
batch = tuple(t.to(device) for t in batch)
# Unpack the inputs from our dataloader
b_input_ids, b_input_mask, b_labels = batch
# Telling the model not to compute or store gradients, saving memory and
# speeding up prediction
with torch.no_grad():
# Forward pass, calculate logit predictions.
result = model(b_input_ids,
token_type_ids=None,
attention_mask=b_input_mask,
return_dict=True)
logits = result.logits
# Move logits and labels to CPU
logits = logits.detach().cpu().numpy()
label_ids = b_labels.to('cpu').numpy()
# Store predictions and true labels
predictions.append(logits)
true_labels.append(label_ids)
true_labels[1]
predictions[1]
output:
array([0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1,
0, 1, 1, 0, 0, 0, 0, 1, 1, 1]) <-- true_labels[1]
array([[ 2.9316974 , -2.855342 ],
[ 3.4540875 , -3.3177233 ],
[ 2.7424026 , -2.6472614 ],
[-3.4326897 , 3.330751 ],
[ 3.7238903 , -3.7757814 ],
[-3.208891 , 3.175109 ],
[ 3.0500402 , -2.8103237 ],
[ 3.8333693 , -3.9073608 ],
[-3.2779126 , 3.231213 ],
[ 1.484127 , -1.2610332 ],
[ 3.686339 , -3.7582958 ],
[-2.1883147 , 2.205132 ],
[-3.274582 , 3.2254982 ],
[-1.606854 , 1.6213335 ],
[ 3.7080388 , -3.6854186 ],
[-2.351147 , 2.365543 ],
[-3.7317555 , 3.4833894 ],
[ 3.2413306 , -3.2116275 ],
[ 3.7413723 , -3.7767386 ],
[-3.6293464 , 3.4446163 ],
[ 3.7779078 , -3.9025154 ],
[-3.5576923 , 3.403335 ],
[ 3.6226897 , -3.6370063 ],
[-3.7081888 , 3.4720154 ],
[ 1.1533121 , -0.8105195 ],
[ 1.0573612 , -0.69238156],
[ 3.4189024 , -3.4764926 ],
[-0.13847755, 0.450572 ],
[ 3.7248163 , -3.7781181 ],
[-3.2015219 , 3.1719215 ],
[-2.1409311 , 2.1202204 ],
[-3.470693 , 3.358798 ]], dtype=float32) <-- predictions[1]
There are two values because you have two classes (0=no, 1=yes). These values are logits, which when fed into a softmax function gives the probability of each class. If you want to know whether the sample is classified as sarcasm or not, just take the class with the highest logit:
predictions = a.max(1)[1]
print(predictions)