Multiclass text classification using hugging face models

Question

I am trying to do sentiment analysis on customer feedback and for that I am using hugging face models (required). The issue is that all the responses I am getting are either Positive or negative , I haven't gotten a neutral response.

this is how my dataset looks like

import pandas as pd
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import numpy as np


# Example DataFrame
df = pd.DataFrame({'text': ['This movie is great!','neutral','Happy this movie!' ,'I feel bored.', 'The weather is nice.',np.nan]})

    # Function to predict sentiment
    def predict_sentiment(text):
        # Load tokenizer and model
        if pd.isna(text):
            return 'N/A'  # Return a default value for NaN
        tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
        model = AutoModelForSequenceClassification.from_pretrained("textattack/bert-base-uncased-imdb")
        tokens = tokenizer.encode_plus(text, padding=True, truncation=True, return_tensors="pt")
        outputs = model(**tokens)
        predicted_class = outputs.logits.argmax().item()
        sentiment_classes = ['negative','positive', 'neutral']
        predicted_sentiment = sentiment_classes[predicted_class]
        return predicted_sentiment
    
    # Apply sentiment prediction on DataFrame column
    df['predicted_sentiment'] = df['text'].apply(predict_sentiment)

             text                      predicted_sentiment
0   This movie is great!                  positive
1   neutral                               positive
2   Happy this movie!                     positive
3   I feel bored.                         negative
4   The weather is nice.                  positive
5   NaN                                     N/A

Now, if I switch the lables like this ['negative','neutral','positive'] I only get results

             text                      predicted_sentiment
0   This movie is great!                  neutral
1   neutral                               neutral
2   Happy this movie!                     neutral
3   I feel bored.                         negative
4   The weather is nice.                  neutral
5   NaN                                     N/A

whereas the results should be

         text                      predicted_sentiment
0   This movie is great!                  positive
1   neutral                               neutral
2   Happy this movie!                     positive
3   I feel bored.                         negative
4   The weather is nice.                  positive
5   NaN                                     N/A

Answer 1

The main reason why you're not getting the neutral label is because the model that you're using (textattack/bert-base-uncased-imdb, HuggingFace link here) has been trained on the IMDB dataset which is binary classification dataset. In other words, the model only has two final layers and will only output a prediction between two outputs, positive or negative. You can see more info about the dataset here. As you can clearly see it's a binary classification dataset so the model trained on it does not allow for 3 different outputs.

If you pass it three or more labels (as you did with ['negative','neutral','positive']) it will only consider the first two and ignore the others. The model is set such that the first label is "negative" and the second is "positive", irrespective of what you set. So your "neutral" is a "positive" label as it takes the second position. Even if you set the labels to random words such as ['hello','there'] then your "hello" label will signify a negative score (first place) and "there" a positive score (second place). Be careful because if you set your labels to ['positive','negative'] then a "positive" review (first item) will effectively be "negative" one. This is just how the model you're using is set up. Hope this makes sense.