Bidirectional LSTM gives Loss as NaN

Question

I am using twitter's emotion dataset to classify the emotions. In order to achieve that i have written below code but when i train it i get loss NaN. I couldnt able to understand what is the issue. Though i managed to find the solution of the issue but why the issue occurred at first place i am not getting that.

Code :

import pandas as pd
import numpy as np
import re

cols = ["id","text","emotion","intensity"]

anger_df_train = pd.read_csv("D:/Dataset/twitter_emotion/train/anger.csv",delimiter='\t',names=cols)
fear_df_train = pd.read_csv("D:/Dataset/twitter_emotion/train/fear.csv",delimiter='\t',names=cols)
joy_df_train = pd.read_csv("D:/Dataset/twitter_emotion/train/joy.csv",delimiter='\t',names=cols)
sadness_df_train = pd.read_csv("D:/Dataset/twitter_emotion/train/sadness.csv",delimiter='\t',names=cols)

df_train = pd.concat([anger_df_train,fear_df_train,joy_df_train,sadness_df_train])

import spacy

nlp = spacy.load('en_core_web_md')

doc = nlp("The big grey dog ate all of the chocolate, but fortunately he wasn't sick!")

def spacy_tokenizer(sentence):
    emails = '[A-Za-z0-9]+@[a-zA-Z].[a-zA-Z]+'
    websites = '(http[s]*:[/][/])[a-zA-Z0-9]'
    mentions = '@[A-Za-z0-9]+'
    sentence = re.sub(emails,'',sentence)
    sentence = re.sub(websites,'',sentence)
    sentence = re.sub(mentions,'',sentence)
    sentence_list=[word.lemma_ for word in nlp(sentence) if not (word.is_stop or word.is_space or word.like_num or len(word)==1)]
    return ' '.join(sentence_list)

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

df_train['new_text']=df_train['text'].apply(spacy_tokenizer)

tokenizer = Tokenizer(num_words=10000)

tokenizer.fit_on_texts(df_train['new_text'].values)

sequences = tokenizer.texts_to_sequences(df_train['new_text'].values)

text_embedding = np.zeros((len(word_index)+1,300))

for word,i in word_index.items():
    text_embedding[i]=nlp(word).vector

labels = df_train['emotion'].unique()

label_tokenizer = Tokenizer()
label_tokenizer.fit_on_texts(labels)

train_label = np.array(label_tokenizer.texts_to_sequences(df_train['emotion'].values))

train_padd = pad_sequences(sequences,maxlen=maxlen,padding='post',truncating='post')

embedding_dim=300

from tensorflow.keras import layers
from tensorflow.keras import models
from tensorflow.keras.layers import Embedding, Flatten, Dense
from tensorflow.keras.layers import Dense, LSTM, Embedding,Dropout,SpatialDropout1D,Conv1D,MaxPooling1D,GRU,BatchNormalization
from tensorflow.keras.layers import Input,Bidirectional,GlobalAveragePooling1D,GlobalMaxPooling1D,concatenate,LeakyReLU

model=models.Sequential()

model.add(Embedding(input_dim=text_embedding.shape[0],output_dim=text_embedding.shape[1],weights=[text_embedding],input_length=maxlen,trainable=False))

model.add(Bidirectional(LSTM(embedding_dim)))
model.add(Dense(embedding_dim,activation='tanh'))
model.add(Dense(4,activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()

X_train,X_test,y_train,y_test=train_test_split(train_padd,train_label,test_size=0.2)

num_epochs = 10
history = model.fit(X_train, y_train,epochs=num_epochs,verbose=1)

Training:

Epoch 1/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0035
Epoch 2/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 3/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 4/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 5/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 6/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 7/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 8/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 9/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 10/10
91/91 [==============================] - 1s 15ms/step - loss: nan - accuracy: 0.0000e+00

Work around that i did to make this run is instead of using tokenizer for label i used below command.

label_map={'anger': 0, 'fear': 1, 'joy': 2, 'sadness': 3}
df_train['emotion'] = df_train['emotion'].map(label_map)
from tensorflow.keras.utils import to_categorical
categorical_labels = to_categorical(df_train['emotion'].values,num_classes=4)

X_train,X_test,y_train,y_test=train_test_split(train_padd,categorical_labels,test_size=0.2,shuffle=True)

Also while compiling the model i changed sparse_categorical_crossentropy to categorical_crossentropy. After that it worked. Please guide me what was wrong with my approach.

Answer 1

The reason sparse_categorical_crossentropy is resulting in NaN is when we Tokenize the Labels using Tokenizer, the Train Labels Array generates, is shown below:

array([[1],
       [2],
       [3],
       [4],
       [1],
       [2],
       [3]])

However, if the sparse_categorical_crossentropy Loss has to be applied, Train Labels Array should be as shown below:

array([0, 1, 2, 3, 0, 1, 2])

So, we can make your code work with sparse_categorical_crossentropy Loss using the below code:

label_map={'anger': 0, 'fear': 1, 'joy': 2, 'sadness': 3}
df['Labels'] = df['Labels'].map(label_map)

sparse_categorical_labels = df['Labels'].values

X_train,X_test,y_train,y_test =  train_test_split(train_padd,sparse_categorical_labels,test_size=0.2,shuffle=True)