Word2Vec - Model with high cross validation score performs incredibly bad for test data

Question

While working on sentiment analysis of twitter data, I encountered a problem that I just can't solve. I wanted to train a RandomForest Classifier to detect hate speech. I, therefore, used a labeled dataset with tweets that are labeled as 1 for hate speech and 0 for normal tweets. For vectorization, I am using Word2Vec. I first performed a hyperparametrization to find good parameters for the classifier. During hyperparametrization I used a repeated stratified KFold cross-validation (scoring = accuracy) Mean accuracy is about 99.6% here. However, once I apply the model to a test dataset and plot a confusion matrix, the accuracy is merely above 50%, which is of course awful for a binary classifier. I successfully use the exact same approach with Bag of Words and had no problems at all here. Could someone maybe have a quick look at my code? That would be so helpful. I just cannot find what is wrong. Thank you so much!

(I also uploaded the code to google collab in case that is easier for you: https://colab.research.google.com/drive/15BzElijL3vwa_6DnLicxRvcs4SPDZbpe?usp=sharing )

First I preprocessed my data:

train_csv = pd.read_csv(r'/content/drive/My Drive/Colab Notebooks/MLDA_project/data2/train.csv')
train = train_csv     
#check for missing values (result shows that there are no missing values)
train.isna().sum()    
# remove the tweet IDs
train.drop(train.columns[0], axis = "columns", inplace = True)    
# create a new column to save the cleansed tweets
train['training_tweet'] = np.nan

# remove special/unknown characters
train.replace('[^a-zA-Z#]', ' ', inplace = True, regex = True)    
# generate stopword list and add the twitter handles "user" to the stopword list
stopwords = sw.words('english')
stopwords.append('user')    
# convert to lowercase
train = train.applymap(lambda i:i.lower() if type(i) == str else i)    
# execute tokenization and lemmatization
lemmatizer = WordNetLemmatizer()

for i in range(len(train.index)):
    #tokenize the tweets from the column "tweet"
    words = nltk.word_tokenize(train.iloc[i, 1])
    #consider words with more than 3 characters
    words = [word for word in words if len(word) > 3] 
    #exclude words in stopword list
    words = [lemmatizer.lemmatize(word) for word in words if word not in set(stopwords)] 
    #Join words again
    train.iloc[i, 2]  = ' '.join(words)  
    words = nltk.word_tokenize(train.iloc[i, 2])
train.drop(train.columns[1], axis = "columns", inplace = True)

majority = train[train.label == 0]
minority = train[train.label == 1]
# upsample minority class
minority_upsampled = resample(minority, replace = True, n_samples = len(majority))      
# combine majority class with upsampled minority class
train_upsampled = pd.concat([majority, minority_upsampled])
train = train_upsampled
np.random.seed(10)
train = train.sample(frac = 1)
train = train.reset_index(drop = True)

Now train has the labels in column 0 and the preprocessed tweets in column 1.

Next I defined the Word2Vec Vectorizer:

def W2Vvectorize(X_train):
tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)
vector=[]
from tqdm import tqdm
for sent in tqdm(tokenize):
    sent_vec=np.zeros(100)
    count =0
    for word in sent: 
        if word in w2v_words:
            vec = w2vec_model.wv[word]
            sent_vec += vec 
            count += 1
    if count != 0:
        sent_vec /= count #normalize
    vector.append(sent_vec)
return vector

I split the dataset into test and training set and vectorized both subsets using W2V as defined above:

x = train["training_tweet"]
y = train["label"]

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])

print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) # change 0,4 & 0,6

train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)

Now I carry out the hyperparametrization:

# define models and parameters
model = RandomForestClassifier()
n_estimators = [10, 100, 1000]
max_features = ['sqrt', 'log2']
# define grid search
grid = dict(n_estimators=n_estimators,max_features=max_features)
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
grid_search = GridSearchCV(estimator=model, param_grid=grid, n_jobs=-1, cv=cv, scoring='accuracy',error_score=0)
grid_result = grid_search.fit(train_tf_w2v, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

This results in the following output:

Best: 0.996628 using {'max_features': 'log2', 'n_estimators': 1000}
0.995261 (0.000990) with: {'max_features': 'sqrt', 'n_estimators': 10}
0.996110 (0.000754) with: {'max_features': 'sqrt', 'n_estimators': 100}
0.996081 (0.000853) with: {'max_features': 'sqrt', 'n_estimators': 1000}
0.995885 (0.000872) with: {'max_features': 'log2', 'n_estimators': 10}
0.996481 (0.000691) with: {'max_features': 'log2', 'n_estimators': 100}
0.996628 (0.000782) with: {'max_features': 'log2', 'n_estimators': 1000}

Next, I wanted to draw a confusion matrix with the test data using the Model:

clf = RandomForestClassifier(max_features = 'log2', n_estimators=1000) 
   
clf.fit(train_tf_w2v, y_train)
name = clf.__class__.__name__
        
expectation = y_test
test_prediction = clf.predict(test_tf_w2v)
acc = accuracy_score(expectation, test_prediction)   
pre = precision_score(expectation, test_prediction)
rec = recall_score(expectation, test_prediction)
f1 = f1_score(expectation, test_prediction)

fig, ax = plt.subplots(1,2, figsize=(14,4))
plt.suptitle(f'{name} \n', fontsize = 18)
plt.subplots_adjust(top = 0.8)
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, ax=ax[0])
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, normalize=True, ax = ax[1])
plt.show()
    
print(f"for the {name} we receive the following values:")
print("Accuracy: {:.3%}".format(acc))
print('Precision score: {:.3%}'.format(pre))
print('Recall score: {:.3%}'.format(rec))
print('F1 score: {:.3%}'.format(f1))

This outputs:

for the RandomForestClassifier we receive the following values: Accuracy: 57.974% Precision score: 99.790% Recall score: 15.983% F1 score: 27.552%

Answer 1

Ouuh... Now I feel stupid. I found what was wrong.

After the train/test-split, I sent both subsets independently to the W2Vvectorize() function.

train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)

From there the W2Vvectorize() function trains two independent Word2Vec models, based on the two independent subsets. Hence when I pass the vectorized test data test_tf_w2v to my trained RandomForest classifier, to check if the accuracy is correct for a test set as well, it appears to the trained RandomForest classifier, as if the test set would be in a different language. The two separate word2vec models just vectorize in a different way.

I solved that as follows:

def W2Vvectorize(X_train):
    tokenize=X_train.apply(lambda x: x.split())
    vector=[]
    for sent in tqdm(tokenize):
        sent_vec=np.zeros(100)
        count =0
        for word in sent: 
            if word in w2v_words:
                vec = w2vec_model.wv[word]
                sent_vec += vec 
                count += 1
        if count != 0:
            sent_vec /= count #normalize
        vector.append(sent_vec)
    return vector

And the Word2Vec training is separate from that :

x = train["training_tweet"]
y = train["label"]

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])

print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) #

tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)

train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)

So the Word2Vec models training is performed only on the training data. The vectorization of test data, however, has to be carried out with that exact same Word2Vec model.