Issue with usages of `transform` vs. `fit_transform` in CountVectorizer

Question

I have successfully trained and tested a Logistic Regression model with CountVectorizer() as such:

def train_model(classifier, feature_vector_train, label):
    # fit the training dataset on the classifier
    classifier.fit(feature_vector_train, label)

    return classifier

def getPredictions (classifier, feature_vector_valid):    
    # predict the labels on validation dataset
    predict = classifier.predict(feature_vector_valid)

    return metrics.accuracy_score(predict, valid_y)

def createTrainingAndValidation(column):
    global train_x, valid_x, train_y, valid_y
    train_x, valid_x, train_y, valid_y = model_selection.train_test_split(finalDF[column], finalDF['DeedType1'])

def createCountVectorizer(column):
    global xtrain_count, xvalid_count
    # create a count vectorizer object 
    count_vect = CountVectorizer()
    count_vect.fit(finalDF[column])

    # transform the training and validation data using count vectorizer object
    xtrain_count =  count_vect.transform(train_x)
    xvalid_count =  count_vect.transform(valid_x)

createTrainingAndValidation('Test')
createCountVectorizer('Test')
classifier = train_model(linear_model.LogisticRegression(), xtrain_count, train_y, xvalid_count)
predictions = getPredictions(classifier, xvalid_count)

I was using a DataFrame called finalDF with all labelled text. Since this model is giving me 0.68 accuracy I was going to test it on a subset of the DataFrame with an unknown label. This was not included in the training and testing phase. I saved the trained model as bestClassifier.

Now I got the subset of unknown text and tried to do the following:

count_vect = CountVectorizer()
count_vect.fit(unknownDf['Text'])
text = unknownDf['Text']
xvalid_count =  count_vect.transform(text)

bestClassifier.predict(xvalid_count)

finalDF has 800 rows while unknownDf has only 32 rows after what I do above.. How do I rectify this?

Answer 1

I think I see what's going on, In this peice of code:

def createCountVectorizer(column):
    global xtrain_count, xvalid_count
    # create a count vectorizer object 
    count_vect = CountVectorizer()
    count_vect.fit(finalDF[column])

    # transform the training and validation data using count vectorizer object
    xtrain_count =  count_vect.transform(train_x)
    xvalid_count =  count_vect.transform(valid_x)

You are declaring a CountVectorizer(), calling fit and then transform. What you need to do is, USE THE SAME CountVectorizer() to transform on unknownDf['Text'].

When you do this:

count_vect = CountVectorizer()
count_vect.fit(unknownDf['Text'])
text = unknownDf['Text']
xvalid_count =  count_vect.transform(text)

You are creating a brand new CountVectorizer(), that creates a new bag of words for unknownDf['Text'], when what you should be doing is, removing these two lines

count_vect = CountVectorizer()
count_vect.fit(unknownDf['Text'])

and let the existing CountVectorizer() which you FIT on finalDF[column], use that to transform unknownDf['Text'].

Find a way to USE the CountVectorizer() in your createCountVectorizer(column) which you declared as count_vect to transform the unknownDf['Text'].