Search code examples
pythonscikit-learnnlptext-classificationcountvectorizer

I get isnan error when I merge two countvectorizers

I'm going dialect text classification and I have this code:

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

vectorizerN = CountVectorizer(analyzer='char',ngram_range=(3,4))
XN = vectorizerN.fit_transform(X_train)

vectorizerMX = CountVectorizer(vocabulary=a['vocabs'])
MX = vectorizerMX.fit_transform(X_train)

from sklearn.pipeline import FeatureUnion
combined_features = FeatureUnion([('CountVectorizer', MX),('CountVect', XN)])
combined_features.transform(test_data)

When I run this code I get this error:

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

I was following the code in this post: Merging CountVectorizer in Scikit-Learn feature extraction

Also, how can I train and predict afterwards?

Solution

  • You should union vectorizerN and vectorizerMX, not MX and XN. Change the line to

    combined_features = FeatureUnion([('CountVectorizer', vectorizerMX), ('CountVect', vectorizerN)])