I am learning about random forests in scikit learn and as an example I would like to use Random forest classifier for text classification, with my own dataset. So first I vectorized the text with tfidf and for classification:
from sklearn.ensemble import RandomForestClassifier
classifier=RandomForestClassifier(n_estimators=10)
classifier.fit(X_train, y_train)
prediction = classifier.predict(X_test)
When I run the classification I got this:
TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.
then I used the .toarray() for X_train and I got the following:
TypeError: sparse matrix length is ambiguous; use getnnz() or shape[0]
From a previous question as I understood I need to reduce the dimensionality of the numpy array so I do the same:
from sklearn.decomposition.truncated_svd import TruncatedSVD
pca = TruncatedSVD(n_components=300)
X_reduced_train = pca.fit_transform(X_train)
from sklearn.ensemble import RandomForestClassifier
classifier=RandomForestClassifier(n_estimators=10)
classifier.fit(X_reduced_train, y_train)
prediction = classifier.predict(X_testing)
Then I got this exception:
File "/usr/local/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 419, in predict
n_samples = len(X)
File "/usr/local/lib/python2.7/site-packages/scipy/sparse/base.py", line 192, in __len__
raise TypeError("sparse matrix length is ambiguous; use getnnz()"
TypeError: sparse matrix length is ambiguous; use getnnz() or shape[0]
The I tried the following:
prediction = classifier.predict(X_train.getnnz())
And got this:
File "/usr/local/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 419, in predict
n_samples = len(X)
TypeError: object of type 'int' has no len()
Two questions were raised from this: How can I use Random forests to classify correctly? and what's happening with X_train?.
Then I tried the following:
df = pd.read_csv('/path/file.csv',
header=0, sep=',', names=['id', 'text', 'label'])
X = tfidf_vect.fit_transform(df['text'].values)
y = df['label'].values
from sklearn.decomposition.truncated_svd import TruncatedSVD
pca = TruncatedSVD(n_components=2)
X = pca.fit_transform(X)
a_train, a_test, b_train, b_test = train_test_split(X, y, test_size=0.33, random_state=42)
from sklearn.ensemble import RandomForestClassifier
classifier=RandomForestClassifier(n_estimators=10)
classifier.fit(a_train, b_train)
prediction = classifier.predict(a_test)
from sklearn.metrics.metrics import precision_score, recall_score, confusion_matrix, classification_report
print '\nscore:', classifier.score(a_train, b_test)
print '\nprecision:', precision_score(b_test, prediction)
print '\nrecall:', recall_score(b_test, prediction)
print '\n confussion matrix:\n',confusion_matrix(b_test, prediction)
print '\n clasification report:\n', classification_report(b_test, prediction)
It is a bit unclear if you are passing the same data structure (type and shape) to the fit method and predict method of the classifier. Random forests will take a long time to run with a large number of features, hence the suggestion to reduce the dimensionality in the post you link to.
You should apply the SVD to both the training and test data so the classifier in trained on the same shaped input as the data you wish to predict for. Check the input to the fit, and the input to the predict method have the same number of features, and are both arrays rather than sparse matrices.
updated with example: updated to use dataframe
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vect= TfidfVectorizer( use_idf=True, smooth_idf=True, sublinear_tf=False)
from sklearn.cross_validation import train_test_split
df= pd.DataFrame({'text':['cat on the','angel eyes has','blue red angel','one two blue','blue whales eat','hot tin roof','angel eyes has','have a cat']\
,'class': [0,0,0,1,1,1,0,3]})
X = tfidf_vect.fit_transform(df['text'].values)
y = df['class'].values
from sklearn.decomposition.truncated_svd import TruncatedSVD
pca = TruncatedSVD(n_components=2)
X_reduced_train = pca.fit_transform(X)
a_train, a_test, b_train, b_test = train_test_split(X, y, test_size=0.33, random_state=42)
from sklearn.ensemble import RandomForestClassifier
classifier=RandomForestClassifier(n_estimators=10)
classifier.fit(a_train.toarray(), b_train)
prediction = classifier.predict(a_test.toarray())
Note the SVD happens before the split into training and test sets, so that the array passed to the predictor has the same n as the array the fit method is called on.