I am trying to implement the LinearSVC
model on a data-set containing 25000 movie reviews. 12,500 are positive labelled reviews and the rest are negative. I am trying to vectorize the data using TfidfVectorizer
.
This is my code:
import pandas as pd
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
k=0
# reading training data from data-set and taking test data as input
rev=input('Enter:')
rev=rev.replace("<br />", " ")
data_folder= "C:/Users/Files/Desktop/Dataset/train"
for sentiment in (["/neg/","/pos/"]):
#path would be the directory path
path=data_folder+sentiment
#filename will store the NAME of the files that you want to access
for filename in sorted(os.listdir(path)):
#open file
with open(path+ filename,"r",encoding= 'utf-8') as f:
review=f.read()
review=review.replace("<br />", " ")
total.append(review)
# removing stop-words from data
for i in total:
for j in stop_words:
if j in i:
i.replace(j,'')
for i in [rev]:
for j in stop_words:
if j in i:
i.replace(j,'')
c=TfidfVectorizer()
f=c.fit_transform(total).toarray()
tst=c.fit_transform([rev]).toarray()
# 0 for negative data and 1 for positive data
while k!=12500:
l.append(0)
k+=1
while k!=25000:
l.append(1)
k+=1
m=LinearSVC(random_state=0,tol=1e-5)
m.fit(f,l)
if(m.predict(tst).tolist().count(1)>m.predict(tst).tolist().count(0)):
print('Positive')
else:
print('Negative')
Every time I run this code, I keep getting this error
ValueError: X has 139 features per sample; expecting 79897
What does this error mean and how do I fix this?
Ok I found the solution, just had to use transform()
in place of fit_transform()
to vectorize the train data i.e.,
c=TfidfVectorizer()
f=c.fit_transform(total).toarray()
tst=c.transform([rev]).toarray()