Get accuracy of guess

Question

I'm currently trying to find the pronounceability of a list of words using this SO question

The following code is as follows:

import random
def scramble(s):
    return "".join(random.sample(s, len(s)))

words = [w.strip() for w in open('/usr/share/dict/words') if w == w.lower()]
scrambled = [scramble(w) for w in words]

X = words+scrambled
y = ['word']*len(words) + ['unpronounceable']*len(scrambled)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

text_clf = Pipeline([
    ('vect', CountVectorizer(analyzer='char', ngram_range=(1, 3))),
    ('clf', MultinomialNB())
    ])

text_clf = text_clf.fit(X_train, y_train)
predicted = text_clf.predict(X_test)

from sklearn import metrics
print(metrics.classification_report(y_test, predicted))

This outputs with random words this

>>> text_clf.predict("scaroly".split())
['word']

I've been checking the scikit documentation but I still can't seem to find out how I would have it print the score of the input word.

Answer 1

Try sklearn.pipeline.Pipeline.predict_proba:

>>> text_clf.predict_proba(["scaroly"])
array([[  5.87363027e-04,   9.99412637e-01]])

It returns the likelihood a given input (in this case, "scaroly") belongs to the classes upon which you trained the model. So there's a 99.94% chance "scaroly" is pronounceable.

Conversely, the Welsh word for "new" is likely unpronounceable:

>>> text_clf.predict_proba(["newydd"])
array([[ 0.99666533,  0.00333467]])