sklearn: Plot confusion matrix combined across training+test sets

Question

I have a question regarding the confusion matrix.

By definition of a confusion matrix, it is used to evaluate the quality of the output of a classifier.

So When you split the data to train, test and validation set, each train and test gives you a different confusion matrix.If I want to add them together how should I do it?

Consider my following snipping code:

X, Y = np.array(data[features]), np.array(data['target'])
logo = LeaveOneGroupOut()
group = data['id'].values    
k_fold = KFold(n_splits=5)

scores =[]
per_person_true_y = []
per_person_pred_y = []

classifier_logistic = LogisticRegression()
    for train, test in logo.split(X, Y, group):
        x_train, x_test = X[train], X[test]
        y_train, y_test = Y[train], Y[test]
        classifier_logistic.fit(x_train, y_train.ravel())
        y_predict = classifier_logistic.predict(x_test)
        scores.append(metrics.accuracy_score(y_test,classifier_logistic.predict(x_test)))  
        per_person_true_y.append(y_test)
        per_person_pred_y.append(y_predict)



plot.confusion_matrix( np.array(per_person_true_y),np.array(per_person_pred_y))
plt.show()

which gives me this error :

TypeError: unhashable type: 'numpy.ndarray'

Thanks for comments.

Answer 1

Currently: you have 4 NumPy arrays: y_test, y_train, y_test_pred, and y_train_pred.

You want: 2 NumPy arrays, y_true and y_pred.

You can combine train + test with np.concatenate. For example:

y_test = np.array([0, 1, 0, 1])
y_train = np.array([0, 0, 1, 1])

y_test_pred = np.array([1, 1, 0, 1])  # from classifier_logistic.predict(x_test)
y_train_pred = np.array([0, 1, 0, 1]) # from classifier_logistic.predict(x_train)

y_true = np.concatenate((y_train, y_test))  # you already have this as `Y`
y_pred = np.concatenate((y_train_pred, y_test_pred))

There's a very good example of plotting a confusion matrix in the sklearn docs.

Here's an example with your case in mind:

import itertools
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import confusion_matrix

# Source: http://scikit-learn.org/stable/auto_examples/model_selection/
#         plot_confusion_matrix.html#confusion-matrix


y_test = np.array([1, 1, 0, 1])
y_train = np.array([0, 0, 1, 1])

y_test_pred = np.array([1, 1, 0, 1])  # from classifier_logistic.predict(x_test)
y_train_pred = np.array([0, 1, 0, 1]) # from classifier_logistic.predict(x_train)

y_true = np.concatenate((y_train, y_test))
y_pred = np.concatenate((y_train_pred, y_test_pred))

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

cm = confusion_matrix(y_true, y_pred)
np.set_printoptions(precision=2)

plt.figure()
plot_confusion_matrix(cm, classes=[0, 1],
                      title='Confusion matrix')