How to find the ROC curve and AUC score of CNN model (keras)

Question

I am new to deep learning. I am trying to generate ROC curve for the following code. I am using keras. The class size is 10 and the image are RGB image of size 100* 100* 3.

I went through [This link][1]. My problem is also same but I could not find the true labels. I am new in this field so please help me. I also looked on [This for true label][2].

The code snippet of my program is:

target_size=(100,100,3)

train_generator = train_datagen.flow_from_directory('path',
    target_size=target_size[:-1],
    batch_size=16,
    class_mode='categorical',
    subset='training',
    seed=random_seed)

valid_generator = ...

test_generator = ...
n_classes = len(set(train_generator.classes))



input_layer = keras.layers.Input(shape=target_size)

conv2d_1 = keras.layers.Conv2D(filters=64, kernel_size=(3,3), strides=1, padding='same', 
activation='relu',
                       kernel_initializer='he_normal')(input_layer)

batchnorm_1 = keras.layers.BatchNormalization()(conv2d_1)
maxpool1=keras.layers.MaxPool2D(pool_size=(2,2))(batchnorm_1)


conv2d_2 = keras.layers.Conv2D(filters=32, kernel_size=(3,3), strides=1, padding='same', 
activation='relu',
                       kernel_initializer='he_normal')(maxpool1)
batchnorm_2 = keras.layers.BatchNormalization()(conv2d_2)

maxpool2=keras.layers.MaxPool2D(pool_size=(2,2))(batchnorm_2)


flatten = keras.layers.Flatten()(maxpool2)
dense_1 = keras.layers.Dense(256, activation='relu')(flatten)

dense_2 = keras.layers.Dense(n_classes, activation='softmax')(dense_1)



model = keras.models.Model(input_layer, dense_3)

model.compile(optimizer=keras.optimizers.Adam(0.001),
      loss='categorical_crossentropy',
      metrics=['acc'])
model.summary()

model.fit_generator(generator=train_generator, validation_data=valid_generator,
            epochs=200)
            
score = model.evaluate_generator(test_generator)

print(score)

Now please help me in getting AUC score and ROC Curve. [1]: How to find the ROC curve and AUC score of this CNN model (keras) [2]: Getting true labels for keras predictions

Answer 1

Add this code. Hope it works.

import numpy as np
from sklearn import metrics

x, y = test_generator.next()
prediction = model.predict(x)

predict_label1 = np.argmax(prediction, axis=-1)
true_label1 = np.argmax(y, axis=-1)

y = np.array(true_label1)

scores = np.array(predict_label1)
fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=9)
roc_auc = metrics.auc(fpr, tpr)


plt.figure()
lw = 2
plt.plot(fpr, tpr, color='darkorange',
 lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic (ROC)')
plt.legend(loc="lower right")
plt.show()