Keras Sequential Model accuracy is bad. Model is Ignoring/neglecting a class

Question

little background: I'm making a simple rock, paper, scissors image classifier program. Basically, I want the image classifier to be able to distinguish between a rock, paper, or scissor image.

problem: The program works amazing for two of the classes, rock and paper, but completely fails whenever given a scissors test image. I've tried increasing my training data and a few other things but no luck. I was wondering if anyone has any ideas on how to offset this.

sidenote: I suspect it also has something to do with overfitting. I say this because the model has about a 92% accuracy with the training data but 55% accuracy on test data.

import numpy as np
import os
import cv2
import random
import tensorflow as tf
from tensorflow import keras

CATEGORIES     = ['rock', 'paper', 'scissors']
IMG_SIZE       = 400  # The size of the images that your neural network will use
CLASS_SIZE     = len(CATEGORIES)
TRAIN_DIR  = "../Train/"

def loadData( directoryPath ):
    data = []
    for category in CATEGORIES:
        path = os.path.join(directoryPath, category)
        class_num = CATEGORIES.index(category)
        for img in os.listdir(path):
            try:
                img_array = cv2.imread(os.path.join(path, img), cv2.IMREAD_GRAYSCALE)
                new_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))
                data.append([new_array, class_num])
            except Exception as e:
                pass
    return data


training_data = loadData(TRAIN_DIR)
random.shuffle(training_data)
X = [] #features
y = [] #labels

for i in range(len(training_data)):
    features = training_data[i][0]
    label    = training_data[i][1]
    X.append(features)
    y.append(label)

X = np.array(X)
y = np.array(y)
X = X/255.0

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(IMG_SIZE, IMG_SIZE)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(CLASS_SIZE)
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])


model.fit(X, y, epochs=25)


TEST_DIR  = "../Test/"
test_data = loadData( TEST_DIR )
random.shuffle(test_data)
test_images = []
test_labels = []

for i in range(len(test_data)):
    features = test_data[i][0]
    label    = test_data[i][1]
    test_images.append(features)
    test_labels.append(label)

test_images = np.array(test_images)
test_images = test_images/255.0
test_labels = np.array(test_labels)

test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

# Saving the model
model_json = model.to_json()
with open("model.json", "w") as json_file :
    json_file.write(model_json)

model.save_weights("model.h5")
print("Saved model to disk")

model.save('CNN.model')

If you want to create a massive amount of training data fast: https://github.com/ThomasStuart/RockPaperScissorsMachineLearning/blob/master/source/0.0-collectMassiveData.py

Thanks in advance to any help or ideas :)

Answer 1

You can simply test overfitting by adding 2 additional layers, one dropout layer and one dense layer. Also be sure to shuffle your train_data after each epoch, so the model keeps the learning general. Also, if I see this correctly, you are doing a multi class classification but do not have a softmax activation in the last layer. I would recommend you, to use it.

With drouput and softmax your model would look like this:

model = keras.Sequential([
keras.layers.Flatten(input_shape=(IMG_SIZE, IMG_SIZE)),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dropout(0.4), #0.4 means 40% of the neurons will be randomly unused
keras.layers.Dense(CLASS_SIZE, activation="softmax")

])

As last advice: Cnns perform in general way better with tasks like this. You might want to switch to a CNN network, for having even better performance.