Accuracy of binary classification on MNIST “1” and “5” get better?

Question

I tried the binary classification using MNIST only number “1” and “5”. But the accuracy isn’t well.. The following program is anything wrong? If you find something, please give me some advice.

loss: -9.9190e+04

accuracy: 0.5599

import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt

mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train / 255.0
x_test = x_test / 255.0
train_filter = np.where((y_train == 1) | (y_train == 5))
test_filter = np.where((y_test == 1) | (y_test == 5))
x_train, y_train = x_train[train_filter], y_train[train_filter]
x_test, y_test = x_test[test_filter], y_test[test_filter]
print("x_train", x_train.shape)
print("x_test", x_test.shape)

# x_train (12163, 28, 28)
# x_test (2027, 28, 28)

model = keras.Sequential(
    [
        keras.layers.Flatten(input_shape=(28, 28)),
        keras.layers.Dense(128, activation="relu"),
        keras.layers.Dense(1, activation="sigmoid"),
    ]
)

model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(x_train, y_train, epochs=5)

loss, acc = model.evaluate(x_test, y_test, verbose=2)
print("accuracy:", acc)

# 2027/1 - 0s - loss: -9.9190e+04 - accuracy: 0.5599
# accuracy: 0.5599408

Answer 1

Your y_train and y_test is filled with class labels 1 and 5, but sigmoid activation in your last layer is squashing output between 0 and 1.

if you change 5 into 0 in your y you will get a really high accuracy:

y_train = np.where(y_train == 5, 0, y_train)
y_test = np.where(y_test == 5, 0, y_test)

result:

64/64 - 0s - loss: 0.0087 - accuracy: 0.9990