validation loss and accuracy rising

Question

I am using an MLP for classification

Here is my model

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(X.shape[1], X.shape[2])),
    keras.layers.Dense(2048, activation='relu'),
    keras.layers.Dropout(0.1),
    keras.layers.Dense(512, activation='relu'),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

lr_schedule = keras.optimizers.schedules.ExponentialDecay(0.00015, decay_steps=1000, decay_rate=0.96, staircase=True)
optimiser = keras.optimizers.Adam(learning_rate=lr_schedule)

model.compile(optimizer=optimiser, loss='sparse_categorical_crossentropy', metrics=['accuracy'])

I noticed that the traning/validation loss and accuracy(image) validation loss increases as the validation accuracy increases.

Isn't loss supposed to decrease as accuracy increases?

Answer 1

I would recommend checkout out this post. Another aspect that is not mentioned there, but I think is worth noting, is the class-balance in your data set. Since you are using Softmax as your final layer, eg multi-class classification, the reason you observe validation loss increases and validation accuracy increases could be because your data is imbalanced in regards to how many observations of each label(class) exist. Your classifier becomes better (more accurate) at determining your most frequent classes, but worse (less accurate) with the more rare classes. You could say that you are "overfitting" your model to more often predict the common classes.

Because of this, accuracy increases since your classifier is overall correct more often, but the loss increases as well, since the loss for the rare classes become large (since it predicts the probability of them to be 0).

To solve this, you could either upsample or downsample your data, or set class-weights as described here.