Keras loss function does not decrease on mean squared error

Question

I implemented a neural network with Keras to predict the rating of an item. I consider each rating as a class, so this is my code (outputY is categorical):

inputLayerU = Input(shape=(features,))
inputLayerM = Input(shape=(features,))
dense1 = Dense(features, activation='relu')
denseU = dense1(inputLayerU)         
denseM = dense1(inputLayerM)    
concatLayer = concatenate([denseU, denseM], axis = 1)
denseLayer = Dense(features*2, activation='relu')(concatLayer)
outputLayer = Dense(5, activation='softmax')(denseLayer)

model = Model(inputs=[inputLayerU, inputLayerM], outputs=outputLayer)
model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=0.01), metrics=['accuracy'])

model.fit([inputU, inputM],outputY , epochs=10, steps_per_epoch=10)

When I train this network I get the following result which is fine:

10/10 [==============================] - 2s 187ms/step - loss: 1.4778 - acc: 0.3209
Epoch 2/10
10/10 [==============================] - 0s 49ms/step - loss: 1.4058 - acc: 0.3625
Epoch 3/10
10/10 [==============================] - 1s 54ms/step - loss: 1.3825 - acc: 0.3824
Epoch 4/10
10/10 [==============================] - 0s 47ms/step - loss: 1.3614 - acc: 0.3923
Epoch 5/10
10/10 [==============================] - 0s 48ms/step - loss: 1.3372 - acc: 0.4060
Epoch 6/10
10/10 [==============================] - 0s 45ms/step - loss: 1.3138 - acc: 0.4202
Epoch 7/10
10/10 [==============================] - 0s 46ms/step - loss: 1.2976 - acc: 0.4266
Epoch 8/10
10/10 [==============================] - 0s 48ms/step - loss: 1.2842 - acc: 0.4325
Epoch 9/10
10/10 [==============================] - 1s 62ms/step - loss: 1.2729 - acc: 0.4402
Epoch 10/10
10/10 [==============================] - 1s 54ms/step - loss: 1.2631 - acc: 0.4464

Then I consider the problem as regression and try to predict the value of user ratings(I need to calculate error in both ways). So this is my code:

inputLayerU = Input(shape=(features,))
inputLayerM = Input(shape=(features,))
dense1 = Dense(features, activation='relu')
denseU = dense1(inputLayerU)         
denseM = dense1(inputLayerM)    
concatLayer = concatenate([denseU, denseM], axis = 1)
denseLayer = Dense(features*2, activation='relu')(concatLayer)
outputLayer = Dense(1, activation='softmax')(denseLayer)

model = Model(inputs=[inputLayerU, inputLayerM], outputs=outputLayer)
model.compile(loss='mean_squared_error', optimizer=Adam(lr=0.01), metrics=['accuracy'])

model.fit([inputU, inputM],outputY , epochs=10, steps_per_epoch=10)

and I get this results:

Epoch 1/10
10/10 [==============================] - 9s 894ms/step - loss: 7.9451 - acc: 0.0563
Epoch 2/10
10/10 [==============================] - 7s 711ms/step - loss: 7.9447 - acc: 0.0563
Epoch 3/10
10/10 [==============================] - 7s 709ms/step - loss: 7.9446 - acc: 0.0563
Epoch 4/10
10/10 [==============================] - 7s 710ms/step - loss: 7.9446 - acc: 0.0563
Epoch 5/10
10/10 [==============================] - 7s 702ms/step - loss: 7.9446 - acc: 0.0563
Epoch 6/10
10/10 [==============================] - 7s 706ms/step - loss: 7.9446 - acc: 0.0563
Epoch 7/10
10/10 [==============================] - 7s 701ms/step - loss: 7.9446 - acc: 0.0563
Epoch 8/10
10/10 [==============================] - 7s 702ms/step - loss: 7.9446 - acc: 0.0563
Epoch 9/10
10/10 [==============================] - 7s 717ms/step - loss: 7.9446 - acc: 0.0563
Epoch 10/10
10/10 [==============================] - 7s 700ms/step - loss: 7.9446 - acc: 0.0563

As you see it decreases a little, some times it doesn't change at all.

So what's wrong with my regression?

Answer 1

First, m not sure if we are supposed to apply the softmax function for regression problem, and secondly try using the Adam Optimizer with default parameters.