Search code examples
pythonkeraslstmmean-square-error

Mean squred error interpretation in LSTM model (bidirectional or multiparallel)

I'm playing with time series and Keras LSTM 1) bidirectional and 2) multiparallel model. I'm saving the best model according to the "mean_squared_error" metrics. My dataset is normalized with MinMaxScaler (default range from 0 to 1). Mean squared error is 0.02 on the test part of the dataset. Does it mean that my model's mean error is 14% - that is 0.02^0.5. Is it a good practical interpretation of the model's precision?

Suppose I want to predict fourth value in this sequence:

[10, 20, 30, 40, 50, 60, 70, 80, 90]

So my x_test and y_test looks like this:

[10 20 30] 40
[20 30 40] 50
[30 40 50] 60
[40 50 60] 70
[50 60 70] 80
[60 70 80] 90

And the code:

cp = [ModelCheckpoint(filepath=path+"/epochBi.h5", monitor='mean_squared_error', verbose=1, save_best_only=True)]
model.compile(optimizer='adam', loss='mse', metrics= ['mean_squared_error'])
history_callback = model.fit(X_train, y_train, epochs=200, verbose=1, callbacks=cp)
model.load_weights(path+"/epochBi.h5")
score = model.evaluate(X_test, y_test, verbose=1)

Suppose that I evaluate on original dataset, how would I interpret the MSE=0.02?

Solution

  • UPDATE The question was updated and hence a new comment

    Again, your MSE of 0.2 DOES NOT MEAN 14% of anything, it is just a scalar and you cannot take it as a percentage of your ground truth.

    Is it a good practical interpretation of the model's precision?

    I would ask this question differently, what does it mean to me?

    Well, looking at your target values that range from 40 to 90, you can say that on average, your error was 0.14. Now if you need more details, it means that when you were predicting 40, you guessed 39.86, when 50 --> 50.14, 60--> 59.86, etc. But this is on average.

    Hope you got the idea by now

    Let's have a look at the formula of mean squared error (MSE):

    enter image description here

    So it is just the average of the squared difference between the predicted and the actual data points. In your case, sqaure root of MSE (RMSE) is 0.02^0.5 or around 0.14 (rounded).

    Does it mean that my model mean error is 14%

    No, you cannot say that your the error is 14% when you get the loss of 0.14. 0.14 is just the value of the error, and your each individual errors could have been negative or positive, making their sum zero but not their squared sum.

    Is it good practical interpretation of the model precision?

    Depends on your goals. You usually measure accuracy depending on some other metrics too like MAE, R sqaured and others. Assuming you did not rescale your data to its original magnitude, then there is no way to know how good or bad this is on absolute terms. So I suggest you measure MSE on your rescaled data and then decide, if not done so already.