I have a dataframe with 220 rows and 1410 columns. The rows represents a datetime in days and the columns represents 470 differents features in three time lags (470 * 3 = 1410) as we can see below:
CEE_SECO_01_lag1 CEE_SECO_02_lag1 ... BM_lag3 PME_lag3
2017-07-31 30553.75 28373.42 ... 266325.00 217874.92
2017-08-01 30656.70 28715.36 ... 266325.00 217874.92
2017-08-02 30600.47 28341.33 ... 266325.00 217874.92
2017-08-03 28468.36 26797.55 ... 266325.00 217874.92
2017-08-04 29081.35 27373.21 ... 266325.00 217874.92
... ... ... ... ...
2018-03-03 33249.36 31572.65 ... 262770.31 218720.93
2018-03-04 36189.80 34308.52 ... 262770.31 218720.93
2018-03-05 36082.87 33824.52 ... 262770.31 218720.93
2018-03-06 35227.69 32910.53 ... 262770.31 218720.93
2018-03-07 35891.20 33809.99 ... 262770.31 218720.93
[220 rows x 1410 columns]
I am trying to run LSTM model for a forecast problem.
First i reshaped my data:
import pandas as pd
import numpy as np
n_samples = len(X_train)
n_steps = 3
n_features = int(len(X_train.columns)/n_steps)
X_train_lstm = X_train.to_numpy().reshape((n_samples,n_steps,n_features))
X_teste_lstm = X_test.to_numpy().reshape((1,n_steps,n_features))
y_treino_lstm = y_train.to_numpy()
y_teste_lstm = y_test.to_numpy()
Then, I create a Vanilla LSTM model (like this https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/):
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(X_train_lstm, y_train_lstm, epochs=1000)
And works perfectly. But when I am trying to make a regularization in this model (like this: https://machinelearningmastery.com/use-weight-regularization-lstm-networks-time-series-forecasting/), I am getting stucked.
If I try this:
from keras.regularizers import L1L2
model = Sequential()
model.add(LSTM(50, activation='relu', kernel_regularizer = L1L2(l1=0.01, l2=0.0) , input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(X_train_lstm, y_train_lstm, epochs=1000)
This error occurs:
ValueError: If a RNN is stateful, it needs to know its batch size. Specify the batch size of your input tensors:
- If using a Sequential model, specify the batch size by passing a `batch_input_shape` argument to your first layer.
- If using the functional API, specify the batch size by passing a `batch_shape` argument to your Input layer.
And if I try to change input_shape to batch_input_shape like this:
n_batchs = 1
model = Sequential()
model.add(LSTM(50, activation='relu', kernel_regularizer = L1L2(l1=0.01, l2=0.0) , batch_input_shape=(n_batchs, n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(X_train_lstm, y_train_lstm, epochs=1000)
I get this error:
InvalidArgumentError: Specified a list with shape [1,470] from a tensor with shape [32,470]
[[node sequential_3795/lstm_3791/TensorArrayUnstack/TensorListFromTensor (defined at C:\Users\Pichau\anaconda3\lib\site-packages\keras\backend.py:4330) ]] [Op:__inference_train_function_48158087]
Errors may have originated from an input operation.
Input Source operations connected to node sequential_3795/lstm_3791/TensorArrayUnstack/TensorListFromTensor:
sequential_3795/lstm_3791/transpose (defined at C:\Users\Pichau\anaconda3\lib\site-packages\keras\backend.py:4199)
Function call stack:
train_function
I've already tried lot of different values of n_batchs, like 32, 28, the number of features, observations, columns or time steps and some random values.
How i can make a regularization with LSTM model?
I have put together a example for you
import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.datasets import make_regression
X, y = make_regression()
print(X.shape, y.shape)
lstm_train = []
for i in range(0,100,10):
for j in range(0,100,10):
lstm_train.append(X[j:j+10])
lstm_train = np.stack(lstm_train, axis=0)
print(lstm_train.shape)
from tensorflow.keras.regularizers import L1L2
model = tf.keras.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(10, 100)))
model.add(tf.keras.layers.LSTM(50, activation='relu', kernel_regularizer = L1L2(l1=0.01, l2=0.0)))
model.add(tf.keras.layers.Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(lstm_train, y, epochs=10)
The crux is in this line. We can add an exclusive InputLayer in the model and specify the dimensions there instead of doing it in the LSTM layer.
model.add(tf.keras.layers.InputLayer(input_shape=(10, 100)))