Keras LSTM model - expected to have 3 dimensions, but got array with 2

Question

Currently working on a financial time series LSTM model with Keras, and ran into this problem.

It seems that my code is generating is generating a node with 2 dimensions where 3 is expected, here is the code,

import pandas as pd
import numpy as np
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Input, Dense, GRU, Embedding, LSTM, Flatten
from tensorflow.python.keras.optimizers import RMSprop
from tensorflow.python.keras.callbacks import EarlyStopping, 
ModelCheckpoint, TensorBoard, ReduceLROnPlateau


batch_size = 500
feature_no = 13
period_no = 8640
def gen(batch_size, periods):
j = 0

features = ['ask_close',
            'ask_open',
            'ask_high',
            'ask_low',
            'bid_close',
            'bid_open',
            'bid_high',
            'bid_low',
            'open',
            'high',
            'low',
            'close',
            'price']
with pd.HDFStore('datasets/eurusd.h5') as store:
    df = store['train_buy']

x_shape = (batch_size, periods, len(features))
x_batch = np.zeros(shape = x_shape, dtype=np.float16)

y_shape = (batch_size, periods)
y_batch = np.zeros(shape = y_shape, dtype=np.float16)

while True:
    i = 0
    while len(x_batch) < batch_size:
        if df.iloc[j+periods]['direction'].values == 1:
            x_batch[i] = df.iloc[j:j+periods][features].values.tolist()
            y_batch[i] = df.iloc[j+periods]['target_buy'][0].round(4)
            i+=1
        j+=1
        if j == 56241737 - periods:
            j = 0
    yield x_batch, y_batch

generator = gen(batch_size, period_no)

model = Sequential()
model.add(LSTM(units = 1, return_sequences=True, input_shape = (None, feature_no,)))


optimizer = RMSprop(lr=1e-3)
model.compile(loss = 'mse', optimizer = optimizer)
model.fit_generator(generator=generator, epochs = 10, steps_per_epoch = 112483)

Here is the error:

Traceback (most recent call last):
model.fit_generator(generator=generator, epochs = 10, steps_per_epoch = 112483)
File "C:\Users\Seok\Anaconda3\lib\site-packages\tensorflow\python\keras\_impl\keras\models.py", line 1198, in fit_generator
initial_epoch=initial_epoch)
File "C:\Users\Seok\Anaconda3\lib\site-packages\tensorflow\python\keras\_impl\keras\engine\training.py", line 2345, in fit_generator
x, y, sample_weight=sample_weight, class_weight=class_weight)
File "C:\Users\Seok\Anaconda3\lib\site-packages\tensorflow\python\keras\_impl\keras\engine\training.py", line 1981, in train_on_batch
check_batch_axis=True)
File "C:\Users\Seok\Anaconda3\lib\site-packages\tensorflow\python\keras\_impl\keras\engine\training.py", line 1514, in _standardize_user_data
exception_prefix='target')
File "C:\Users\Seok\Anaconda3\lib\site-packages\tensorflow\python\keras\_impl\keras\engine\training.py", line 139, in _standardize_input_data
'with shape ' + str(data_shape))
ValueError: Error when checking target: expected lstm_1 to have 3 dimensions, but got array with shape (500, 8640)

I have seen similar issues on gibhub, and they seem to have resolved this problem, however the solution there did not seem to work on this one.

Answer 1

LSTM (and GRU) layers require 3 dimensional inputs: a batch size, a number of time steps, and a number of features. In input_shape terms, they are specified as (batch size, time steps, no. of features). So just eyeballing your code, you should change

model.add(LSTM(units = 1, return_sequences=True, input_shape = (None, feature_no,)))

to

model.add(LSTM(units = 1, return_sequences=True, input_shape = (batch_size, periods, len(features)))

EDIT: Mistake on my part, input_shape is not specified as a 3-dimensional array, but expects a 3-d array as input to the model.

I believe the error here is actually caused by the output shape. With return_sequences = True, the output of an LSTM has the shape (batch_size, timesteps, units), and so the generator should be producing y_batch arrays of shape (batch_size, periods, 1)