Problem with dimensionality in Keras RNN - reshape isn't working?

Question

Let's consider this random dataset on which I want to perform RNN:

import random
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from keras.optimizers import SGD
import numpy as np
df_train = random.sample(range(1, 100), 50)

I want to apply RNN with lag equal to 1. I'll use my own function:

def create_dataset(dataset, lags):
    dataX, dataY = [], []
    for i in range(lags):
        subdata = dataset[i:len(dataset) - lags + i]
        dataX.append(subdata)
    dataY.append(dataset[lags:len(dataset)])
    return np.array(dataX), np.array(dataY)

which narrows dataframe with respect to number of lags. It outputs two numpy arrays - first is independent variables, and second one is dependent variable.

x_train, y_train = create_dataset(df_train, lags = 1)

But now when I'm trying to run the function:

model = Sequential()
model.add(SimpleRNN(1, input_shape=(1, 1)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer=SGD(lr = 0.1))
history = model.fit(x_train, y_train, epochs=1000, batch_size=50, validation_split=0.2)

I obtain error:

ValueError: Error when checking input: expected simple_rnn_18_input to have 3 dimensions, but got array with shape (1, 49)

I've read about it and the solution is just to apply reshape:

x_train = np.reshape(x_train, (x_train.shape[0], 1, x_train.shape[1]))

but when I apply it I obtain error:

ValueError: Error when checking input: expected simple_rnn_19_input to have shape (1, 1) but got array with shape (1, 49)

and I'm not sure where is the mistake. Could you please tell me what I'm doing wrong?

Answer 1

What you are calling lags is called look back in literature. This technique allow to feed the RNN with more contextual data and learn mid/long range dependencies.

The error is telling you that you are feeding the layer (shape: 1x1) with the dataset (shape: 1x49)

There are 2 reasons behind the error:

1. The first is due to your create_dataset which is building a stack of 1x(50 - lags) = 1x49 vectors, which is the opposite of what you want 1x(lags) = 1x1.

   In particular this line is the responsible:

subdata = dataset[i:len(dataset) - lags + i]

# with lags = 1 you have just one
# iteration in range(1): i = 0
subdata = dataset[0:50 - 1 + 0]
subdata = dataset[0:49] # which is a 1x49 vector

# In order to create the right vector
# you need to change your function:

def create_dataset(dataset, lags = 1):
    dataX, dataY = [], []
    # iterate to a max of (50 - lags - 1) times
    # because we need "lags" element in each vector
    for i in range(len(dataset) - lags - 1):
        # get "lags" elements from the dataset
        subdata = dataset[i:i + lags]
        dataX.append(subdata)
        # get only the last label representing
        # the current element iteration
        dataY.append(dataset[i + lags])
    return np.array(dataX), np.array(dataY)

2. If you use look back in your RNN you also need to increase the input dimensions, because you are looking also at precendent samples.

   The network indeed is looking to more data than just 1 sample, because it needs to "look back" to more samples to understand mid/long range dependencies.

   This is more conceptual than actual, in your code is fine because lags = 1:

model.add(SimpleRNN(1, input_shape=(1, 1)))

# you should use lags in the input shape
model.add(SimpleRNN(1, input_shape=(1, LAGS)))