The question is quite simple and I have searched the internet wide and thin for a solution. How do I reshape a dataframe to have a custom time step that I feed the LSTM?
look_back=90
train_X = np.reshape(
train_dataset, (train_dataset.shape[0], look_back, train_dataset.shape[1]))
test_X = np.reshape(
test_dataset, (test_dataset.shape[0], look_back, `test_dataset.shape[1]))
The above throws the following error ValueError: cannot reshape array of size 446208 into shape (3984,90,112) So how do I change the look-back to a variable without it being 1 and thereby rendering the memory feature of an LSTM useless.
EDIT Using the solution provided below I change my code to
## train test split
train_split = 0.8
train_size = int(n_sample*train_split)
X_train = X[:train_size] # (train_size, n_features)
X_test = X[train_size:] # (n_sample-train_size, n_features)
print(X_train.shape, X_test.shape)
y_train = y[:train_size] # (train_size,)
y_test = y[train_size:] # (n_sample-train_size,)
print(y_train.shape, y_test.shape)
look_back = 90
y_train = y_train[look_back:] # (train_size-look_back,)
y_test = y_test[look_back:] # ((n_sample-train_size),)
print(y_train.shape, y_test.shape)
X_train = view_as_windows(X_train, (look_back,n_features))[:-1,0] # (train_size-look_back, look_back, n_features)
X_test = view_as_windows(X_test, (look_back,n_features))[:-1,0] # ((n_sample-train_size)-look_back, look_back, n_features)
print(X_train.shape, X_test.shape)
It now works like a charm!
this is a trick I suggest u to create sliding windows...
import numpy as np
from skimage.util.shape import view_as_windows
## create dummy data
n_sample = 2000
n_features = 5
X = np.tile(np.arange(n_sample), (n_features,1)).T
X_train = X[:int(n_sample*0.8)]
X_test = X[int(n_sample*0.8):]
## create windows
look_back = 90
X_train = view_as_windows(X_train, (look_back,n_features), step=1)[:-1,0]
X_test = view_as_windows(X_test, (look_back,n_features), step=1)[:-1,0]
print(X_train.shape, X_test.shape) # (1510, 90, 5) (310, 90, 5)