Keras LSTM loading data from CSV "expected ndim=3, found ndim=2. Full shape received: (None, 150)"

Question

I am a beginner with LSTMs so sorry if this is a basic question. I've been trying to make a simple LSTM model that loads data from a csv text file for training

    trainX = pd.read_csv("Train\\X_Data.txt", header=None, delim_whitespace=True).to_numpy()
    trainY = pd.read_csv("Train\\Y_Data.txt", header=None, delim_whitespace=True).to_numpy()

    testX = pd.read_csv("Test\\X_Data.txt", header=None, delim_whitespace=True).to_numpy()
    testY = pd.read_csv("Test\\Y_Data.txt", header=None, delim_whitespace=True).to_numpy()

    n_timesteps = trainX.shape[0]
    n_features = trainX.shape[1]

    model = Sequential()
    model.add(LSTM(100, input_shape=trainX.shape, return_sequences=True))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='relu'))
    #may need 2 neurons as there are two classes
    model.add(Dense(1, activation='sigmoid'))

    model.summary()

    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    # fit network
    model.fit(trainX, trainY, epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=1)
    # evaluate model
    evalLosses, evalAccuracy = model.evaluate(testX, testY, batch_size=BATCH_SIZE, verbose=1)

    print("Overall Accuracy: " + str(evalAccuracy))
    print("Overall Loss: " + str(evalLosses))

Where my inputs are:

trainY.shape = (35, 1)
trainX.shape = (35, 150)


trainX = [[0.48597709 0.52190752 0.62556772 ... 0.09958187 0.12535847 0.0833305 ]
 [0.40917949 0.40525872 0.24515716 ... 0.33276069 0.40186229 0.36288622]
 [0.16203835 0.14811591 0.1618184  ... 0.08745848 0.09398027 0.1056776 ]
 ...
 [0.21770377 0.24859037 0.20659391 ... 0.01323494 0.01249982 0.01307911]
 [0.27596078 0.26605097 0.36028712 ... 0.10316001 0.10662966 0.10724351]
 [0.34860233 0.3500129  0.35434798 ... 0.04347154 0.02899346 0.02327774]]

trainY = [[0]
 [0]
 [0]
 [0]
  .
  .
  .
 [0]
 [0]
 [1]
 [1]
 [1]]

When I try to fit the data to my model I get the following error

ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: (None, 150)

How do I get my data to load? the shape is 2 dimensions (35,150), so why does keras only see (None, 150)?

Thanks

Answer 1

trainX.shape = (35, 150) which means that you have 35 samples of 150. But you need to pass the data with the batch_size in the first position according to Keras. So you would have to expand the 2D input to 3D:

trainX = tf.expand_dims(trainX, axis=-1) # new shape = (35, 150, 1)
trainY = tf.expand_dims(trainY, axis=-1) # new shape = (35, 150, 1)

You can then pass the data to the model:

model = Sequential()
model.add(LSTM(100,input_shape=(trainX.shape[1], trainX.shape[2]), return_sequences=True)
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

Edit:

Since you are dealing with a binary classification task change the loss from categorical_crossentropy to binary_crossentropy.