Getting NaN Error While Using LSTM Autoencoder

Question

I am trying to train a model with LSTM Autoencoder using Keras to reconstruct the input that I gave to the model and I am getting NaN error in result which I obtain after decode part. Here is my code;

    # lstm autoencoder recreate sequence
    from numpy import array
    import numpy as np
    from keras.models import Sequential
    from keras.layers import LSTM
    from keras.layers import Dense
    from keras.layers import RepeatVector
    from keras.layers import TimeDistributed
    from keras.utils import plot_model
    import pandas as pd

    df = pd.read_csv('flight_data.csv',sep=',',header=None)
    data = df.to_numpy()
    print(data.shape)


    # define input sequence
    sequence1 = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])
    sequence2 = array([0.2, 0.4, 0.6, 0.4, 1.0, 1.2, 1.4, 1.6, 1.8])
    # reshape input into [samples, timesteps, features]
    n_in = 100
    data = data[73666:,:]
    sequence = data.reshape((1,100,24))
    print(sequence)
    # define model
    model = Sequential()
    model.add(LSTM(100, activation='relu', input_shape=(n_in,24)))
    model.add(RepeatVector(n_in))
    model.add(LSTM(100, activation='relu', return_sequences=True))
    model.add(TimeDistributed(Dense(24)))
    model.compile(optimizer='adam', loss='mse')
    # fit model
    model.fit(sequence, sequence, epochs=300, verbose=0)
    plot_model(model, show_shapes=True, to_file='reconstruct_lstm_autoencoder.png')
    # demonstrate recreation
    yhat = model.predict(sequence, verbose=0)

    print(yhat)

I got the output as;

[[[9.46687355e+14 1.00000000e+01 4.42748822e+08 ... 0.00000000e+00
   0.00000000e+00 0.00000000e+00]
  [9.46687355e+14 1.00000000e+01 4.42748822e+08 ... 0.00000000e+00
   0.00000000e+00 0.00000000e+00]
  [9.46687355e+14 1.00000000e+01 4.42748823e+08 ... 0.00000000e+00
   0.00000000e+00 0.00000000e+00]
  ...
  [9.46687359e+14 1.00000000e+01 4.42748824e+08 ... 0.00000000e+00
   0.00000000e+00 0.00000000e+00]
  [9.46687359e+14 1.00000000e+01 4.42748824e+08 ... 0.00000000e+00
   0.00000000e+00 0.00000000e+00]
  [9.46687359e+14 1.00000000e+01 4.42748825e+08 ... 0.00000000e+00
   0.00000000e+00 0.00000000e+00]]]

[[[nan nan nan ... nan nan nan]
  [nan nan nan ... nan nan nan]
  [nan nan nan ... nan nan nan]
  ...
  [nan nan nan ... nan nan nan]
  [nan nan nan ... nan nan nan]
  [nan nan nan ... nan nan nan]]]

Which part might causing a problem? What should I do?

Answer 1

This looks like you have exploding gradients, which LSTM have a tendency to create. Clipping the gradients can solve this, try setting clipnorm to 1.

ADAM = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False, clipnorm=1.)
model.compile(optimizer=ADAM, loss='mse')