Search code examples
pythontensorflowdeep-learningshapesforecast

Imcompatible shapes: [1020,1,1] vs. [1019,1,1] - Tensorflow

Get a problem when setting the output of my neural network to 1, it gave a problem in the data array, accusing which shape is bigger, if I use periods = len(valuesAnalisys) - 1, everything works!

Periods:

periods = 1

Returned:

Imcompatible shapes: [1020,1,1] vs. [1019,1,1]

Neural network:

datecollect = [x[0] for x in dataSet]
servers = [x[1] for x in dataSet]
valuesAnalisys = [float(x[2]) for x in dataSet]

base = np.array(valuesAnalisys)

periods = 1
future_forecast = 1

X = base[0:(len(base) - (len(base) % periods))]
X_batches = X.reshape(-1, periods, 1)

y = base[1:(len(base) - (len(base) % periods)) + future_forecast]
y_batches = y.reshape(-1, periods, 1)

X_test = base[-(periods + future_forecast):]
X_test = X_test[:periods]
X_test = X_test.reshape(-1, periods, 1)
y_test = base[-(periods):]
y_test = y_test.reshape(-1, periods, 1)

tf.reset_default_graph()

appetizer = 1
hidden_neurons = 100
exit_neurons = 1

xph = tf.placeholder(tf.float32, [None, periods, appetizer])
yph = tf.placeholder(tf.float32, [None, periods, exit_neurons])

cell = tf.contrib.rnn.BasicRNNCell(num_units = hidden_neurons, activation = tf.nn.relu)

cell = tf.contrib.rnn.OutputProjectionWrapper(cell, output_size = 1)

exit_rnn, _ = tf.nn.dynamic_rnn(cell, xph, dtype = tf.float32)
calculateError = tf.losses.mean_squared_error(labels = yph, predictions = exit_rnn)
otimizador = tf.train.AdamOptimizer(learning_rate = 0.001)
training = otimizador.minimize(calculateError)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for epoch in range(2000):
        _, cost = sess.run([training, calculateError], feed_dict = {xph: X_batches, yph: y_batches})
        if epoch % 100 == 0:
            print("[INFO] Epoch: {} - Level Error: {}".format(epoch,cost))

    forecast = sess.run(exit_rnn, feed_dict = {xph: X_test})

y_test.shape
y_test2 = np.ravel(y_test)

final_forecast = np.ravel(forecast)

mae = mean_absolute_error(y_test2, final_forecast)

for (host, forecast, date) in list(zip(servers, final_forecast, datecollect)):
    send.postForecastMemory(host, forecast, cost, date)

Output:

A:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1020,1,1] vs. [1019,1,1]
         [[{{node gradients/mean_squared_error/SquaredDifference_grad/BroadcastGradientArgs}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "teste.py", line 61, in <module>
    _, custo = sess.run([treinamento, erro], feed_dict = {xph: X_batches, yph: y_batches})
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1020,1,1] vs. [1019,1,1]
         [[node gradients/mean_squared_error/SquaredDifference_grad/BroadcastGradientArgs (defined at teste.py:55) ]]

Click data view

Shapes:

[INFO] base shape: (1020,)
[INFO] X shape: (1019,)
[INFO] X batches shape: (1, 1019, 1)
[INFO] Y batches shape: (1, 1019, 1)
[INFO] X teste shape: (1, 1019, 1)
[INFO] Y teste shape: (1, 1019, 1)
Solution

  • The main culprit seems the fixed temporal dimension in your RNN cell.

    xph = tf.placeholder(tf.float32, [None, periods, appetizer])
yph = tf.placeholder(tf.float32, [None, periods, exit_neurons])

cell = tf.contrib.rnn.BasicRNNCell(num_units = hidden_neurons, activation = tf.nn.relu)

    Here, in both xph and yph you have specified the temporal dimension as periods. So, if you have a longer or shorter signal, there will be an error.

    I couldn't infer the exact dimension of your model layers as you didn't specify the input shape or model summary. So, using placeholder numbers.

    There are two possible fixes.

    1. Instead of using a fixed temporal dimension = period, use None. 
    xph = tf.placeholder(tf.float32, [None, None, appetizer])
yph = tf.placeholder(tf.float32, [None, None, exit_neurons])

    But, downside is you must have signals with same length in each batch, or you can simly use batch size = 1 to train without worrying about the temporal length.

    1. Use truncating/padding to fix the length issue. Just pass your signal to a pre-processing function to add/ remove extra temporal points.
    import numpy as np
def pre_process(x, fixed_len = 1000): # x.shape -> (100, 1000, 1)

    if x.shape[1] >= fixed_len:
       return x[:,:fixed_len,:]
    else:
       z_ph = np.zeros((x.shape[0], fixed_len, x.shape[2]))
       z_ph[:,:x.shape[1],:] = x
       return z_ph


    X_batches = pre_process(X_batches, YOU_CHOOSE_THIS_LENGTH) # based on the length of your data
X_test = pre_process(X_test, YOU_CHOOSE_THIS_LENGTH)