Skip connection in a neural network for one feature

Question

I have 1000 objects, each with 100 time stamps and 5 features, but one is very important, so I don't want to pass it through the LSTM, but immediately transfer it to the final layer, how can I do this? Need a lot of input layers in a neural network?

Answer 1

I think either of these should do it for you:

import tensorflow as tf

# dummy data
inp1 = tf.random.uniform(shape=(1000, 100, 5))

# ALTERNATIVE 1 (use lambda layer to split input)
inputs = tf.keras.layers.Input((100, 5), name='inputs')

# assuming that the important feature is at index -1
input_lstm = tf.keras.layers.Lambda(lambda x: x[:, :, :4])(inputs)
input_dense = tf.keras.layers.Lambda(lambda x: x[:, :, -1])(inputs)
x = tf.keras.layers.LSTM(
    units=64, 
    recurrent_initializer='ones', 
    kernel_initializer='ones')(input_lstm)
x = tf.keras.layers.Concatenate()([x, input_dense])
out = tf.keras.layers.Dense(units=1, kernel_initializer='ones')(x)

model = tf.keras.Model(inputs=inputs, outputs=out)

# print(model.summary())
out = model(inp1)
print(out[:5])

# ALTERNATIVE 2 (split data before neural net)
# assuming that the important feature is at index -1
inp2 = inp1[:, :, -1]
inp1 = inp1[:, :, :4]

input_lstm = tf.keras.layers.Input((100, 4), name='lstm_input')
input_dense = tf.keras.layers.Input((100,), name='dense_input')
x = tf.keras.layers.LSTM(
    units=64, 
    recurrent_initializer='ones', 
    kernel_initializer='ones')(input_lstm)
x = tf.keras.layers.Concatenate()([x, input_dense])
out = tf.keras.layers.Dense(units=1, kernel_initializer='ones')(x)

model = tf.keras.Model(inputs=[input_lstm, input_dense], outputs=out)

# print(model.summary())
out = model([inp1, inp2])
print(out[:5])

# output:
# tf.Tensor(
# [[118.021736]
#  [117.11683 ]
#  [115.341644]
#  [120.00911 ]
#  [114.4716  ]], shape=(5, 1), dtype=float32)
# tf.Tensor(
# [[118.021736]
#  [117.11683 ]
#  [115.341644]
#  [120.00911 ]
#  [114.4716  ]], shape=(5, 1), dtype=float32)

The layers weights are initialized to ones just to illustrate that they give the same output.