Search code examples
python-3.xmergekeras-layer

How to merge two layers of different shapes for an LSTM input?

I would like to combine outputs of 2 different layers in my network, as follows: 

l1.shape
TensorShape([Dimension(None), Dimension(10), Dimension(100)])
l2.shape
TensorShape([Dimension(None), Dimension(20), Dimension(30)])

I would like to combine the layers l1 and l2 then feed them to a bi-LSTM layer. I tried the "Concatenate" layer, but it doesn't work. I want something that could pad the layer with lower last dimension to get the same dimension as the other layer. ie: padding the last dimension of l2 two get the following:

l2_padded = some_function(l2, axis=-1, dim=l1.shape[-1])
l2_padded.shape
TensorShape([Dimension(None), Dimension(20), Dimension(100)])

Then perform the concatenation, 

c = Concatenate(axis=1)([l1, l2_padded])
c.shape
TensorShape([Dimension(None), Dimension(30), Dimension(100)])
bilstm = Bidirectional(LSTM(100))(c)
# other layers ...

Could you give some example and/or references?

Solution

  • You can use a combination of reshape and ZeroPadding1D:

    import tensorflow.keras.backend as K
from tensorflow.keras.layers import ZeroPadding1D

x1 = Input(shape=(10, 100))
x2 = Input(shape=(20, 30))
x2_padded = K.reshape(
    ZeroPadding1D((0, x1.shape[2] - x2.shape[2]))(
        K.reshape(x2, (-1, x2.shape[2], x2.shape[1]))
    ),
    (-1, x2.shape[1], x1.shape[2])
)

    It looks a bit clunky but unfortunately the ZeroPadding1D doesn't allow for specifying a padding axis and will always use axis=1. Same for K.transpose which, unlike Numpy, does not provide a way to specify the axes that should be swapped (hence using reshape).