How to merge two tensors at the beginning of a network in Torch?

Question

Given the following beginning of a network

local net = nn.Sequential()
net:add(SpatialConvolution(3, 64, 4, 4, 2, 2, 1, 1))

with an input tensor input

local input = torch.Tensor(batchSize, 3, 64, 64)

// during training
local output = net:forward(input)

I want to modify the network to accept a second tensor cond as input

local cond = torch.Tensor(batchSize, 1000, 1, 1)

// during training
local output = net:forward({input, cond})

I modified the network by adding a JoinTable before the SpatialConvolution is added, like so:

local net = nn.Sequential()
net:add(nn.JoinTable(2, 4))
net:add(SpatialConvolution(3, 64, 4, 4, 2, 2, 1, 1))

This is not working because both tensors have different sizes in dimensions 2, 3, and 4. Giving the cond tensor as size of (batchSize, 1000, 64, 64) is not an option since its a waste of memory.

Is there any best practise for merging two different tensors at the beginning of a network to be feed into the first layer.

Answer 1

There is no such thing as "merging" tensors which do not have compatible shapes. You should simply pass a table of tensors and start your network with SelectTable operation and work with nngraph, not simple Sequential. In particular - how would you expect Spatial Convolution to work on such odd "tensor" which "narrows down" to your cond? There is no well defined operation in mathematics for such use case, thus you have to be more specific (which you will achieve with nngraph and SelectTable).