Horizontal stacking in Pytorch

Question

I am trying to implement transformers and stuck at one point.

Say I have input sequence of shape [2,20] where 2 is the number of sample and 20 is the number of words in sequence ( sequence length ).

So, I create an array like [0,1,2, ... 19] of shape [1,20]. Now I want to stack it , something like the final shape should be [2,20] to be in-line with input sequence. Like below

[[0,1,2, ... 19],
[0,1,2, ... 19]]

Is there a torch function for doing so. I can loop and create the data and arrays but wanted to avoid it.

Answer 1

If the tensors you want to stack are of shape [1,20], you can use torch.cat()

t1 = torch.zeros([1,5]) # tensor([[0., 0., 0., 0., 0.]])
t2 = torch.ones([1,5]) # tensor([[1., 1., 1., 1., 1.]])

torch.cat([t1, t2]) # tensor([[0., 0., 0., 0., 0.],
                              [1., 1., 1., 1., 1.]])

If the tensors are 1-D, you can simply use torch.stack()

t1 = torch.zeros([5]) # tensor([0., 0., 0., 0., 0.])
t2 = torch.ones([5]) # tensor([1., 1., 1., 1., 1.])

torch.stack([t1, t2]) # tensor([[0., 0., 0., 0., 0.],
                                [1., 1., 1., 1., 1.]])

Now, for a shorter method for your case, you can do:

torch.arange(0,20).repeat(2,1) # tensor([[0,1,2, ... 19],
                                         [0,1,2, ... 19]])