Search code examples
pythonnumpymatrixtheano

Broadcasting for subtensor created from matrix (Theano)


I want to create two subtensors from a matrix, using indices to select the respective rows. One subtensor has several rows, the other just one, which should be broadcast to allow for element-wise addition.

My question is: how do I indicate that I want to allow for broadcasting on the specific dimension in the sub-tensor resulting given the indices (subtensorRight in the example below)?

Here is the example showing what I want to do:

import theano
import numpy as np
import theano.tensor as T

def embedding_matrix(D, N, name):
    W_values = np.random.uniform(size=(D, N))
    return theano.shared(value=W_values, name=name)

rE = embedding_matrix(4, 5, "rE")
lis = T.ivector('lis')# [1,2]
subtensorLeft = rE[lis,:]
ri = T.ivector('ri')#[1]
subtensorRight = rE[ri,:]


def fnsim(left, right):
    return - T.sqrt(T.sum(T.sqr(left - right), axis=1))

distances_test = theano.function(
    inputs=[lis, ri],
    outputs=fnsim(subtensorLeft, subtensorRight)
)

print distances_test([1,2],[1])

It throws this error:

ValueError: Input dimension mis-match. (input[0].shape[0] = 2, input[1].shape[0] = 1)
Apply node that caused the error: Elemwise{Composite{sqr((i0 - i1))}}[(0, 0)](AdvancedSubtensor1.0, AdvancedSubtensor1.0)
Toposort index: 2
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Inputs shapes: [(2, 5), (1, 5)]
Inputs strides: [(40, 8), (40, 8)]
Inputs values: ['not shown', array([[ 0.39528934,  0.4414946 ,  0.36837258,  0.52523446,  0.35431748]])]
Outputs clients: [[Sum{axis=[1], acc_dtype=float64}(Elemwise{Composite{sqr((i0 - i1))}}[(0, 0)].0)]]

===

UPDATE 1:

It stops complaining and gives the expected result when reshaping subtensorRight this way:

subtensorRight = rE[ri,:]
subtensorRight = subtensorRight.reshape((1, subtensorRight.shape[1]))

Question: Is this the right way to go?

UPDATE 2:

It does not work if I try to reshape as below (which I thought to be eqivalent to the reshaping above):

subtensorRight = rE[ri,:]
subtensorRight = subtensorRight.reshape(subtensorRight.shape)

The error is:

ValueError: Input dimension mis-match. (input[0].shape[0] = 2, input[1].shape[0] = 1)
Apply node that caused the error: Elemwise{Composite{sqr((i0 - i1))}}[(0, 0)](AdvancedSubtensor1.0, Reshape{2}.0)
Toposort index: 6
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Inputs shapes: [(2, 5), (1, 5)]
Inputs strides: [(40, 8), (40, 8)]
Inputs values: ['not shown', array([[ 0.54193252,  0.36793023,  0.89009085,  0.02487759,  0.95955664]])]
Outputs clients: [[Sum{axis=[1], acc_dtype=float64}(Elemwise{Composite{sqr((i0 - i1))}}[(0, 0)].0)]]

Question: Why does reshaping with taking dimension 0 from the subtensor give a different result?


Solution

  • The problem is that your theano function does not know in advance that the right (ri) indices will have only 1 element (so for all in knows you'll be trying to subtract a NxD matrix from a MxD matrix, which won't work in general. However for your case you only ever want N=1.)

    The solution is to declare your right index as a scalar.

    The following code, I believe, does what you want:

    import theano
    import numpy as np
    import theano.tensor as T
    
    def embedding_matrix(D, N, name):
        W_values = np.random.uniform(size=(D, N))
        return theano.shared(value=W_values, name=name)
    
    rE = embedding_matrix(4, 5, "rE")
    lis = T.ivector('lis')# [1,2]
    subtensorLeft = rE[lis,:]
    ri = T.iscalar('ri')  # Instead of: ri = T.ivector('ri')
    subtensorRight = rE[ri,:]
    
    
    def fnsim(left, right):
        return - T.sqrt(T.sum(T.sqr(left - right), axis=1))
    
    distances_test = theano.function(
        inputs=[lis, ri],
        outputs=fnsim(subtensorLeft, subtensorRight)
    )
    
    print distances_test([1,2],1)  # Instead of: distances_test([1,2],[1])
    

    (Outputs [-0. -1.01565315])

    Shameless Self Promotion:

    You can use the Plato library to make more readable theano code. In your case:

    from plato.core import symbolic
    import numpy as np
    import theano.tensor as T
    
    @symbolic
    def distances_test(matrix, test_rows, reference_row):
        left = matrix[test_rows]
        right = matrix[reference_row]
        return - T.sqrt(T.sum(T.sqr(left - right), axis=1))
    
    f = distances_test.compile()
    
    print f(np.random.uniform(size=(4, 5)), np.array([1,2]), 1)