merge three images with Tensorflow with indices

Question

I have a problem with using Tensorflow. I have four images with their corresponding indices. I want to make an image from them. I tried for loops, tf.gather, tf.assign, and so on but all show error. If somebody help me, it would be really appreciated. I explain my question with one small example: We have 4 tensors and their indices from tensor tf.ktop function: (I write like MATLAB for just for simplicity)

a = [1, 2; 5, 6] a_idx = [0, 1; 2, 3] b = [3, 4; 7, 8] b_idx = [0, 1; 2, 3] c = [9, 10; 13, 14] c_idx = [0, 1; 2, 3] d = [11, 12; 15, 16] d_idx = [0, 1; 2, 3]

I am looking for a big image from a, b, c, and d and their indices like:

image = [a b; c d] image = [1, 2, 3, 4; 5, 6, 7, 8;9 10, 11, 12;13, 14, 15, 16]

In python I have something like:

a, a_idx, b, b_idx, c, c_idx, d, d_idx

n_x = tf.Variable(tf.zeros([1, 4, 4, 1]))

n_patches = tf.extract_image_patches(
    n_x,
    [1, 2, 2, 1],
    [1, 2, 2, 1],
    [1, 1, 1, 1],
    "SAME"
)

So, n_patches is 4 tensors and I need to put a to d values to each patch corresponding to a_idx to d_idx. Its really easy for me in MATLAB or Numpy to do that using for loop but in tensorflow I can not

Answer 1

In your comments, I suspect you made a tiny error in your desired output, image.

I interpret that you want is given

values = np.array([[2, 5],\
                   [4, 6]])
indices = np.array([[0, 3],\
                    [2, 1]])

your result would be

[[2. 0. 0. 0.]
 [0. 0. 0. 5.]
 [0. 0. 4. 0.]
 [0. 6. 0. 0.]]

So you want to obtain a sort of one hot encoded matrix, but with values corresponding to given indices. This can be obtained like so:

import numpy as np
values = np.array([[2, 5],\
                   [4, 6]])
indices = np.array([[0, 3],\
                    [2, 1]])

# Make a matrix with only zeros
n_hots = np.zeros_like((indices))
# Now row 0,1,2 and 3 should have values corresponding to the
# indices. That is we should first "unpack" the values and indices:
indices=indices.ravel()
values=values.ravel()
# values are now: [2,5,4,6]
# indices are now: [0,3,2,1]
# values:
# n_hots[row,indices[row]]=values[indices[row]]
# e.g.
# n_hots[0,0]=2
# n_hots[1,3]=5
# n_hots[2,2]=4
# n_hots[3,1]=6
# Notice how the first slices are a ascending range of values:
# [0,1,2,3], and the second slice are the raveled indices, and the
# right hand side of the equal sign are the ravele values!
# That means we can just do the following:
n_hots[np.arange(4),indices]=values
print(n_hots)

In tensorflow it would be a bit different. First generating a one_hot tensor that have ones at the 2nd axis value: at the indices, and then multiplying that with the corresponding indices:

import numpy as np
import tensorflow as tf

indices=tf.placeholder(shape=(None),dtype=tf.int32)
values=tf.placeholder(shape=(None),dtype=tf.float32)
one_hots=tf.one_hot(indices, tf.shape(indices)[0])
n_hots=one_hots*tf.gather(values, indices)

with tf.Session() as sess:
    _values = np.array([[2, 5],\
                       [4, 6]])
    _indices = np.array([[0, 3],\
                        [2, 1]])
    n_h=sess.run(n_hots, {indices: _indices.ravel(), values:_values.ravel()})
    print(n_h)