Tensorflow variable_scope: tf.placeholder and tf.get_variable within a function

Question

I just tried to get an understanding of the TensorFlow naming behavior and I still need some clarification. I worked on a project where I went into trouble with the naming of tensor, because they were predefined in a function, which was called later.

So I have the following example here:

import tensorflow as tf


    def foo():

        with tf.variable_scope("foo", reuse=True):

            a = tf.placeholder(tf.float32,name="a")
            b = tf.placeholder(tf.float32,name="b")

        return a,b
    ##

    a,b  = foo()

    print(a)
    print(b)

I get the output:

Tensor("foo/a:0", dtype=float32)
Tensor("foo/b:0", dtype=float32)

When I call it again though, I get the output:

Tensor("foo_1/a:0", dtype=float32)
Tensor("foo_1/b:0", dtype=float32)

Why is this the case? I set reuse on true, so I expect the tensors to be in the same variable_scope "foo" again or that the program throws an error like "tensors already defined".

So, I tried a workaround with tf.get_variable:

    def foo():
    with tf.variable_scope("foo", reuse=True):

        a = tf.get_variable("v", [1])


    return a
##

a1 = foo()
print(a1)

graph = tf.get_default_graph()
#call tensors by name in tensorflow to avoid confusion with the naming
graph.get_tensor_by_name("foo/v:0")

Here, i get always the same output:

<tf.Variable 'foo/v:0' shape=(1,) dtype=float32_ref>

Unfortunately, I cannot work with variables because you cannot define a dynamic shape for them. You need placeholders to define variable shapes. Can someone explain me why the program continues to create new variable_scopes for placeholders but not when I call tf.get_variable() ?

Thank you!

Answer 1

You can force the reuse of a scope by adding a '/' after the name i.e.: tf.variable_scope("foo/", reuse=True):

However that won't solve your problem.

In the case of variables, calling tf.Variable will always create a new variable, whereas calling tf.get_variable will reuse it if it already exists.

But with Placeholders there is no tf.get_placeholder.

What you can do is define your placeholders outside of foo, only once, and get them by name using tf.get_default_graph().get_tensor_by_name(name) or directly using the python variable whenever you need them.

example with get_tensor_by_name:

import tensorflow as tf

with tf.name_scope("scope"):
    tf.placeholder(tf.float32,name="a")
    tf.placeholder(tf.float32,name="b")

def foo():
    a = tf.get_default_graph().get_tensor_by_name("scope/a:0")
    b = tf.get_default_graph().get_tensor_by_name("scope/b:0")

    return a,b

a,b = foo()

print(a)
print(b)

Note that placeholders, unlike variables, do not maintain a state that can be reused or not. They are merely a "pointer" to a tensor which will be fed later. They should not be part of your model, but an input to it, so you should not be creating them several times anyway.