For my project, I need to convert a directed graph into a tensorflow implementation of the graph as if it was a neural network. In tensorflow version 1, I could just define all of my inputs as placeholders and then just generate the dataflow graph for the outputs using a breadthfirst search of the graph. Then I would just feed in my inputs using a feed_dict. However, in TensorFlow v2.0 they have decided to do away with placeholders entirely.
How would I make a tf.function for each graph that takes in a variable amount of inputs and returns a variable amount of outputs without using a placeholder?
I want to generate a tf.function like this that works for an arbitrary acyclic directed graph so that I can take advantage of tensorflow GPU support to run the graph feed forward a few thousand times in a row after I have generated it.
Edit for code example:
My graph is defined as a dictionary. Each key represents a node and has a corresponding value of another dictionary specifying incoming and outgoing links with weights.
{
"A": {
"incoming": [("B", 2), ("C", -1)],
"outgoing": [("D", 3)]
}
}
I have omitted the entries for B,C, and D for brevity. Here is how I would construct the code I want in tensorflow v1.0 where inputs is just a list of key values that are strictly inputs to the graph
def construct_graph(graph_dict, inputs, outputs):
queue = inputs[:]
make_dict = {}
for key, val in graph_dict.items():
if key in inputs:
make_dict[key] = tf.placeholder(tf.float32, name=key)
else:
make_dict[key] = None
# Breadth-First search of graph starting from inputs
while len(queue) != 0:
cur = graph_dict[queue[0]]
for outg in cur["outgoing"]:
if make_dict[outg[0]]: # If discovered node, do add/multiply operation
make_dict[outg[0]] = tf.add(make_dict[outg[0]], tf.multiply(outg[1], make_dict[queue[0]]))
else: # If undiscovered node, input is just coming in multiplied and add outgoing to queue
make_dict[outg[0]] = tf.multiply(make_dict[queue[0]], outg[1])
for outgo in graph_dict[outg[0]]["outgoing"]:
queue.append(outgo[0])
queue.pop(0)
# Returns one data graph for each output
return [make_dict[x] for x in outputs]
I would then be able to run the outputs many times as they are simply graphs with placeholders that I would provide a feed_dict for.
Obviously, this is not the intended way in TensorFlow v2.0 as they seem to strongly discourage the use of placeholders in this new version.
The point is that I only have to do this preprocessing for a graph once, as it returns a datagraph which is independent of the graph_dict definition.
Below is a sample code which you can use with TF 2.0.
It relies on the compatibility API
that is accessible as tensorflow.compat.v1, and requires to disable v2 behaviors.
I don't know if it behaves as you expected.
If not, then provide us more explanation of what you try to achieve.
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
@tf.function
def construct_graph(graph_dict, inputs, outputs):
queue = inputs[:]
make_dict = {}
for key, val in graph_dict.items():
if key in inputs:
make_dict[key] = tf.placeholder(tf.float32, name=key)
else:
make_dict[key] = None
# Breadth-First search of graph starting from inputs
while len(queue) != 0:
cur = graph_dict[queue[0]]
for outg in cur["outgoing"]:
if make_dict[outg[0]]: # If discovered node, do add/multiply operation
make_dict[outg[0]] = tf.add(make_dict[outg[0]], tf.multiply(outg[1], make_dict[queue[0]]))
else: # If undiscovered node, input is just coming in multiplied and add outgoing to queue
make_dict[outg[0]] = tf.multiply(make_dict[queue[0]], outg[1])
for outgo in graph_dict[outg[0]]["outgoing"]:
queue.append(outgo[0])
queue.pop(0)
# Returns one data graph for each output
return [make_dict[x] for x in outputs]
def main():
graph_def = {
"B": {
"incoming": [],
"outgoing": [("A", 1.0)]
},
"C": {
"incoming": [],
"outgoing": [("A", 1.0)]
},
"A": {
"incoming": [("B", 2.0), ("C", -1.0)],
"outgoing": [("D", 3.0)]
},
"D": {
"incoming": [("A", 2.0)],
"outgoing": []
}
}
outputs = construct_graph(graph_def, ["B", "C"], ["A"])
print(outputs)
if __name__ == "__main__":
main()
[<tf.Tensor 'PartitionedCall:0' shape=<unknown> dtype=float32>]
While the above snippet is valid, it is still tied to TF 1.0. To migrate it to TF 2.0 you have to refactor a little bit your code.
Instead of returning a list of tensors, which were callables with TF 1.0, I advise you to return a list of
keras.layers.Model.
Below is a working example:
import tensorflow as tf
def construct_graph(graph_dict, inputs, outputs):
queue = inputs[:]
make_dict = {}
for key, val in graph_dict.items():
if key in inputs:
# Use keras.Input instead of placeholders
make_dict[key] = tf.keras.Input(name=key, shape=(), dtype=tf.dtypes.float32)
else:
make_dict[key] = None
# Breadth-First search of graph starting from inputs
while len(queue) != 0:
cur = graph_dict[queue[0]]
for outg in cur["outgoing"]:
if make_dict[outg[0]] is not None: # If discovered node, do add/multiply operation
make_dict[outg[0]] = tf.keras.layers.add([
make_dict[outg[0]],
tf.keras.layers.multiply(
[[outg[1]], make_dict[queue[0]]],
)],
)
else: # If undiscovered node, input is just coming in multiplied and add outgoing to queue
make_dict[outg[0]] = tf.keras.layers.multiply(
[make_dict[queue[0]], [outg[1]]]
)
for outgo in graph_dict[outg[0]]["outgoing"]:
queue.append(outgo[0])
queue.pop(0)
# Returns one data graph for each output
model_inputs = [make_dict[key] for key in inputs]
model_outputs = [make_dict[key] for key in outputs]
return [tf.keras.Model(inputs=model_inputs, outputs=o) for o in model_outputs]
def main():
graph_def = {
"B": {
"incoming": [],
"outgoing": [("A", 1.0)]
},
"C": {
"incoming": [],
"outgoing": [("A", 1.0)]
},
"A": {
"incoming": [("B", 2.0), ("C", -1.0)],
"outgoing": [("D", 3.0)]
},
"D": {
"incoming": [("A", 2.0)],
"outgoing": []
}
}
outputs = construct_graph(graph_def, ["B", "C"], ["A"])
print("Builded models:", outputs)
for o in outputs:
o.summary(120)
print("Output:", o((1.0, 1.0)))
if __name__ == "__main__":
main()
What to notice here?
placeholder to keras.Input, requiring to set the shape of the input.keras.layers.[add|multiply] for computation.
This is probably not required, but stick to one interface.
However, it requires to wrap factors inside a list (to handle batching)keras.Model to returnHere is the output of the code.
Builded models: [<tensorflow.python.keras.engine.training.Model object at 0x7fa0b49f0f50>]
Model: "model"
________________________________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
========================================================================================================================
B (InputLayer) [(None,)] 0
________________________________________________________________________________________________________________________
C (InputLayer) [(None,)] 0
________________________________________________________________________________________________________________________
tf_op_layer_mul (TensorFlowOpLayer) [(None,)] 0 B[0][0]
________________________________________________________________________________________________________________________
tf_op_layer_mul_1 (TensorFlowOpLayer) [(None,)] 0 C[0][0]
________________________________________________________________________________________________________________________
add (Add) (None,) 0 tf_op_layer_mul[0][0]
tf_op_layer_mul_1[0][0]
========================================================================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0
________________________________________________________________________________________________________________________
Output: tf.Tensor([2.], shape=(1,), dtype=float32)