TF - how do I setup the model signature correctly for serving with Docker?

Question

I am trying to understand how to setup a TF model for serving with docker. I manged to install docker, I know how to export a trained model as a .pb. What I dont understand is how to correctly define the model signature for serving. I want to simply call a trained model in the terminal with docker. Could you explain me what I have to change in the following example?

I am doing the following steps:

1) create directory /tmp/serving_minimal, cd to it in the terminal $cd /tmp/serving_minimal

2) save in /tmp/serving_minimal the file generate_model.py for the following code

import numpy as np
import tensorflow as tf
import os, shutil

#%% Data

# Input (2D)
x = np.array([[x1,x2] for x1 in np.linspace(10,20,4) for x2 in np.linspace(-7,-3,3)])

# Output (3D)
f = np.array([[np.sin(np.sum(xx)),np.cos(np.sum(xx)),np.cos(np.sum(xx))**2] for xx in x])

#%% Model

print('**********************************************')
print('TF - save')

# Dimension of input x and output f
d_x = x.shape[-1]
d_f = f.shape[-1]

# Placeholders
x_p = tf.placeholder(tf.float64,[None,d_x],'my_x_p')
f_p = tf.placeholder(tf.float64,[None,d_f],'my_f_p')

# Model
model = x_p
model = tf.layers.dense(model,7,tf.tanh)
model = tf.layers.dense(model,5,tf.tanh)
model = tf.layers.dense(model,d_f,None)
model = tf.identity(model,'my_model')

# Session
sess = tf.Session()
sess.run(tf.global_variables_initializer())

# Evaluate for later check of serving
f_model = sess.run(model,{x_p:x})
folder = 'data'
if not os.path.exists(folder):
    os.mkdir(folder)
np.savetxt('data/x.dat',f_model)
np.savetxt('data/f_model.dat',f_model)

# Save model
folder = 'saved/model/001'
if os.path.exists(folder):
    shutil.rmtree(folder)
    print('Old model deleted')
saver = tf.saved_model.builder.SavedModelBuilder(folder)
############################################
# HOW DO I SET UP THE SIGNATURE CORRECTLY?
############################################
info_input = tf.saved_model.utils.build_tensor_info(x_p)
info_output = tf.saved_model.utils.build_tensor_info(model)
signature = tf.saved_model.signature_def_utils.build_signature_def(
        inputs={'x':info_input}
        ,outputs={'f':info_output}
        ,method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
        )
saver.add_meta_graph_and_variables(
        sess
        ,[tf.saved_model.tag_constants.SERVING]
        ,signature_def_map={'predict':signature}
        ####################################################################
        ### WHAT DO I NEED TO PUT HERE IN ORDER TO CALL THE MODEL LATER ON 
        ### WHILE SERVING WITH DOCKER AND HOW DO I CALL IT IN DOCKER??
        ####################################################################
        )
saver.save()

# Close and clean up
sess.close()
tf.reset_default_graph()

#%% Load in Python and check

print('**********************************************')
print('TF - load in Python')

# Session
sess = tf.Session()

# Load
tf.saved_model.loader.load(
        sess
        ,[tf.saved_model.tag_constants.SERVING]
        ,folder
        )

# Extract operations from graph
graph = tf.get_default_graph()
x_p = graph.get_tensor_by_name('my_x_p:0')
f_p = graph.get_tensor_by_name('my_f_p:0')
model = graph.get_tensor_by_name('my_model:0')

# Evaluate model
f_model2 = sess.run(model,{x_p:x})
print(f_model - f_model2)

# Close and clean up
sess.close()
tf.reset_default_graph()

4) run script in the terminal $python generate_model.py (exports the model and loads it in Python for checking)

5) start docker in the terminal $sudo docker ps

6) run model in docker

$ sudo docker run \
    -p 8501:8501 \
    --name my_container \
    --mount type=bind,source=/tmp/serving_minimal/saved/model,target=/models/model1 \
    -e MODEL_NAME=model1 \
    -t tensorflow/serving &

7) Check that model is active (it is)

$ sudo docker ps

8) [Error] Try to evaluate active model

$ curl -d '{"x": [[1.0,2.0],[10.0,20.0]]}' -X POST http://localhost:8501/v1/models/model1:predict

At 8) I get the error

{ "error": "Serving signature name: \"serving_default\" not found in signature def" }

but I do not understand the TF signature definition commands well enough in order to know what to do. Can you show me what needs to be corrected? Thank you!

Answer 1

See this line in your training code

signature_def_map={'predict':signature}

So you should change your post request body like this

{
  'signature_name': 'predict',
  'x': [[1.0,2.0],[10.0,20.0]]
}

Or change 'predict' to 'serving_default' in your training code, and you could not specify signature_name in post request