I'm experimenting with creating a super simple Tensorflow network (one data processing Lambda layer), and then converting the model to ONNX and verifying the results match when calling the ONNX model from onnxruntime. I'm using Tensorflow v2.5.0. & onnxruntime v1.8.1.
example_input2 = tf.convert_to_tensor([0,1000,2000,2000,2000,3000,3000,4000],dtype=tf.float32)
Model definition:
inp = keras.layers.Input(name="input", type_spec=tf.TensorSpec(shape=[None], dtype=tf.float32))
output = keras.layers.Lambda(lambda x: tf.roll(x,shift=-1,axis=0),name="output") (inp)
model = keras.Model(inp,output,name="pipeline")
Then I can feed my example_input2 into the network:
model.predict(example_input2)
Which provides the desired output (simply a tf.roll operation):
array([1000., 2000., 2000., 2000., 3000., 3000., 4000., 0.],
dtype=float32)
Great! Now I can save my tensorflow model,
model.save("OnnxInvestigateData/pipeline2", overwrite=True, include_optimizer=False, save_format='tf')
Then at the shell, I can convert it to an ONNX format using tf2onnx:
python -m tf2onnx.convert --opset 14 --saved-model pipeline2 --output pipeline2.onnx
Then, back in python, I can load the onnx model and try to feed through the same input:
sess = rt.InferenceSession("OnnxInvestigateData/pipeline2.onnx", log_verbosity_level=2)
xinput = example_input2.numpy()
sess.run(['output'],{"args_0":xinput})
Which provides output that matches the input, and not the desired output (which should have been tf.roll'd by -1):
[array([ 0., 1000., 2000., 2000., 2000., 3000., 3000., 4000.],
dtype=float32)]
I'm totally at a loss as to why the output here isn't matching when I call model.predict from within python on my original keras model. Any ideas?
Looks like you found a bug in tf2onnx. Here's a fix: https://github.com/onnx/tensorflow-onnx/pull/1616
If you were just doing a test and don't want to wait for the fix to merge, try using a positive shift value.
output = keras.layers.Lambda(lambda x: tf.roll(x,shift=2,axis=0),name="output") (inp)
Also if you want to convert directly from keras in your script you can do:
onnx_model, _ = tf2onnx.convert.from_keras(model, opset=14)
sess = rt.InferenceSession(onnx_model.SerializeToString())