I am transitioning to new version of TF for stability reasons (I was using a nightly docker build on Ubuntu 18.04 from before mainline switched to CUDA 10). When I attempt to run my models in the new version I get the following error, which I assume to mean that there is an incompatibility with the models trained on the older version.
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 426, in import_graph_def
graph._c_graph, serialized, options) # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: NodeDef mentions attr 'explicit_paddings' not in Op<name=Conv2D; signature=input:T, filter:T -> output:T; attr=T:type,allowed=[DT_HALF, DT_BFLOAT16, DT_FLOAT, DT_DOUBLE]; attr=strides:list(int); attr=use_cudnn_on_gpu:bool,default=true; attr=padding:string,allowed=["SAME", "VALID"]; attr=data_format:string,default="NHWC",allowed=["NHWC", "NCHW"]; attr=dilations:list(int),default=[1, 1, 1, 1]>; NodeDef: {{node FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
What do I need to do to update the previously trained models to work with the new version of TF or do I need to continue running that version until my next training session?
After doing some looking, the graph has to be updated. Since I did not still have the training checkpoints, I was successful in updating the graph by exporting from the previously frozen graph as the checkpoint.
python3 export_inference_graph.py --input_type image_tensor --pipeline_config_path FROZENGRAPHDIRECTORY/pipeline.config --trained_checkpoint_prefix FROZENGRAPHDIRECTORY/model.ckpt --output_directory FROZENGRAPHDIRECTORY_tfNEWTFVERSION