Using batch normalization with dropout in test phase gives Nan

Question

I am trying to add batch normalization to my CNN and have read a lot of posts about how to do so but still my implementation yields an array of nans as a result when setting training to False.

When setting training to True even on test time, results are not Nan but they are worse than on training time if I test on training images.

I used a decay of 0.9 and trained on 15 000 iterations

Here is my graph building, adding update ops as a dependency as recommended in tf.layers.batch_normalization documentation and then using sess

extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
  with tf.control_dependencies(extra_update_ops):
    phase_train = tf.placeholder(tf.bool, name='phase_train')

    ###### Other placeholders and variables declarations ######

    # Build a Graph that computes the logits predictions from the inference model.

    loss, eval_prediction = inference(train_data_node, train_labels_node, batch_size, phase_train, dropout_out_keep_prob)

    # Build a Graph that trains the model with one batch of examples and updates the model parameters.

    ###### Should I rather put the dependency here ? ######
    train_op = train(loss, global_step)

    saver = tf.train.Saver(tf.global_variables())

    with tf.Session() as sess:
          init = tf.global_variables_initializer()
          sess.run(init)

          # Start the queue runners.
          coord = tf.train.Coordinator()
          threads = tf.train.start_queue_runners(sess=sess, coord=coord)

          for step in range(startstep, startstep + max_steps):
            feed_dict = fill_feed_dict(train_labels_node, train_data_node, dropout_out_keep_prob, phase_train, batch_size, phase_train_val=True,drop_out_keep_prob_val=1.)
            _, loss_value = sess.run([train_op, loss], feed_dict=feed_dict)

here is my batch_norm function call:

def batch_norm_layer(inputT, is_training, scope):
    return tf.layers.batch_normalization(inputT, training=is_training, center=False, reuse=None, momentum=0.9)

and now here is how I restore the model for testing:

phase_train = tf.placeholder(tf.bool, name='phase_train')

###### Other placeholder definitions ######

loss, logits = inference(test_data_node, test_labels_node, batch_size, phase_train, drop_out_keep_prob)
pred = tf.argmax(logits, dimension=3)

saver = tf.train.Saver()

with tf.Session() as sess:
  saver.restore(sess, test_ckpt)

  threads = tf.train.start_queue_runners(sess=sess)

  feed_dict = fill_feed_dict(test_labels_node, test_data_node, drop_out_keep_prob, phase_train, batch_size=1, phase_train_val=False, drop_out_keep_prob_val=1.)

  pred_loss, dense_prediction, predicted_image = sess.run([loss, logits, pred], feed_dict=feed_dict)

Here dense_prediction gives an array of Nans and thus predicted_image is all 0 Is there an error in my construction ? How can I fix it / diagnose it ?

Any help would be welcome, I have read a lot of tutorials to use "hand made" batch norm but I couldn't find a well guided tutorial on how to use the official batch norm, guess it is because it's too obvious, but it's not for me !

Answer 1

It appears the problem was coming from the fact that I was using batch normalization along with tf.nn.dropout dropout implementation.

Switching to tf.layers.dropout solved the issue.