I'm having an issue finding the AUC value in TensorFlow. I did the following to find the AUC:
auc,update_op = tf.contrib.metrics.streaming_auc(predictions=y_pred_class,labels=y_true_class,curve='ROC')
Then, I did the following:
session.run(tf.initialize_all_variables())
session.run(tf.initialize_local_variables())
auc_value, _ = session.run([auc,update_op], feed_dict=feed_dict_validate)
The code runs, but always getting the value 0. What might I be doing wrong?
Is it correct to pass the classes of the prediction and the correct labels in tf.contrib.metrics.streaming_auc?
EDIT
You can get the full code from, here.
Thanks.
Unfortunately, and confusingly, tf.contrib.metrics.streaming_auc() takes different parameters than tf.confusion_matrix() for example. You should not pass the class labels to this one, unlike for tf.confusion_matrix().
Here predictions should be the probability of each class (the value you get y_pred_class from with tf.argmax()...).
Also labels should be in the one_hot format, furthermore in a boolean Tensor. (You can simply cast it to boolean, 0 will become false, 1 true.)
Looking at your code, a further problem is that you have to run the AUC update_op for each batch you run so that the AUC can accumulate the data. On the other hand, once you're done with a certain amount, you need to reset the internal variables of the AUC to zero so that further calculations, like AUC for validation, or the next training batch don't include the previous data. For this to happen, you need to be able to fish out those variables among all variables, therefore I've put this in a variable scope I'm calling "AUC". Putting this all together, a few parts where I've changed the code (will attach the full running, tested code for reference at the bottom):
Where the auc is defined (note, I've switched to the current version, tf.metrics.auc() from the deprecated one in contrib):
with tf.variable_scope( "AUC" ):
auc, auc_update_op = tf.metrics.auc( predictions=y_pred, labels=y_true, curve = 'ROC' )
auc_variables = [ v for v in tf.local_variables() if v.name.startswith( "AUC" ) ]
auc_reset_op = tf.initialize_variables( auc_variables )
The show_progress() function (note some formatting changes in the msg = ... line, those are not strictly necessary but reflect my taste only):
def show_progress(epoch, feed_dict_train, feed_dict_validate, val_loss):
acc, auc_value = session.run([ accuracy, auc_update_op ], feed_dict=feed_dict_train)
session.run( auc_reset_op )
val_acc, val_auc_value = session.run([ accuracy, auc_update_op ], feed_dict=feed_dict_validate)
session.run( auc_reset_op )
msg = 'Training Epoch {} --- Tr Acc: {:>6.1%}, Tr AUC: {:>6.1%}, Val Acc: {:>6.1%}, Val AUC: {:>6.1%}, Val Loss: {:.3f}'
print(msg.format(epoch + 1, acc, auc_value, val_acc, val_auc_value, val_loss))
And finally, in the train() function, note the inserted auc_update_op:
session.run( [ optimizer, auc_update_op ], feed_dict=feed_dict_tr)
Also, put session initialization at the end, just as "best practice", this is not strictly necessary:
with tf.Session() as session:
#session.run(tf.initialize_all_variables())
# need to init local variables for internal auc calculations
init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
session.run( init )
train( num_iteration=10000 )
With this, the output is:
Training Epoch 1 --- Tr Acc: 51.6%, Tr AUC: 51.0%, Val Acc: 46.1%, Val AUC: 46.1%, Val Loss: 4.893
Training Epoch 2 --- Tr Acc: 53.9%, Tr AUC: 53.0%, Val Acc: 53.5%, Val AUC: 55.3%, Val Loss: 0.691
Training Epoch 3 --- Tr Acc: 65.6%, Tr AUC: 63.9%, Val Acc: 65.2%, Val AUC: 69.0%, Val Loss: 0.647
Training Epoch 4 --- Tr Acc: 71.1%, Tr AUC: 71.6%, Val Acc: 68.0%, Val AUC: 74.8%, Val Loss: 0.586
Training Epoch 6 --- Tr Acc: 73.0%, Tr AUC: 76.8%, Val Acc: 69.5%, Val AUC: 75.9%, Val Loss: 0.588
Training Epoch 7 --- Tr Acc: 77.3%, Tr AUC: 82.4%, Val Acc: 73.8%, Val AUC: 77.7%, Val Loss: 0.563
Training Epoch 8 --- Tr Acc: 81.2%, Tr AUC: 87.0%, Val Acc: 78.9%, Val AUC: 85.2%, Val Loss: 0.475
Training Epoch 9 --- Tr Acc: 83.6%, Tr AUC: 90.9%, Val Acc: 75.0%, Val AUC: 83.5%, Val Loss: 0.517
Training Epoch 11 --- Tr Acc: 91.8%, Tr AUC: 94.3%, Val Acc: 73.0%, Val AUC: 81.4%, Val Loss: 0.646
(see more in attachedtrain.htmlortrain.ipynbbelow)
Data: Data is on Kaggle, from the Dogs vs. Cats competition (train.zip)
Runnable tested main code (please note you need to adapt the data paths in the code to your specific setup and the batch size to your hardware capacity) in three formats: train.ipynb train.html train.py
Required (imported) file, place it in the same folder as train:
dataset.py