Keras accuracy of only most certain predictions

Question

I am using keras to perform binary classification (single label, one or zero).

I would like to create a custom metric that takes the highest 10% from the predictions and calculates the accuracy of those.

So for example with the labels [1, 0, 1, 1, 0, 1, 0, 1, 0, 1]

and the predictions [0.5, 0.7, 0.98, 0.1, 0.2, 0.5, 0.2, 0.1, 0.9, 0.8]

It only computes the accuracy of the 0.98 prediction.

Answer 1

I managed to solve my problem using this code:

import tensorflow as tf

def topacc(y_true, y_pred):
  k = tf.cast(len(y_true) // 10, 'int64')
  y_true, y_pred = tf.transpose(y_true), tf.transpose(y_pred)
  return tf.keras.metrics.top_k_categorical_accuracy(y_true, y_pred, k=k)

This works as a full keras metric