How to use F-score as error function to train neural networks?

Question

I am pretty new to neural networks. I am training a network in tensorflow, but the number of positive examples is much much less than negative examples in my dataset (it is a medical dataset). So, I know that F-score calculated from precision and recall is a good measure of how well the model is trained. I have used error functions like cross-entropy loss or MSE before, but they are all based on accuracy calculation (if I am not wrong). But how do I use this F-score as an error function? Is there a tensorflow function for that? Or I have to create a new one?

Thanks in advance.

Answer 1

I think you are confusing model evaluation metrics for classification with training losses.

Accuracy, precision, F-scores etc. are evaluation metrics computed from binary outcomes and binary predictions.

For model training, you need a function that compares a continuous score (your model output) with a binary outcome - like cross-entropy. Ideally, this is calibrated such that it is minimised if the predicted mean matches the population mean (given covariates). These rules are called proper scoring rules, and the cross-entropy is one of them.

Also check the thread is-accuracy-an-improper-scoring-rule-in-a-binary-classification-setting

If you want to weigh positive and negative cases differently, two methods are

• oversample the minority class and correct predicted probabilities when predicting on new examples. For fancier methods, check the under sampling module of imbalanced-learn to get an overview.
• use a different proper scoring rule for training loss. This allows to e.g. build in asymmetry in how you treat positive and negative cases while preserving calibration. Here is review of the subject.

I recommend just using simple oversampling in practice.