I've got an array of paired binary labels: y_true
, y_pred
.
My array contains ~50 million elements, and I wish to evaluate success using f1 score preferably, or AUC.
However, calculating f1 using sklearn
takes relatively long time – about half the time needed for an entire epoch. Calculating AUC was faster, but too slow as well.
Similar question yielded Faster AUC in sklearn or python , but I'm not sure I can try this one.
Is there a way to speed up those calculations, perhaps with multiprocessing?
Alright, so apparently Scikit-learn implementation ran too slow on my two vectors, so I modified this implementation, so it fits numpy arrays and it is now much faster (0.25 seconds compared to 50+ seconds on sk-learn). Original implementation using torch.Tensors.
def f1_loss(y_true, y_pred, beta=1) -> numpy.float32:
'''Calculate F1 score.
The original implmentation is written by Michal Haltuf on Kaggle.
Reference
---------
- https://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
- https://discuss.pytorch.org/t/calculating-precision-recall-and-f1-score-in-case-of-multi-label-classification/28265/6
'''
assert y_true.shape[1] == 1
assert y_pred.shape[1] == 1
tp = (y_true * y_pred).sum()
tn = ((1 - y_true) * (1 - y_pred)).sum()
fp = ((1 - y_true) * y_pred).sum()
fn = (y_true * (1 - y_pred)).sum()
epsilon = 1e-7
precision = tp / (tp + fp + epsilon)
recall = tp / (tp + fn + epsilon)
f1 = (1 + beta**2)* (precision*recall) / (beta**2 * precision + recall + epsilon)
return f1