Search code examples
pythonscikit-learnmultiprocessinglarge-dataauc

How to calculate f1 score using multiprocessing in python?


I've got an array of paired binary labels: y_true, y_pred. My array contains ~50 million elements, and I wish to evaluate success using f1 score preferably, or AUC.

However, calculating f1 using sklearn takes relatively long time – about half the time needed for an entire epoch. Calculating AUC was faster, but too slow as well. Similar question yielded Faster AUC in sklearn or python , but I'm not sure I can try this one.

Is there a way to speed up those calculations, perhaps with multiprocessing?


Solution

  • Alright, so apparently Scikit-learn implementation ran too slow on my two vectors, so I modified this implementation, so it fits numpy arrays and it is now much faster (0.25 seconds compared to 50+ seconds on sk-learn). Original implementation using torch.Tensors.

    def f1_loss(y_true, y_pred, beta=1) -> numpy.float32:
        '''Calculate F1 score.
        
        The original implmentation is written by Michal Haltuf on Kaggle.
        
        Reference
        ---------
        - https://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric
        - https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
        - https://discuss.pytorch.org/t/calculating-precision-recall-and-f1-score-in-case-of-multi-label-classification/28265/6
        
        '''
        assert y_true.shape[1] == 1
        assert y_pred.shape[1] == 1
            
        
        tp = (y_true * y_pred).sum()
        tn = ((1 - y_true) * (1 - y_pred)).sum()
        fp = ((1 - y_true) * y_pred).sum()
        fn = (y_true * (1 - y_pred)).sum()
        
        epsilon = 1e-7
        
        precision = tp / (tp + fp + epsilon)
        recall = tp / (tp + fn + epsilon)
        
        f1 = (1 + beta**2)* (precision*recall) / (beta**2 * precision + recall + epsilon)
    
        return f1