Calculation of AUC 95 % CI from Cross Validation (Python, sklearn)

Question

I am looking for the right way to calculate the AUC 95 % CI from my 5-fold CV.

n = 81 of my Training Dataset

So, if I apply 5-fold CV that equals a mean of approx . n = 16 in every fold in the test group.

My Python code below.

folds = 5
seed = 42

# Grid Search
fit_intercept=[True, False]
C = [np.arange(1,41,1)]
penalty = ['l1', 'l2']
params = dict(C=C, fit_intercept = fit_intercept, penalty = penalty)

logreg = LogisticRegression(random_state = seed)

logreg_grid = GridSearchCV(logreg, param_grid = params , cv=folds, scoring='roc_auc',  iid='False')

# fit the grid with data
logreg_grid.fit(X_train, y_train)

# fit best estimator
logreg = logreg_grid.best_estimator_

# Calculate AUC in 5-fold Stratified CV
logreg_scores = cross_val_score(logreg, X_train, y_train, cv=folds, scoring='roc_auc')
print('LogReg:',logreg_scores.mean())

# LogReg Scores: [0.95714286, 0.85, 0.98333333, 0.85, 0.56666667]  
# Mean: 0.8414285714285714````

#AUC from LogReg = 0.8414

#Three ways I have tried to calculate the 95 % CI:

#LogReg Scores: [0.95714286, 0.85, 0.98333333, 0.85, 0.56666667]  
# Mean: 0.8414285714285714


                    ### First try ###
import statsmodels.stats.api as sms
conf = sms.DescrStatsW(logreg_scores).tconfint_mean(.05)
print(conf)

#Out: Lower 0.636, Upper: 1.047

                    ### Second Try ###
import scipy.stats
def mean_confidence_interval(data, confidence=0.95):
    a = 1.0 * np.array(data)
    n = len(a)
    m, se = np.mean(a), scipy.stats.sem(a)
    h = se * scipy.stats.t.ppf((1 + confidence) / 2, n-1)
    return m, m-h, m+h


mean_confidence_interval(logreg_scores, confidence=0.95)

#Out: Mid: 0.84, Lower: 0.64, Upper: 1.05)

                      ### Third ###
# interval = t * np.sqrt( (AUC * (1 - AUC)) / n)
# n = 16 (validation set), because the mean in of alle 5 folds is 16 aof my n = 81
# t = 2.120 (Source: https://www.sjsu.edu/faculty/gerstman/StatPrimer/t-table.pdf)

interval = 2.120 * np.sqrt( (0.8414285714285714 * (1 - 0.8414285714285714)) / 16)
print((.84 + interval)*100)
print(.84)
print((.84 - interval)*100)
print(interval)

# Output: Lower: 64.64 , Mid: 0.84, Upper: 103.36 , Interval: 0.194

My Question: All of the results look similar. But, what I am doing wrong, because I do not understand how the AUC can be > 1.0?

Thank you for reeding, I am looking forward for your answers.

Cheers Mischa

Answer 1

I am not sure whether it solves your questions, but I guess it is because you are applying a t-test for a extremly small sample size (n=5). A large variance is kind of expected, that's why mean + SD > 1 in your case. Notice that all your three methods were t-test based.

To obtain enough number of comparison, you might want to try 1) multiple repeated CV with different subclassing OR 2) bootstrappin. Some useful discussion on CV: CV