How to change decision threshold on a loaded logistic regression model

Question

I´m working on a logistic regression model using Python and I managed to adjust the threshold manually. However, when I save the model using pickle, the threshold doesn´t seem to change. I get exactly the same results for different thresholds. Here´s the code:

filename = 'model202104.sav'
pickle.dump(logreg, open(filename, 'wb'))
loaded_model2 = pickle.load(open(filename, 'rb'))
result = loaded_model2.score(X_test, y_pred)
print(result)

Here´s the code I use to manually change thresholds:

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.2,random_state=7)
logreg = LogisticRegression(max_iter=10000)
logreg.fit(X_train,y_train)
#y_pred=logreg.predict(X_test)
THRESHOLD=0.5
y_pred=np.where(logreg.predict_proba(X_test)[:,1] > THRESHOLD, 1, 0)

Thanks in advance :)

Answer 1

The second argument for score is supposed to be the true observed values, not y_pred.

# Load model
loaded_model2 = pickle.load(open('model202104.sav', 'rb'))

# Score model with `y_test`
result = loaded_model2.score(X_test, y_test) # You had `y_pred` here
print(result)

Moreover, you always have to set the threshold manually in sklearn. Otherwise, LogisticRegression always classifies as 1 if the predicted probability is greater than or equal to 0.5. So to score your model with a custom threshold:

# Import accuracy score function
from sklearn.metrics import accuracy_score

# Classify with custom threshold (for example, 0.85)
thr = 0.85
y_pred = np.where(loaded_model2.predict_proba(X_test)[:, 1] >= thr, 1, 0)

# Score
print('Accuracy with threshold set to', str(thr) + ':', accuracy_score(y_test, y_pred))