Very strange accuracy change?

Question

Below is my catboost model:

from sklearn.metrics import r2_score
cb_model = CatBoostRegressor(iterations=500,
                             learning_rate=0.05,
                             depth=10,
                             random_seed = 42,
                             bagging_temperature = 0.2,
                             od_type='Iter',
                             metric_period = 50,
                             od_wait=20)
cb_model.fit(X, y)
r2_score(cb_model.predict(X), y)

The output is: 0.9999993582437351

When I try and print a confusion matrix:

classificationSummary(y, clf.predict(X))

the output consistently becomes the following:

Confusion Matrix (Accuracy 0.9700)

       Prediction
Actual  0  1
     0 50  1
     1  2 47

Why does the accuracy decrease?

Answer 1

You are using the wrong metric. r2_score calculates r squared or coefficient of determination, this is used for regression and not for calculating accuracy.

You should use accuracy score