Optimal parameter estimation for a classifier with multiple parameters

Question

The image on the left shows a standard ROC curve formed by sweeping a single threshold and recording the corresponding True Positive Rate (TPR) and False Positive Rate (FPR).

The image on the right shows my problem setup where there are 3 parameters, and for each, we have only 2 choices. Together, it produces 8 points as depicted on the graph. In practice, I intend to have thousands of possible combinations of 100s of parameters, but the concept remains the same in this down-scaled case.

I intend to find 2 things here:

• Determine the optimum parameter(s) for the given data
• Provide an overall performance score for all combinations of parameters

In the case of the ROC curve on the left, this is done easily using the following methods:

• Optimal parameter: Maximal difference of TPR and FPR with a cost component (I believe it is called the J-statistic?)
• Overall performance: Area under the curve (the shaded portion in the graph)

However, for my case in the image on the right, I do not know if the methods I have chosen are the standard principled methods that are normally used.

• Optimal parameter set: Same maximal difference of TPR and FPR

  Parameter score = TPR - FPR * cost_ratio

• Overall performance: Average of all "parameter scores"

I have found a lot of reference material for the ROC curve with a single threshold and while there are other techniques available to determine the performance, the ones mentioned in this question is definitely considered a standard approach. I found no such reading material for the scenario presented on the right.

Bottomline, the question here is two-fold: (1) Provide methods to evaluate the optimal parameter set and overall performance in my problem scenario, (2) Provide reference that claims the suggested methods to be a standard approach for the given scenario.

P.S.: I had first posted this question on the "Cross Validated" forum, but didn't get any responses, in fact, got only 7 views in 15 hours.

Answer 1

I'm going to expand a little on aberger's previous answer on a Grid Search. As with any tuning of a model it's best to optimise hyper-parameters using one portion of the data and evaluate those parameters using another proportion of the data, so GridSearchCV is best for this purpose.

First I'll create some data and split it into training and test

import numpy as np
from sklearn import model_selection, ensemble, metrics

np.random.seed(42)

X = np.random.random((5000, 10))
y = np.random.randint(0, 2, 5000)

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.3)

This gives us a classification problem, which is what I think you're describing, though the same would apply to regression problems too.

Now it's helpful to think about what parameters you may want to optimise. A cross-validated grid search is a computational expensive process, so the smaller the search space the quicker it gets done. I will show an example for a RandomForestClassifier because it's my go to model.

clf = ensemble.RandomForestClassifier()    
parameters = {'n_estimators': [10, 20, 30],
              'max_features': [5, 8, 10],
              'max_depth': [None, 10, 20]}

So now I have my base estimator and a list of parameters that I want to optimise. Now I just have to think about how I want to evaluate each of the models that I'm going to build. It seems from your question that you're interested in the ROC AUC, so that's what I'll use for this example. Though you can chose from many default metrics in scikit or even define your own.

gs = model_selection.GridSearchCV(clf, param_grid=parameters,
                                  scoring='roc_auc', cv=5)
gs.fit(X_train, y_train)

This will fit a model for all possible combinations of parameters that I have given it, using 5-fold cross-validation evaluate how well those parameters performed using the ROC AUC. Once that's been fit, we can look at the best parameters and pull out the best performing model.

print gs.best_params_
clf = gs.best_estimator_

Outputs:

{'max_features': 5, 'n_estimators': 30, 'max_depth': 20}

Now at this point you may want to retrain your classifier on all of the training data, as currently it's been trained using cross-validation. Some people prefer not to, but I'm a retrainer!

clf.fit(X_train, y_train)

So now we can evaluate how well the model performs on both our training and test set.

print metrics.classification_report(y_train, clf.predict(X_train))
print metrics.classification_report(y_test, clf.predict(X_test))

Outputs:

             precision    recall  f1-score   support

          0       1.00      1.00      1.00      1707
          1       1.00      1.00      1.00      1793

avg / total       1.00      1.00      1.00      3500

             precision    recall  f1-score   support

          0       0.51      0.46      0.48       780
          1       0.47      0.52      0.50       720

avg / total       0.49      0.49      0.49      1500

We can see that this model has overtrained by the poor score on the test set. But this is not surprising as the data is just random noise! Hopefully when performing these methods on data with a signal you will end up with a well-tuned model.

EDIT

This is one of those situations where 'everyone does it' but there's no real clear reference to say this is the best way to do it. I would suggest looking for an example close to the classification problem that you're working on. For example using Google Scholar to search for "grid search" "SVM" "gene expression"