So I am using the sample weights function and I don't want to measure performance using the default sklearn scoring function.
It looks like it says Here that I can pass GridSearchCV the string 'roc_auc' and it should calculate auc for me but will the auc it calculates be a weighted auc or just a vanilla auc?
Thanks for the nerd snipe.
I crafted a binary classification dataset to test this problem.
x y weight
0 0 1
1 1 1
<repeated 25 times>
0 1 0
1 0 0
<repeated 25 times>
Using python:
X = np.array([[0], [1]] * 25 + [[0], [1]] * 25)
y = np.array([ 0 , 1 ] * 25 + [ 1 , 0 ] * 25)
w = np.array([ 1 , 1 ] * 25 + [ 0 , 0 ] * 25)
In this dataset, we can see that if sample weights are present, the model will produce a very good fit and have a very low log loss. If the weights are not present, the loss will be extremely high.
Then we can build use GridSearchCV to see if the weights are used.
clf = LogisticRegression(solver='newton-cg', C=100)
gs = GridSearchCV(clf, {},
fit_params={"sample_weight": w},
scoring="log_loss", cv=KFold(y.shape[0],10, shuffle=True))
gs.fit(X,y)
gs.grid_scores_
[mean: -2.68562, std: 0.68038, params: {}]
We can see the loss is fairly high, which would indicate the weights are not used.
I wrote a patch to scikit-learn to fix this. Please consider it experimental. https://github.com/scikit-learn/scikit-learn/compare/master...dmaust:master
After applying the patch, we can enable score_sample_weight, repeat the previous test, and can see the log loss we would expect from the weights being taken into account.
gs.score_sample_weight=True
gs.fit(X,y)
gs.grid_scores_
[mean: -0.00486, std: 0.00016, params: {}]