how to score precision, recall and f1-score in a multi-class dataset using cross-validate?

Question

This code is working for the dataset which has 2 classes, but not in multi-class

scoring = {'accuracy' : make_scorer(accuracy_score), 
       'precision' : make_scorer(precision_score),
       'recall' : make_scorer(recall_score), 
       'f1_score' : make_scorer(f1_score)}
scores = cross_val_score(gnb,x,y, cv=5, scoring=scoring)
print(scores)

the error show

ValueError: For evaluating multiple scores, use sklearn.model_selection.cross_validate instead. {'accuracy': make_scorer(accuracy_score), 'precision': make_scorer(precision_score, average=None), 'recall': make_scorer(recall_score), 'f1_score': make_scorer(f1_score)} was passed

when I check the code with changed it like this

scores = cross_val_score(gnb,x,y, cv=5, scoring='precision')

the error show

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

it's not working when I set the average in make_scorer

Answer 1

Use scoring function 'f1_macro' or 'f1_micro' for f1.
Likewise, 'recall_macro' or 'recall_micro' for recall.

When calculating precision or recall, it is important to define positive class, but in multi-class dataset, it is hard to define it.

So, average of precision (recall) per each class value (i.e., iterating each value and regard it as positive class) should be calculated.

Edit.

Try use the following code (micro average precision, recall, f1).

scoring = {'accuracy' : make_scorer(accuracy_score), 
       'precision' : make_scorer(precision_score, average = 'micro'),
       'recall' : make_scorer(recall_score, average = 'micro'), 
       'f1_score' : make_scorer(f1_score, average = 'micro')}