How to calculate cross-validation with multiple scores for multiclass?

Question

I'm starting my first machine learning code with python. But, I encountered an error while computing the recall, precision and f1 for my multiclass model.

X = pd.read_excel(path, dtype=int)
allarray = X.values
X_data = allarray[:,0:-1]
Y = allarray[:,-1]
X_scaled = scaler.fit_transform(X_data)

create_model = create_custom_model(n_features, n_classes, 8, 3)

estimator = KerasClassifier(build_fn=create_model, epochs=100, batch_size=100, verbose=0)
scores = cross_validate(estimator, X_scaled, Y, cv=10, scoring=('precision', 'recall', 'f1'), return_train_score=False)
print(scores['precision'])
print(scores['recall'])
print(scores['f1'])

I'm getting this error:

ValueError: Target is multiclass but average='binary'. Please choose another average setting.

But cross_validate has no parameter average

Answer 1

The problem is that the default average setting for precision, recall, and F1 scores applies to binary classification only.

What you should do is replace the scoring=('precision', 'recall', 'f1') argument in your cross_validate with something like

scoring=('precision_macro', 'recall_macro', 'f1_macro')

There are several suffix options available for each metric - macro, micro, weighted etc. See the documentation, the example, and the score links therein.