During hyperparameter tuning of a multioutput regressor, are the hyperparameters optimized separately for each target or for the entire regressor?

Question

I'm using multioutput regressor to predict three targets. My base model is xgboost. I'm trying to tune the hyperparameters using 'HalvingRandomSearchCV'. My question is: when performing tuning on multioutput regressor, does it tune the parameters for each target or the entire regressor? If it's for each target, how can I access the tuned hyperparameters?

hyperparameters = {
    'estimator__n_estimators': [100, 300,500, 600,700, 800, 900,1000, 1200],
    'estimator__max_depth': [3,4, 5,6, 7, 10, 11, 12, 15],
    'estimator__learning_rate': [0.05,0.04, 0.06,0.08, 0.1, 0.2, 0.3,0.4, 0.5]
}
reg=XGBRegressor()
multioutput_regressor = MultiOutputRegressor(reg)   ### Wrapping the lgb models
kf = KFold(n_splits=10, shuffle=True, random_state=42) 
search = HalvingRandomSearchCV(multioutput_regressor, hyperparameters, n_jobs=-1, cv=kf, random_state=42)

search=search.fit(X_train, y_train)

y_pred_train = search.predict(X_train)

#predictions on the test set
y_pred = search.predict(X_test)   ### testing the trained model
print('Best hyperparameters:', search.best_params_)

When I print the best_params_, it only returns one set of hyperparameters.

Answer 1

Your code tunes one set of hyperparameters for all targets (optimizing the average performance across targets). The tuner doesn't know much about its estimator, in this case that it's a multioutput wrapper; it just sets candidate hyperparameters, fits, scores, cuts candidates and increases resources, and repeats. "Sets candidate hyperparameters" means setting the multioutput regressor's estimator's hyperparameters, and so they're set for all the clones that are fit to the various targets.

If you want to tune hyperparameters for each target, you just need to reorder the tuning and multioutput wrapper:

hyperparameters = {
    'n_estimators': [100, 300,500, 600,700, 800, 900,1000, 1200],
    'max_depth': [3,4, 5,6, 7, 10, 11, 12, 15],
    'learning_rate': [0.05,0.04, 0.06,0.08, 0.1, 0.2, 0.3,0.4, 0.5]
}
reg=XGBRegressor()
kf = KFold(n_splits=10, shuffle=True, random_state=42) 
search = HalvingRandomSearchCV(reg, hyperparameters, n_jobs=-1, cv=kf, random_state=42)

model = MultiOutputRegressor(search)

model.fit(X_train, y_train)

In this version, the multioutput wrapper clones the entire tuning object for each target, and those are fit independently. The individual best hyperparameters are now available through model.estimators_[i].best_params_ for each target i.