I've tuned succesfully several models using {tidymodels} and {workflow_set}. However, when testing the validation dataset with tune::last_fit(), the parameters obtained by tune::select_best don't behave well. This makes me want to manually test other sets of parameters on the validation set. I find tune::show_best() and tune::select_best() very limited for doing so, since they only consider one metric when choosing the right parameters. I've managed to filter the tibbles with a more complex logic involving several metrics using pure {dplyr} but this is not optimal and is time consuming, involving manually finalizing each model every time I want to test one of the models.
Is there a way to cherry pick a set of parameters based on the workflow_id and .config data of the tibble?
It would be really helpful that tune::select_best() could take more conditions to pick a model.
This is the classical process to get the "best" set of parameters (which unfortunately is not in my case since I get a model with a very high roc_auc but very bad spec for example).
models_tuned <- models %>%
workflow_map("tune_bayes",
resamples = cv_folds,
initial = 20,
iter= 10,
metrics = mm_metrics,
verbose = TRUE)
best_results <- models_tuned %>%
extract_workflow_set_result(id = "norm_nnet") %>%
select_best(metric = "accuracy")
fitted_workflow <- models_tuned %>%
extract_workflow(id = "norm_nnet") %>%
finalize_workflow(best_results) %>%
last_fit(split=split_df,
metrics=mm_metrics)
As answered here:
You can pass any parameters you want to finalize_workflow(). The parameters argument takes any tibble that includes values for the tuning parameters.
There are also other select_*() functions to use as well as the unreleased desirability2 package.
I get a model with a very high roc_auc but very bad spec for example
That sounds like a class imbalance. You might look into using a cost-sensitive learning method or rebalance (perhaps with themis).