I would like to compare the performance of several machine learning algorithms (e.g., decisions trees from rpart, xgb, ...) including their hyperparameter tuning using mlr3. In other words, I would like to compare already tuned instances of different algorithms rather than the algorithms with their default hyperparameter values.
mlr3 provides AutoTuner-Objects to carry out nested resampling and hyperparameter tuning. There is also a benchmark() function to conduct comparisons of several learners. The benchmark() function in turn uses benchmark_grid() to set up the benchmarking. According to this manual one can pass "an AutoTuner object to mlr3::resample() or mlr3::benchmark()". I do not understand how I can pass an AutoTuner object to benchmark_grid().
The following code (benchmarking a tuned decision tree with the default version; based on the code in this book) is not working. It returns an error message: "Error: Element with key 'rpart_tuned' not found in DictionaryLearner!"
library("mlr3verse")
### Benchmarking including hyperparameter tuning
# nested resampling:
# - inner sampling: 5-fold CV
# - outer sampling: manually defined hold-out sample
# defining AutoTuner for the inner resampling
learner = lrn("classif.rpart")
resampling = rsmp("cv", folds = 5)
# resampling = rsmp("holdout")
measure = msr("classif.acc")
search_space = ps(maxdepth = p_int(lower = 1, upper = 10))
terminator = trm("none")
tuner = tnr("grid_search", resolution = 5)
rpart_tuned = AutoTuner$new(learner, resampling, measure, terminator, tuner, search_space)
## Outer re-sampling
# hold-out sample with pre-defined partitioning into train and test set
outer_resampling = rsmp("custom")
train_sets = list(1:120)
test_sets = list(121:150)
outer_resampling$instantiate(task, train_sets, test_sets)
## Defining benchmark design
design = benchmark_grid(
tasks = tsks(c("iris")),
learners = lrns(c("rpart_tuned", "classif.rpart"),
predict_type = "prob", predict_sets = c("train", "test")),
resamplings = outer_resampling
)
The problem in your code is that you're trying to create a new learner instead of reusing your own in
lrns(c("rpart_tuned", "classif.rpart"),
predict_type = "prob", predict_sets = c("train", "test")),
lrns(c("rpart_tuned")) is trying to retrieve a rpart_tuned from dictionary of built in learners in mlr3.
If you want to reuse rpart_tuned, just do this:
design = benchmark_grid(
tasks = tsks(c("iris")),
learners = c(rpart_tuned, lrn("classif.rpart",
predict_type = "prob", predict_sets = c("train", "test"))),
resamplings = outer_resampling
)
This in turn, uses rpart_tuned autotuner and creates a new learner classif.rpart from dictionary.