Search code examples
cross-validationresamplingmlr3

Aggregating performance measures in mlr3 ResampleResult when some iterations have NaN values

I would like to calculate an aggregated performance measure (precision) for all iterations of a leave-one-out resampling.

For a single iteration, the result for thie measure can only be 0, 1 (if positive class is predicted) or NaN (if negative class is predicted.

I want to aggregate this over the existing values of the whole resampling, but the aggregation result is always NaN (naturally, it will be NaN for many iterations). I could not figure out (from the help page for ResampleResult$aggregate()) how to do this:

gr = po(lrn("classif.kknn", predict_type = "prob"),
        param_vals = list(k = 10, distance=2, kernel='rectangular' )) %>>%
  po("threshold", param_vals = list(thresholds = 0.5))

glrn = GraphLearner$new(gr)

resampling = rsmp("loo")
 
rr = resample(task, glrn, resampling, store_models = TRUE)

rr$aggregate(msr("classif.precision"))
Solution

  • I have doubts if this is a statistically sound approach, but technically you can set the aggregating function for a measure by overwriting the aggregator slot:

    m = msr("classif.precision")
m$aggregator = function(x) mean(x, na.rm = TRUE)