My goal is to create Resampling using mlr3 package that uses some sort of rolling CV. More concretely, I want to use n months of data in training set (say 6 months) and one month of data in test set.
Here is example of my data set:
DT <- structure(list(Not_FLS_positive = c(0.408197129345391, 0.765784452003651,
0.44694266987472, 0.261843524433751, 0.823612378660914, 0.463701982908819,
0.50286235791919, 0.202937028125778, 0.728864183190907, 0.396498796980005,
0.0645482452501452, 0.386210901850162, 0.518874968887414, 0.748527337592301,
0.453414087778976, 0.758566332033519, 0.544926574296856, 0.758151497552477,
0.641583008379657, 0.15000414834481, 0.271384717497718, 0.516634862689787,
0.379988384634531, 0.220277109433336, 0.368373019165353, 0.367294449514644,
0.924583091346553, 0.702895544677674, 0.560192483199204, 0.61212976022567,
0.0189164523355181, 0.308139052518045), Not_FLS_negative = c(0.690284576453995,
0.406288890732598, 0.965402804281092, 0.981830249730358, 0.750850410686136,
0.884676014270306, 0.978760474570646, 0.846013440637186, 0.319754417987223,
0.70256367709284, 0.0308636853895296, 0.247905085870738, 0.886999087364142,
0.28017920849581, 0.697253795735502, 0.720069692192815, 0.838131585497387,
0.967559943582511, 0.755745457562433, 0.97593960009956, 0.886833153571725,
0.587156724466938, 0.959097320169252, 0.0548411183937609, 0.957769849829918,
0.479382726292209, 0.626897867750767, 0.772670704388949, 0.9822450842114,
0.736829005226914, 0.420642163776653, 0.723886169418402), bin_aroundzero_ret_excess_stand_22 = structure(c(2L,
1L, 3L, 1L, 1L, 3L, 1L, 1L, 2L, 2L, 2L, 1L, 3L, 1L, 2L, 2L, 1L,
1L, 1L, 3L, 2L, 1L, 3L, 2L, 2L, 2L, 3L, 2L, 1L, 2L, 3L, 2L), levels = c("0",
"1", "-1"), class = "factor"), monthid = c("20141", "20141",
"20141", "20141", "20141", "20141", "20141", "20141", "20141",
"20141", "20142", "20142", "20142", "20142", "20142", "20142",
"20142", "20142", "20142", "20142", "20142", "20143", "20143",
"20143", "20143", "20143", "20143", "20143", "20143", "20143",
"20143", "20143")), row.names = c(NA, -32L), class = c("data.table",
"data.frame"))
Now, I want to use months 20141 and 20142 in train set and 20143 in the test set. I thougt the best way is to set monthid group role:
task <- as_task_classif(DT, id = "aroundzero", target = "bin_aroundzero_ret_excess_stand_22")
task$set_col_roles("monthid", "group")
and now use ResamplingRollingWindowCV from mlr3temporal package:
resampling = rsmp("forecast_cv", folds = 5, fixed_window = TRUE, horizon = 1L, window_size = 6)
resampling$instantiate(task)
but this return an error:
Error in max(ids) - self$param_set$values$horizon :
non-numeric argument to binary operator
I have then tried to use custom resampling:
custom = rsmp("custom")
train_sets = list(1:2)
test_sets = list(3)
custom$instantiate(task, train_sets, test_sets)
custom$train_set(1)
custom$test_set(1)
but this returns only 1,2 in train and 3 in test set. It seems it does't use groups bt individual observations. Is it possible to vreate my by month rolling cv in current Resampling scheme or should I create completely new class for it?
Grouping is not supported by ResamplingForecastCV. You could try Leave-time-out resampling and tag monthid with the "time" row role. But with this method you also predict months in the past e.g. training set is 20143 and 20142 and test set is 20141. If that's not working for you, you need to use ResamplingCustom.
this returns only 1,2 in train and 3 in test set
This is the correct behavior. You pass the row ids of the task. You have to do the grouping yourself before and pass the grouped IDs to the resampling.
Update
With you data:
custom = rsmp("custom")
train_sets = list(1:21)
test_sets = list(22:32)
custom$instantiate(task, train_sets, test_sets)