Search code examples
rsubsetr-caretdata-partitioning

How does createDataPartition function from caret package split data?


From the documentation:

For bootstrap samples, simple random sampling is used.

For other data splitting, the random sampling is done within the levels of y when y is a factor in an attempt to balance the class distributions within the splits.

For numeric y, the sample is split into groups sections based on percentiles and sampling is done within these subgroups.

For createDataPartition, the number of percentiles is set via the groups argument.

I don't understand why this "balance" thing is needed. I think I understand it superficially, but any additional insight would be really helpful.


Solution

  • It means, if you have a data set ds with 10000 rows

    set.seed(42)
    ds <- data.frame(values = runif(10000))
    

    with 2 "classes" with unequal distribution (9000 vs 1000)

    ds$class <- c(rep(1, 9000), rep(2, 1000))
    ds$class <- as.factor(ds$class)
    table(ds$class)
    #    1    2 
    # 9000 1000 
    

    you can create a sample, which tries to maintain the ratio / "balance" of the factor classes.

    dpart <- createDataPartition(ds$class, p = 0.1, list = F)
    dsDP <- ds[dpart, ]
    table(dsDP$class)
    #   1   2 
    # 900 100