How to fix 'incompatible dimensions' error from cor() function

Question

I am using the predict() function to predict the Purchase variable in blackFriday_test. When I use cor() with theses variables as arguments, I get an 'incompatible dimensions' error message.

I tried looking at the dimension of the Purchas variable in blackFriday_test which is 107516, but the predicted values turn out to be only 32955.

The data was downloaded from https://www.kaggle.com/mehdidag/black-friday.

library(caret)

blackFriday <- read.csv("BlackFriday.csv", stringsAsFactors = T)

Here I remove the first two features because they are identifiers

nblackFriday <- blackFriday[, 3:12]

set.seed(189)
train <- sample(nrow(nblackFriday), as.integer(0.8 * nrow(nblackFriday)), replace = F)

blackFriday_train <- nblackFriday[train, ]
blackFriday_test <- nblackFriday[-train, ]

Removing NA's from the two variables where they are present

nblackFriday$Product_Category_2 <- ifelse(is.na(nblackFriday$Product_Category_2), mean(nblackFriday$Product_Category_2, na.rm = T), nblackFriday$Product_Category_2)
nblackFriday$Product_Category_3 <- ifelse(is.na(nblackFriday$Product_Category_3), mean(nblackFriday$Product_Category_3, na.rm = T), nblackFriday$Product_Category_3)

blackFriday_train$Product_Category_2 <- nblackFriday$Product_Category_2[train]
blackFriday_train$Product_Category_3 <- nblackFriday$Product_Category_3[train]

m <- train(Purchase ~ ., data = blackFriday_train, method = "rpart")

p <- predict(m, blackFriday_test)

cor(p, blackFriday_test$Purchase)
```
#This is where I get the error

I expect the number of predicted values to be the same as the number of rows in blackFriday_test, but they are not.

Answer 1

You replaced NAs in your training set, but not in your testing set, so those cases are being omitted.

> head(blackFriday_test)
   Gender   Age Occupation City_Category Stay_In_Current_City_Years Marital_Status Product_Category_1
3       F  0-17         10             A                          2              0                 12
6       M 26-35         15             A                          3              0                  1
15      F 51-55          9             A                          1              0                  5
16      F 51-55          9             A                          1              0                  4
21      M 26-35         12             C                         4+              1                  5
22      M 26-35         12             C                         4+              1                  8
   Product_Category_2 Product_Category_3 Purchase
3                  NA                 NA     1422
6                   2                 NA    15227
15                  8                 14     5378
16                  5                 NA     2079
21                 14                 NA     8584
22                 NA                 NA     9872

Just impute them like you did for the training set it works as expected.

blackFriday_test$Product_Category_2 <- nblackFriday$Product_Category_2[-train]
blackFriday_test$Product_Category_3 <- nblackFriday$Product_Category_3[-train]
p <- predict(m, blackFriday_test)

> length(p) == nrow(blackFriday_test)
[1] TRUE
> cor(p, blackFriday_test$Purchase)
[1] 0.7405558

Try using the partitioning and preprocessing features of caret, itself. For me they help to avoid these types of easy errors.