I'm doing a binary classification with a GBM using the h2o package. I want to assess the predictive power of a certain variable, and if I'm correct I can do so by comparing the AUC of a model with the specific variable and a model without the specific variable.
I'm taking the titanic dataset as an example.
So my hypothesis is: Age has significant predictive value whether someone will survive.
df <- h2o.importFile(path = "http://s3.amazonaws.com/h2o-public-test-data/smalldata/gbm_test/titanic.csv")
response <- "survived"
df[[response]] <- as.factor(df[[response]])
## use all other columns (except for the name) as predictors
predictorsA <- setdiff(names(df), c(response, "name"))
predictorsB <- setdiff(names(df), c(response, "name", "age"))
splits <- h2o.splitFrame(
data = df,
ratios = c(0.6,0.2), ## only need to specify 2 fractions, the 3rd is implied
destination_frames = c("train.hex", "valid.hex", "test.hex"), seed = 1234
)
train <- splits[[1]]
valid <- splits[[2]]
test <- splits[[3]]
gbmA <- h2o.gbm(x = predictorsA, y = response, distribution="bernoulli", training_frame = train)
gbmB <- h2o.gbm(x = predictorsB, y = response, distribution="bernoulli", training_frame = train)
## Get the AUC
h2o.auc(h2o.performance(gbmA, newdata = valid))
[1] 0.9631624
h2o.auc(h2o.performance(gbmB, newdata = test))
[1] 0.9603211
I know the pROC package has a roc.test function to compare AUC of two ROC curves and I would like to apply this function on the outcomes of my h2o model.
You can do something like this-
valid_A <- as.data.frame(h2o.predict(gbmA,valid))
valid_B <- as.data.frame(h2o.predict(gbmB,valid))
valid_df <- as.data.frame(valid)
roc1 <- roc(valid_df$survived,valid_A$p1)
roc2 <- roc(valid_df$survived,valid_B$p1)
> roc.test(roc1,roc2)
DeLong's test for two correlated ROC curves
data: roc1 and roc2
Z = -0.087489, p-value = 0.9303
alternative hypothesis: true difference in AUC is not equal to 0
sample estimates:
AUC of roc1 AUC of roc2
0.9500141 0.9504367