Calculating Brier Score and Integrated Brier Score using ranger R package

Question

I want to calculate Brier score and integrated Brier score for my analysis using "ranger" R package.

As an example, I use the veteran data from the "survival" package as follows

install.packages("ranger")
library(ranger)
install.packages("survival")
library(survival)
#load veteran data
data(veteran)
data <- veteran
# training and test data
n <- nrow(data)
testind <- sample(1:n,n*0.7)
trainind <- (1:n)[-testind]
#train ranger
rg <- ranger(Surv(time, status) ~ ., data = data[trainind,])
# use rg to predict test data
pred <- predict(rg,data=data[testind,],num.trees=rg$num.trees)
#cummulative hazard function for each sample
pred$chf
#survival probability for each sample
pred$survival

How can I calculate Brier score and integrated Brier score?

Answer 1

The Integrated Brier Score (IBS) can be calculated using the pec function of the pec package but you need to define a predictSurvProb command to extract survival probability predictions from the ranger modeling approach (?pec:::predictSurvProb for a list of available models).
A possibile solution is:

predictSurvProb.ranger <- function (object, newdata, times, ...) {
    ptemp <- ranger:::predict.ranger(object, data = newdata, importance = "none")$survival
    pos <- prodlim::sindex(jump.times = object$unique.death.times, 
        eval.times = times)
    p <- cbind(1, ptemp)[, pos + 1, drop = FALSE]
    if (NROW(p) != NROW(newdata) || NCOL(p) != length(times)) 
        stop(paste("\nPrediction matrix has wrong dimensions:\nRequested newdata x times: ", 
            NROW(newdata), " x ", length(times), "\nProvided prediction matrix: ", 
            NROW(p), " x ", NCOL(p), "\n\n", sep = ""))
    p
}

This function can be used as follows:

library(ranger)
library(survival)
data(veteran)
dts <- veteran
n <- nrow(dts)
set.seed(1)
testind <- sample(1:n,n*0.7)
trainind <- (1:n)[-testind]
rg <- ranger(Surv(time, status) ~ ., data = dts[trainind,])

# A formula to be inputted into the pec command
frm <- as.formula(paste("Surv(time, status)~",
       paste(rg$forest$independent.variable.names, collapse="+")))

library(pec)
# Using pec for IBS estimation
PredError <- pec(object=rg,
    formula = frm, cens.model="marginal",
    data=dts[testind,], verbose=F, maxtime=200)

The IBS can be evaluated using the print.pec command, indicating in times the time points at which to show the IBS:

print(PredError, times=seq(10,200,50))

# ...
# Integrated Brier score (crps):
# 
#            IBS[0;time=10) IBS[0;time=60) IBS[0;time=110) IBS[0;time=160)
# Reference          0.043          0.183           0.212           0.209
# ranger             0.041          0.144           0.166           0.176