Search code examples
rgrid-searche1071

Visualize optimization path of grid search for SVM classification parameter tuning

I have done a classification using radial kernel svm of the e1071 package and tried to tune the cost and gamma parameters with the tune function.

Here an example code with the iris data set:

data("iris", package = "datasets")

if (!require("e1071")) {install.packages("e1071")} else {library("e1071")}

Data <- list(Lab = as.character(iris[,5]),
             Mat = prcomp(iris[,-5])$x[,1:2])

set.seed(240319)

Data$svm.tune <- tune(svm, train.x = Data$Mat, train.y = as.factor(Data$Lab), 
                      ranges = list(cost = c(10**(-3:3)), 
                                    gamma = c(10**(-3:3))),
                      type = "C-classification", kernel = "radial", scale = FALSE,
                      tunecontrol = tune.control(sampling = "cross", cross = 5))

Data$svm.tune$Mat <- Data$svm.tune$performances[,-ncol(Data$svm.tune$performances)]
Data$svm.tune$Mat_2 <- xtabs(error~., data = Data$svm.tune$Mat)

par(pty = "m",
    mar = c(2,2,1,1),
    mgp = c(1,0,0),
    tck = -0.01, 
    cex.axis = 0.75, 
    font.main = 1)
image(x = log10(as.double(rownames(Data$svm.tune$Mat_2))),
      y = log10(as.double(colnames(Data$svm.tune$Mat_2))), 
      z = Data$svm.tune$Mat_2,
      col = viridis::inferno(50),
      xlab = expression(log[10](c)),
      ylab = expression(log[10](gamma)))
points(log10(Data$svm.tune$best.parameters$cost),
       log10(Data$svm.tune$best.parameters$gamma),
       pch = 19,
       col = "white")
dev.off()

resulting in the following graph:

best results

I marked the 'best' parameter combination with a white dot.

Is there a way to visualize the optimization path of a grid search, like this question (but without animation)?

Solution

  • Not exactly the result I was looking for, but in case someone else has done a grid search and want to vizualise the path of the minima, you can calculate the minima of your error parameter for each grid parameter.

    First, I calculated the value and the index of the respective minima:

    min.val <- sapply(unique(Data$svm.tune$Mat[,"cost"]), function(i) min(Data$svm.tune$Mat[,"error"][Data$svm.tune$Mat[,"cost"] == i]))
min.idx <- sapply(unique(Data$svm.tune$Mat[,"cost"]), function(i) which.min(Data$svm.tune$Mat[,"error"][Data$svm.tune$Mat[,"cost"] == i]))

    Then, I sorted the indices according to the minima values and vizualise the path with arrows:

    par(pty = "m",
    mar = c(2,2,1,1),
    mgp = c(1,0,0),
    tck = -0.01, 
    cex.axis = 0.75, 
    font.main = 1)
image(x = log10(as.double(rownames(Data$svm.tune$Mat_2))),
      y = log10(as.double(colnames(Data$svm.tune$Mat_2))), 
      z = Data$svm.tune$Mat_2,
      col = viridis::inferno(50),
      xlab = expression(log[10](C)),
      ylab = expression(log[10](gamma)))
box()
for(i in seq_len(length(min.val) - 1))
{
  arrows(log10(unique(Data$svm.tune$Mat[,"cost"]))[order(min.val, decreasing = TRUE)][i],
         log10(unique(Data$svm.tune$Mat[,"gamma"]))[min.idx[order(min.val, decreasing = TRUE)]][i],
         log10(unique(Data$svm.tune$Mat[,"cost"]))[order(min.val, decreasing = TRUE)][i + 1], 
         log10(unique(Data$svm.tune$Mat[,"gamma"]))[min.idx[order(min.val, decreasing = TRUE)]][i + 1],
         length = 0.05, 
         angle = 30,
         code = 2,
         lwd = 1.5,
         col = "white")
}
dev.off()

    resulting in the following graph: optimization path