VIF No intercept: vifs may not be sensible

Question

I am trying to test that the assumption of my Multinomial Logistic Regression model holds or fails.

Multinomial Logistic Regression model is a model which could be created and compared, although it has this assumption:

Multinomial logistic regression does have assumptions, such as the assumption of independence among the dependent variable choices. This assumption states that the choice of or membership in one category is not related to the choice or membership of another category (i.e., the dependent variable). (source)

So after searching for a while I came across variance inflation factors which measure how much the behavior (variance) of an independent variable is influenced, or inflated, by its interaction/correlation with the other independent variables.

#Multinomial Logisitic Regression
mlrModel = train(
  label~.,
  data = overallDataset,
  method = "multinom",
  trControl = trainControl(method = "cv", number = 5),
  trace = FALSE
)
mlrModel$results
head(predict(mlrModel, type = "prob"))
library(car)
vif(mlrModel$finalModel)

But I get this warning:

Warning message:
In vif.default(mlrModel$finalModel) :
  No intercept: vifs may not be sensible.

And the results varied massively: 8.575035e+07 to -7.188586e+13

I have not removed the intercept from the model.

I found this question: vif(): "Warning message: No intercept: vifs may not be sensible." Trying to check multicollinearity with multinomial logistic regression but it had no answer.

Answer 1

First of all, in what have quoted, note that it says "Multinomial logistic regression does have assumptions, such as the assumption of independence among the dependent variable choices." When you look for variance inflation factor, you are looking at multicollinearity between independent variables . I don't quite see the connection here.

There is an intercept in the multinomial model. But you need to remember there are more than 1 set of coefficients, if there are n classes in your labels, one of them is treated as the intercept and you have n-1 set of coefficients.

Hence you see that error, because in the vif function called (car:::vif.default), there is a line in the code:

if (names(coefficients(mod)[1]) == "(Intercept)") {
        v <- v[-1, -1]
        assign <- assign[-1]
    }
    else warning("No intercept: vifs may not be sensible.")

So, we can use an example below to demonstrate why it returns the error:

library(nnet)
fit = multinom(Species ~.,data=iris)
vif(fit)
 Sepal.Length   Sepal.Width  Petal.Length   Petal.Width 
-1.878714e+16 -8.846005e+15 -1.827592e+15 -4.954974e+15 
Warning message:
In vif.default(fit) : No intercept: vifs may not be sensible.

Same error, and we look at the coefficients:

coefficients(fit)
           (Intercept) Sepal.Length Sepal.Width Petal.Length Petal.Width
versicolor    18.69037    -5.458424   -8.707401     14.24477   -3.097684
virginica    -23.83628    -7.923634  -15.370769     23.65978   15.135301

names(coefficients(fit))
NULL

Because it is multinomial, the coefficients are stored as a matrix (using one class as reference, you estimate log-odds of other classes), hence the names() function doesn't work, and returns you the error.

There might not be an easy method or way to calculate the vif for a multinomial model, without diving deep into the statistics, what I suggest below is a rough approximation using logistic on the difference classes:

labels = rownames(coefficients(fit))
ref = set.diff(fit$lab,labels)
t(sapply(labels,function(i){
       dat = iris
       dat$Species = as.numeric(dat$Species == i)
       vif(glm(Species ~ .,data=dat,family="binomial"))
}))

           Sepal.Length Sepal.Width Petal.Length Petal.Width
versicolor     6.860029    1.511153    27.926317   14.795556
virginica      1.624657    2.797563     1.985489    3.093114