Is there something similar to R's brglm to help deal with quasi-separation in Python using statsmodels Logit?

Question

I am using Logit from statsmodels to create a regression model.

I get the error: LinAlgError: Singular matrix and then when I remove 1 variable at a time from my dataset, I finally got a different error: PerfectSeparationError: Perfect separation detected, results not available.

I suspect that the original error (LinAlgError) is related to perfect separation because I had the same problem in R and got around it using a brglm (bias reduced glm).

I have a boolean y variable and 23 numeric and boolean x variables.

I have already run a VIF function to remove any variables which have high multicollinearity scores (I started with 26 variables).

I have tried using the firth_regression.py instead to account for perfect separation but I got a memory error: MemoryError.(https://gist.github.com/johnlees/3e06380965f367e4894ea20fbae2b90d)

I have tried the LogisticRegression from sklearn but cannot get the p values which is no good to me.

I even tried removing 1 variable at a time from my dataset. When I got down to 4 variables left (I had 23), then I got PerfectSeparationError: Perfect separation detected, results not available.

Has anyone experienced this and how do you get around it?

Appreciate any advice!

    X = df.loc[:, df.columns != 'VehicleMake']
    y = df.iloc[:,0]
    # Split data
    X_train, X_test, y_train, y_test = skl.model_selection.train_test_split(X, y, test_size=0.3)

Code in question:

    # Perform logistic regression and get p values
    logit_model = sm.Logit(y_train, X_train.astype(float))
    result = logit_model.fit()

This is the firth_regression I tried instead which got me a memory error:

# For the firth_regression
import sys
import warnings
import math
import statsmodels
from scipy import stats
import statsmodels.formula.api as smf


def firth_likelihood(beta, logit):
    return -(logit.loglike(beta) + 0.5*np.log(np.linalg.det(-logit.hessian(beta))))

step_limit=1000
convergence_limit=0.0001

logit_model = smf.Logit(y_train, X_train.astype(float))

start_vec = np.zeros(X.shape[1])

beta_iterations = []
beta_iterations.append(start_vec)
for i in range(0, step_limit):
    pi = logit_model.predict(beta_iterations[i])
    W = np.diagflat(np.multiply(pi, 1-pi))
    var_covar_mat = np.linalg.pinv(-logit_model.hessian(beta_iterations[i]))

    # build hat matrix
    rootW = np.sqrt(W)
    H = np.dot(np.transpose(X_train), np.transpose(rootW))
    H = np.matmul(var_covar_mat, H)
    H = np.matmul(np.dot(rootW, X), H)

    # penalised score
    U = np.matmul(np.transpose(X_train), y - pi + np.multiply(np.diagonal(H), 0.5 - pi))
    new_beta = beta_iterations[i] + np.matmul(var_covar_mat, U)

    # step halving
    j = 0
    while firth_likelihood(new_beta, logit_model) > firth_likelihood(beta_iterations[i], logit_model):
        new_beta = beta_iterations[i] + 0.5*(new_beta - beta_iterations[i])
        j = j + 1
        if (j > step_limit):
            sys.stderr.write('Firth regression failed\n')
            None

    beta_iterations.append(new_beta)
    if i > 0 and (np.linalg.norm(beta_iterations[i] - beta_iterations[i-1]) < convergence_limit):
        break

return_fit = None
if np.linalg.norm(beta_iterations[i] - beta_iterations[i-1]) >= convergence_limit:
    sys.stderr.write('Firth regression failed\n')
else:
# Calculate stats
    fitll = -firth_likelihood(beta_iterations[-1], logit_model)
    intercept = beta_iterations[-1][0]
    beta = beta_iterations[-1][1:].tolist()
    bse = np.sqrt(np.diagonal(-logit_model.hessian(beta_iterations[-1])))

    return_fit = intercept, beta, bse, fitll
#print(return_fit)

Answer 1

I fixed my problem by changing the default method in the logit regression to method ='bfgs'.

result = logit_model.fit(method = 'bfgs')