Why doesn't my custom lightgbm loss decrease?

Question

I have implemented a custom mean average error (MAE) loss in lightgbm. The gradient is nonzero, but the loss stays constant. How could that be?

My implementation:

def abs_obj(preds, dtrain):
    y_true = dtrain.get_label()
    a = preds - y_true
    grad = np.sign(a)
    hess = np.zeros(len(a))
    return grad, hess

def abs_eval(preds, dtrain):
    y_true = dtrain.get_label()
    loss = np.abs(preds - y_true).sum()
    return "error", loss, False

A minimal reproducible example: the loss stays constant.

dtrain = pd.DataFrame({'x':np.random.rand(100),
                      'y':np.random.rand(100)})
ytrain = dtrain.x + 2 * dtrain.y
dval = dtrain
yval = ytrain
lgb_train = lgb.Dataset(dtrain, ytrain)
lgb_valid = lgb.Dataset(dval, yval)
params = {'objective':None,
         'learning_rate':30,
         'num_leaves':33}
clf = lgb.train(params,
               lgb_train,
               valid_sets=[lgb_valid],
               num_boost_round=10,
               verbose_eval=1,
               fobj=abs_obj,
               feval=abs_eval)

Answer 1

For a custom loss in lightgbm, you need a twice differentiable function with a positive second derivative.

To speed up their algorithm, lightgbm uses Newton's approximation to find the optimal leaf value:

y = - L' / L''

(See this blogpost for details).

When the second derivative is zero or the function is not twice differentiable, this approximation is very wrong. Lightgbm has built-in objective functions which do not fit this criterion, such as MAE. For these functions they have different, special implementations.