Keras custom loss function huber

Question

I wrote huber loss using Keras backend functions and this works well:

def huber_loss(y_true, y_pred, clip_delta=1.0):
    error = y_true - y_pred
    cond  = K.abs(error) < clip_delta
         
    squared_loss = 0.5 * K.square(error)
    linear_loss  = clip_delta * (K.abs(error) - 0.5 * clip_delta)

    return tf_where(cond, squared_loss, linear_loss)

But I need a more complex loss function:

1. if error <= A, use squared_loss
2. if A <= error < B, use linear_loss
3. if error >= B, used sqrt_loss

I wrote smth like this:

def best_loss(y_true, y_pred, A, B):
    error = K.abs(y_true - y_pred)
    cond  = error <= A
    cond2 = tf_logical_and(A < error, error <= B)

    squared_loss = 0.5 * K.square(error)
    linear_loss  = A * (error - 0.5 * A)
    sqrt_loss = A * np.sqrt(B) * K.sqrt(error) - 0.5 * A**2

    return tf_where(cond, squared_loss, tf_where(cond2, linear_loss, sqrt_loss))

But it does not work, the model with this loss function does not converge, what is the bug?

Answer 1

I like to debug custom functions by graphing them using a program like Desmos. I graphed the Huber Loss using your implementation and it looks like how it should.

When I tried to graph your second function it looks like a valid loss function too. The only problem is when B is less than A. If your value of B is greater than A then it shouldn't be a problem with your loss function. If that isn't the issue then you can try switching the subtraction between the targets and outputs as I'm not familiar with how tensorflow handles differentiation but the order affects the direction of the gradients.