taking the gradient of huber loss in theano

Question

I have two functions that are suppose to produce equal results: f1(x,theta)=f2(x,theta).

Given input x, I need to find the parameters theta that makes this equality hold as well as possible.

Initially I was thinking of using squared loss and minimizing (f1(x,theta)-f2(x,theta))^2 and solving via SGD.

However I was thinking of making the loss more precise and using huber (or absolute loss) of the difference. Huber loss is a piecewise function (ie initially it is quadratic and then it changes into a linear function).

How can I take the gradient of my huber loss in theano?

Answer 1

A pretty simple implementation of huber loss in theano can be found here

Here is a code snippet

import theano.tensor as T
delta = 0.1
def huber(target, output):
    d = target - output
    a = .5 * d**2
    b = delta * (abs(d) - delta / 2.)
    l = T.switch(abs(d) <= delta, a, b)
    return l.sum()

The function huber will return a symbolic representation of the loss which you can then plug in theano.tensor.grad to get the gradient and use it to minimize using SGD