Chainer: custome sigmoid activation function

Question

I want to implement the following sigmoid function with a custom slope parameter k.

y = f(x)= 1/ ( 1+exp(-1*k*x))
gradient gy = k * f(x)*(1-f(x))

I want to use this in my autoencoder. How do I implement this in Chainer?

Answer 1

If k is constant (i.e., a hyperparameter), F.sigmoid(k * x) should just work.

If k is a parameter that should be learned in the same way as other weights, you may want to subclass a link like L.PReLU, and use it just like other links, e.g. L.Linear and L.Convolution2D. You can still implement the forward method of the link like the above simple expression.