Implementing the Square Non-linearity (SQNL) activation function in Keras

Question

I have been trying to implement the square non-linearity activation function function as a custom activation function for a keras model. It's the 10'th function on this list https://en.wikipedia.org/wiki/Activation_function.

I tried using the keras backend but i got nowhere with the multiple if else statements i require so i also tried using the following:

import tensorflow as tf
def square_nonlin(x):
    orig = x
    x = tf.where(orig >2.0, (tf.ones_like(x)) , x)
    x = tf.where(0.0 <= orig <=2.0, (x - tf.math.square(x)/4), x)
    x = tf.where(-2.0 <= orig < 0, (x + tf.math.square(x)/4), x)
    return tf.where(orig < -2.0, -1, x)

As you can see there's 4 different clauses i need to evaluate. But when i try to compile the Keras model i still get the error:

Using a `tf.Tensor` as a Python `bool` is not allowed

Could anyone help me to get this working in Keras? Thanks a lot.

Answer 1

I've just started a week ago digging into tensorflow and am actively playing around with different activation functions. I think I know what two of your problems are. In your second and third assignments you have compound conditionals you need to put them in under tf.logical_and. The other problem you have is that the last tf.where on the return line returns a -1 which is not a vector, which tensorflow expects. I haven't tried the function with Keras, but in my "activation function" tester this code works.

def square_nonlin(x):
    orig = x
    x = tf.where(orig >2.0, (tf.ones_like(x)) , x)
    x = tf.where(tf.logical_and(0.0 <= orig, orig <=2.0), (x - tf.math.square(x)/4.), x)
    x = tf.where(tf.logical_and(-2.0 <= orig, orig < 0), (x + tf.math.square(x)/4.), x)
    return tf.where(orig < -2.0, 0*x-1.0, x)

As I said I'm new at this so to "vectorize" -1, I multiplied the x vector by 0 and subtracted -1 which produces a array filled with -1 of the right shape. Perhaps one of the more seasoned tensorflow practioners can suggest the proper way to do that.

Hope this helps.

BTW, tf.greater is equivlent to tf.__gt__ which means that orig > 2.0 expands under the covers in python to tf.greater(orig, 2.0).

Just a follow up. I tried it with the MNIST demo in Keras and the activation function works as coded above.

UPDATE:

The less hacky way to "vectorize" -1 is to use the tf.ones_like function

so replace the last line with

   return tf.where(orig < -2.0, -tf.ones_like(x), x)

for a cleaner solution