How to implement the derivative of Leaky Relu in python?

Question

How would I implement the derivative of Leaky ReLU in Python without using Tensorflow?

Is there a better way than this? I want the function to return a numpy array

def dlrelu(x, alpha=.01):
     # return alpha if x < 0 else 1

     return np.array ([1 if i >= 0 else alpha for i in x])

Thanks in advance for the help

Answer 1

The method you use works, but strictly speaking you are computing the derivative with respect to the loss, or lower layer, so it might be wise to also pass the value from lower layer to compute the derivative (dl/dx).

Anyway, you can avoid using the loop which is more efficient for large x. This is one way to do it:

def dlrelu(x, alpha=0.01):
  dx = np.ones_like(x)
  dx[x < 0] = alpha
  return dx

If you passed the error from lower layer, it looks like this:

def dlrelu(dl, x, alpha=0.01):
  """ dl and x have same shape. """
  dx = np.ones_like(x)
  dx[x < 0] = alpha
  return dx*dl