Sigmoid function returns 1 for large positive inputs

Question

I wrote the following function in Python to calculate sigmoid function of a scalar, vector or matrix.

def sigmoid(z):
    sig = 1.0/(1.0 + np.exp(-z))
    return sig

For relatively large positive values of z, e^-z returns a very small value close to zero (0) and hence the value of sig is rounded to 1. My final objective is to determine cost function of logistic regression algorithm. Since sigmoid returns exactly 1, log(1-1) return 'nan'. How can I solve the problem in such a way that my sigmoid() function will return the proper value and not round the e^-z to 0?

When I normalized the input features using mean and standard deviation, it worked fine. But is there a way to make it work with larger values of z?

Also, I tried the same on Matlab and it worked fine without normalization.

>>> Z = np.array([[60, 100],[20, 80]])
>>> Z
array([[ 60, 100],
       [ 20,  80]])
>>> np.exp(-Z)
array([[8.75651076e-27, 3.72007598e-44],
       [2.06115362e-09, 1.80485139e-35]])
>>> 1.0/(1.0 + np.exp(-Z))
array([[1., 1.],
       [1., 1.]])

Answer 1

I overcame this issue by wrapping the sigmoid function with np.minimum & np.maximum:

def sigmoid(x):
    sig = 1 / (1 + np.exp(-x))     # Define sigmoid function
    sig = np.minimum(sig, 0.9999)  # Set upper bound
    sig = np.maximum(sig, 0.0001)  # Set lower bound
    return sig

As a result my losses started looking like this

But both training and test losses converged well & I received ~90% of accuracy on a tiny dataset using logistic regression.