machine learning and optimizing scipy

Question

I have coding machine learning and for optimizing my cost function i used scipy.optimize.minimum for it and scipy doesn't return right answer.so what should i do?

code:

data1 = pd.read_csv('ex2data1.txt', header = None, names = 
['exam1','exam2', 'y'])
data1['ones'] = pd.Series(np.ones(100), dtype = int)
data1 = data1[['ones', 'exam1', 'exam2', 'y']]
X = np.matrix(data1.iloc[:, 0:3])
y = np.matrix(data1.iloc[:, 3:])  

def gFunction(z):
    return sc.special.expit(-z)  

def hFunction(theta, X):
    theta = np.matrix(theta).T
    h = np.matrix(gFunction(X.dot(theta)))
    return h  

def costFunction(theta, X, y):
    m = y.size
    h = hFunction(theta, X).T
    j = (-1 / m) * (np.dot(np.log(h), y) + np.dot(np.log(1-h), (1-y)))
    return j

def gradientDescent(theta, X, y):
    theta = np.matrix(theta)
    m = y.size
    h = hFunction(theta, X)
    gradient = (1 / m) * X.T.dot(h - y)
    return gradient.flatten()  

initial_theta = np.zeros(X.shape[1])
cost = costFunction(initial_theta, X, y)
grad = gradientDescent(initial_theta, X, y)
print('Cost: \n', cost)
print('Grad: \n', grad)
Cost: 
   [[ 0.69314718]]
Grad: 
   [[ -0.1        -12.00921659 -11.26284221]]

def optimizer(costFunction, theta, X, y, gradientDescent):
    optimum = sc.optimize.minimize(costFunction, theta, args = (X, y), 
     method = None, jac = gradientDescent, options={'maxiter':400})
    return optimum  

 Warning: Desired error not necessarily achieved due to precision loss.
      Current function value: 0.693147
      Iterations: 0
      Function evaluations: 106
      Gradient evaluations: 94
Out[46]:
     fun: matrix([[ 0.69314718]])
hess_inv: array([[1, 0, 0],
                 [0, 1, 0],
                 [0, 0, 1]])
    jac: matrix([[ -0.1       , -12.00921659, -11.26284221]])
message: 'Desired error not necessarily achieved due to precision loss.'
   nfev: 106
    nit: 0
   njev: 94
 status: 2
success: False
      x: array([ 0.,  0.,  0.])

this is the message that says success False i have done everything right i don't know what's happen

Answer 1

It's hard to debug something like this when:

• code is not reproducible because of external data
• question does not even try to explain what is optimized here

There are some strange design-decisions:

• use of np.matrix -> do use np.array!
• don't call the jacobian gradientDescent

And then in regards to your observation:

Iterations: 0
Function evaluations: 106
Gradient evaluations: 94

zero iterations while doing so many function-evaluations is a very bad sign. Something is very broken. Probably line-search is going crazy above, but that's just a guess.

Now what's broken?:

• your jacobian is definitely broken!
  • i did not check the math, but:
    • your jacobian-shape is dependent on the number of samples when number of variables is fixed -> no ! that does not make sense!

Steps to do:

• run with jac=False
  • If working: your cost-fuc looks ok
  • If not working: your trouble probably (no proof) starts even there
• repair the jacobian!
• check the jacobian against check_grad

I wonder why you don't get any shape errors here. I do, when trying to mimic your input shapes and playing around with sample-size!