Scipy.optimize.minimize is not giving the minimum value even though it sees that value

Question

I am using scipy.optimize.minimize to find optimal parameters for my objective function.

My code :

import numpy as np
from scipy.optimize import minimize
from scipy.optimize import Bounds

bounds = Bounds([26,26,8,6,400,100,0,25,2],[36,38,28,28,1800,800,100,50,7])

energy_history = []
x_values = []

def objective(x):
    x_trail = x.reshape(1,-1)
    x_trail = sc_X.transform(x_trail)
    y_trail = regressorSVR.predict(x_trail)
    y_trail = y_trail.reshape(1,-1)
    y_trail = sc_Y.inverse_transform(y_trail)
    return y_trail[0]


def callback(x,y):
    fobj = objective(x)
    energy_history.append(fobj)
    x_values.append(x)

x0 = np.array([26,28,15,7,400,377,40,43,4.3])
res = minimize(objective, x0, method='trust-constr',
               options={'verbose': 1}, bounds=bounds,callback=callback)

optimal_values = res.x
energy = res.fun

With the initial values given, the minimized value(res.fun) that I get is -7.1. I am creating a list(energy_history) to see how it is reaching this value. I see some values which are less than -7.1 in that list, but still, why is -7.1 being returned as the minimal value.

There are multiple times where objective function reached a value of -21, but why is still -7 being returned as a minimum ?

Answer 1

If we take a look at the scipy.optimization documentation we can see that scipy.optimize.minimize is listed under local optimization. The main problem is that your problem is non-convex and thus scipy.optimize.minimize cannot guarantee the proper convergence. As it's also very much non-differentiable, many algorithms won't be suited at all.

scipy.optimize does provide some global optimization algorithms though that can be found on the documentation page under global optimization, namely basinhopping, brute, and differential_evolution. Look at this answer for some short explanation.

Basically you can try brute first, just to see any systematic problems. It's basically a brute force solution and will be slow, but find your minimum. The more sophisticated method would be using differential_evolution. Since your function isn't really smooth, basinhopping might not work, but it's still worth a shot and would probably converge the fastest.