Search code examples
python-3.xscipyfloating-pointscipy-optimizescipy-optimize-minimize

Is ftol and gtol needed in scipy.optimize.minimize is it proper to give it a very low value?

I am using the bounded limited memory BFGS optimizer to minimize the value of a black box function. I have simulated randomly many input parameter combinations and realized that the ftol and gtol parameter is only in the way, and it doesn't contribute anything to decreasing the value of my function (there is a positive correlation between the outputs and the random inputs for ftol and gtol, so the smaller the better). So I set both to 1E-18 and focused on configuring the other parameters, thus the exit message CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH meaning that the entire optimization depended on the correct value for eps I guess.

Then I set both ftol and gtol to 1E-20 to not stand in the way, but then I started getting sub-optimal results.

So my optimizer is:

scipy.optimize.minimize(function, x0=guess.flatten(), method='L-BFGS-B', bounds=bounds, options={ 'maxcor': maxcor, 'ftol': 1E-20, 'gtol': 1E-20, 'eps': eps, 'maxfun': maxrounds, 'maxiter': maxrounds, 'maxls': maxls})

So I set it to 1E-20 and the other values are fed randomly. The average output for a bigger sample is smaller with 1E-20 than with 1E-18, I don't understand why, they are supposed to be very small numbers that are negligible. I also started getting exit messages like CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL which I don't know how it was possible for such a small tolerance. So I have the following questions:

1) Is it worth setting ftol and gtol to such low values as 1E-20?

2) Should I set tol the (external tolerance value) if ftol and gtol is already set? I don't want it to exit ahead of time. Or does tol as an exit threshold get disabled if gtol and ftol is enabled?

3) Is it possible that Scipy, Numpy or Python3 itself can't handle floating point values with 20 decimals. I noticed Python mostly prints 18 digits for floats, so could the problem be that I put too many digits. If so then what is the maximum number of digits handled by scipy.optimize? (Scipy v1.4.1 | Numpy v1.18.1 | Python 3.5.3)

Solution

  • The tolerance that you are setting is not really achievable due to round-off errors. You can read here more about the floating point precision in Python. You should choose a small, but sensible number for gtol and ftol, usually 1e-6 - 1e-8 works.