Custom convergence criterion in scipy optimise

Question

I am optimising a function using scipy.optimize in the following manner:

yEst=minimize(myFunction, y0, method='L-BFGS-B', tol=1e-6).x

My problem is that I don't want to stop simply when the tolerance is less than a value (e.g. if on the nth iteration stop is |y_n - y_(n-1)|<tol). Instead I have a slightly more complex function of y_n and y_(n-1), say tolFun, and I want to stop when tolFun(y_n, y_(n-1))<tol.

To give more detail my tolerance function is the following. It partitions y into chunks and then checks if any of the individual partitions have a norm difference within tolerance and, if any do, then the minimisation should stop.

# Takes in current and previous iteration values and a pre-specified fixed scalar r.
def tolFun(yprev,ycurr,r):

  # The minimum norm so far (initialized to a big value)
  minnorm = 5000

  for i in np.arange(r):

    # Work out the norm of the ith partition/block of entries
    norm = np.linalg.norm(yprev[np.arange(r)+i*r],ycurr[np.arange(r)+i*r])

    # Update minimum norm
    minnorm = np.min(norm, minnorm)

  return(minnorm)

My question is similar to this question here but differs in the fact that this user needed only the current iterations value of y, whereas my custom tolerance function needs both the current iterations value of y and the previous value. Does anyone know how I could do this?

Answer 1

You cannot do directly what you want since the callback function receives only the current parameter vector. To solve your problem you can modify second solution from https://stackoverflow.com/a/30365576/8033585 (which I prefer to the first solution that uses global) in the following way or so:

class Callback:
    def __init__(self, tolfun, tol=1e-8):
        self._tolf = tolfun
        self._tol = tol
        self._xk_prev = None

    def __call__(self, xk):
        if self._xk_prev is not None and self._tolf(xk, self._xk_prev) < self._tol:
            return True

        self._xk_prev = xk
        return False

cb = Callback(tolfun=tolFun, tol=tol)  # set tol here to control convergence
yEst = minimize(myFunction, y0, method='L-BFGS-B', tol=0, callback=cb)

or

yEst = optimize.minimize(
    myFunction, y0, method='L-BFGS-B',
    callback=cb, options={'gtol': 0, 'ftol': 0}
)

You can find available options for a solver/method using:

optimize.show_options('minimize', 'L-BFGS-B')