Search code examples
pythonoptimizationscipyscipy-optimizescipy-optimize-minimize

SciPy "Successfully" finding incorrect optimal solution and "Unsuccessfully" finding optimal solution (Portfolio Construction)


I'm building a trading bot and I'm trying to implement an optimiser to maximise alpha while adhering to certain constraints.

My variable is a vector containing the weights of the securities in the portfolio. I also have a vector that contains the respective alpha scores for each security. My objective function is -sum(weights*alphas) (to minimise negative alpha). The constraints that I have are:

  • Min and max weight of stock in the portfolio
  • Min trade (as a percentage of total portfolio value - there is a fixed cost to trade so I don't want to trade if it's just a couple of basis points change)
  • Max turnover (maximum total trade value as a percentage of total portfolio value)
  • The sum of the absolute weights must add up to 1
  • The sum of the weights must equal either 0 (for long/short) or 1 (for long only)

I have created a class below which implements this using scipy.optimize.minimize:

class Optimiser:

    def __init__(self, initial_portfolio, turnover, min_trade, max_wt, longshort=True):
        self.symbols = initial_portfolio.index.to_numpy()
        self.init_wt = initial_portfolio['weight'].to_numpy()
        self.alpha = initial_portfolio['alpha'].to_numpy()
        self.longshort = longshort
        self.turnover = turnover
        self.min_trade = self.init_wt.copy()
        self.set_min_trade(min_trade)
        self.max_wt = max_wt
        if self.longshort:
            self.wt_sum = 0
            self.abs_wt_sum = 1
        else:
            self.wt_sum = 1
            self.abs_wt_sum = 1

    def set_min_trade(self, min_trade):
        for i in range(len(self.init_wt)):
            if abs(self.init_wt[i]) > min_trade:
                self.min_trade[i] = 0.1

    def optimise(self):
        wt_bounds = self.get_stock_wt_bounds()
        constraints = self.get_constraints()
        result = minimize(
            fun=self.minimise_negative_alpha,
            x0=self.init_wt,
            bounds=wt_bounds,
            constraints=constraints,
            options={
                'disp': True,
            }
        )
        return result

    def minimise_negative_alpha(self, opt_wt):
        return -sum(opt_wt * self.alpha)

    def get_stock_wt_bounds(self):
        if self.longshort:
            return tuple((-self.max_wt, self.max_wt) for s in self.init_wt)
        else:
            return tuple((0, self.max_wt) for i in range(len(self.init_wt)))

    def get_constraints(self):
        min_trade = {'type': 'ineq', 'fun': self.min_trade_fn}
        turnover = {'type': 'ineq', 'fun': self.turnover_fn}
        wt_sum = {'type': 'eq', 'fun': self.wt_sum_fn}
        abs_wt_sum = {'type': 'eq', 'fun': self.abs_wt_sum_fn}
        return turnover, wt_sum, abs_wt_sum

    def min_trade_fn(self, opt_wt):
        return self.min_trade - abs(opt_wt - self.init_wt)

    def turnover_fn(self, opt_wt):
        return sum(abs(opt_wt - self.init_wt)) - self.turnover*2

    def wt_sum_fn(self, opt_wt):
        return sum(opt_wt)

    def abs_wt_sum_fn(self, opt_wt):
        return sum(abs(opt_wt)) - self.abs_wt_sum

As you can see I am not using the min_trade constraint and I'll touch upon this later in the question.

Here are two examples I'm passing into it (these examples only contain 4 stocks and in the proper implementation I am looking to pass arrays of 50-100 securities):

a)

def run_optimisation():
    initial_portfolio = pd.DataFrame({
        'symbol': ['AAPL', 'MSFT', 'GOOGL', 'TSLA'],
        'weight': [-0.3, -0.2, 0.45, 0.05],
        'alpha': [-0.2, -0.3, 0.25, 0],
    }).set_index('symbol')

    opt = Optimiser(initial_portfolio, turnover=0.3, min_trade=0.1, max_wt=0.4)
    result = opt.optimise()

b)

def run_optimisation():
    initial_portfolio = pd.DataFrame({
        'symbol': ['AAPL', 'MSFT', 'GOOGL', 'TSLA'],
        'weight': [-0.25, -0.25, 0.25, 0.25],
        'alpha': [-0.2, -0.3, 0.25, 0],
    }).set_index('symbol')

    opt = Optimiser(initial_portfolio, turnover=0.3, min_trade=0.1, max_wt=0.4)
    result = opt.optimise()

The result that I am getting from a) for this long-short example is: [-0.1, -0.4, 0.25, 0.25] which is obviously not optimal [-0.1, -0.4, 0.4, 0.1].

I am getting this message:

Optimization terminated successfully.    (Exit mode 0)
            Current function value: -0.20249999999999585
            Iterations: 7
            Function evaluations: 42
            Gradient evaluations: 7

It's saying it successfully found the minimum... It's like it's trying to max out the turnover constraint. Is this because the initial weights do not adhere to the constraints? If so, how can I modify that as ideally I would like to pass it the current weights of the portfolio as x0.

In b) I am getting the optimal solution [-0.1, -0.4, 0.4, 0.1] but I am getting a False for result.success.

I am also getting this message:

Positive directional derivative for linesearch    (Exit mode 8)
            Current function value: -0.23999999999776675
            Iterations: 8
            Function evaluations: 34
            Gradient evaluations: 4

I think this message may mean that it is unable to increase/decrease the objective function much with changes and thus it does not know if it is at the minimum, please correct me if I am wrong. I have tried messing around with the ftol setting to no avail although I am not too sure how to set it optimally.

Is there a way to modify this optimiser so that a) it achieves the optimal solution and produces the correct status accordingly and b) can take initial weights that do not conform to the constraints? The hope for the future is to also include sector and industry constraints so that I cannot be over-invested in certain areas.

Also, as a side question (albeit not as important as I would just like to get it working to begin with): How might I implement the min trade constraint? I would like it so that either the stock is not traded at all or it has a trade value over this amount or trading away it's full value (to zero weight if it's less than the min_trade weight in the portfolio).

As you can see this is a very long question but I would really appreciate any help, guidance or answers you could provide for this problem! Please ask for any clarification or extra information as this has taken me a long time to cobble together and I am probably not explaining something well or missing something. Thank you!


Solution

  • Following on from Sascha's comments above I wanted to post the correct implementation of this problem in cvxpy:

    import cvxpy as cv
    
    
    class Optimiser:
    
        def __init__(self, initial_portfolio, turnover, max_wt, longshort=True):
            self.symbols = initial_portfolio.index.to_numpy()
            self.init_wt = initial_portfolio['weight'].to_numpy()
            self.opt_wt = cv.Variable(self.init_wt.shape)
            self.alpha = initial_portfolio['alpha'].to_numpy()
            self.longshort = longshort
            self.turnover = turnover
            self.max_wt = max_wt
            if self.longshort:
                self.min_wt = -self.max_wt
                self.net_exposure = 0
                self.gross_exposure = 1
            else:
                self.min_wt = 0
                self.net_exposure = 1
                self.gross_exposure = 1
    
        def optimise(self):
            constraints = self.get_constraints()
            optimisation = cv.Problem(cv.Maximize(cv.sum(self.opt_wt*self.alpha)), constraints)
            optimisation.solve()
            if optimisation.status == 'optimal':
                print('Optimal solution found')
            else:
                print('Optimal solution not found')
            return optimisation.solution.primal_vars
    
        def get_constraints(self):
            min_wt = self.opt_wt >= self.min_wt
            max_wt = self.opt_wt <= self.max_wt
            turnover = cv.sum(cv.abs(self.opt_wt-self.init_wt)) <= self.turnover*2
            net_exposure = cv.sum(self.opt_wt) == self.net_exposure
            gross_exposure = cv.sum(cv.abs(self.opt_wt)) <= self.gross_exposure
            return [min_wt, max_wt, turnover, net_exposure, gross_exposure]
    

    Many thanks to Sascha for the help and guidance.