I am using R studio and the CVXR package to optimize a portfolio. I have a separate function that optimizes the portfolio with the requested characteristics. The code has worked perfectly, but just now I tried to perform the code with inputs that have been calculated slightly differently.
For some reason, I now get the following error message depending on the inputs:
Error in construct_intermediate_chain(object, candidate_solvers, gp = gp) : Problem does not follow DCP rules.
The error message happens after running the solve-line of the code.As far as I understand, the error is related to the problem not fulfilling all the rules of convex optimization, but I don't understand the reason for this, as the same code has worked fine before.
The inputs are mu (expected returns vector), covareiance matrix, and a vector of previous weights for transaction costs. Here I have an example of inputs which did not work.
Covariance matrix:
Energy Materials Industrials Consumer Discretionary Consumer Staples Health Care Financials IT Communication Services Utilities Real Estate
Energy 0.0030897173 0.002379169 0.0023784794 0.0018105179 9.912975e-04 9.773525e-04 0.002644624 1.591055e-03 1.084336e-03 2.924996e-03 0.0019053206
Materials 0.0023791688 0.005259421 0.0033595945 0.0032327858 1.609355e-03 8.728170e-04 0.003128297 2.843850e-03 1.446845e-03 1.548390e-03 0.0027542633
Industrials 0.0023784794 0.003359594 0.0038704462 0.0030296529 9.267397e-04 1.174425e-03 0.003014640 4.269281e-03 2.002612e-03 2.007975e-03 0.0020530142
Consumer Discretionary 0.0018105179 0.003232786 0.0030296529 0.0043492339 4.066661e-04 6.214639e-04 0.002876835 5.505643e-03 2.923230e-03 9.575217e-04 0.0023722226
Consumer Staples 0.0009912975 0.001609355 0.0009267397 0.0004066661 2.166096e-03 1.377174e-03 0.001026166 -1.284662e-03 1.229443e-05 1.068868e-03 0.0011005523
Health Care 0.0009773525 0.000872817 0.0011744248 0.0006214639 1.377174e-03 2.613830e-03 0.001074507 -7.264024e-05 1.030068e-03 1.529622e-03 0.0008058046
Financials 0.0026446238 0.003128297 0.0030146396 0.0028768352 1.026166e-03 1.074507e-03 0.004320200 2.867530e-03 2.007997e-03 2.831038e-03 0.0021558241
IT 0.0015910549 0.002843850 0.0042692808 0.0055056428 -1.284662e-03 -7.264024e-05 0.002867530 1.549390e-02 4.075777e-03 2.739874e-04 0.0026158084
Communication Services 0.0010843358 0.001446845 0.0020026123 0.0029232304 1.229443e-05 1.030068e-03 0.002007997 4.075777e-03 5.511017e-03 -7.925487e-06 0.0023098958
Utilities 0.0029249959 0.001548390 0.0020079747 0.0009575217 1.068868e-03 1.529622e-03 0.002831038 2.739874e-04 -7.925487e-06 5.636343e-03 0.0020081775
Real Estate 0.0019053206 0.002754263 0.0020530142 0.0023722226 1.100552e-03 8.058046e-04 0.002155824 2.615808e-03 2.309896e-03 2.008177e-03 0.0019311267
mu (expected returns):
Energy Materials Industrials Consumer.Discretionary Consumer.Staples Health.Care Financials Information.Technology Communication.Services Utilities Real.Estate
33 -0.003017939 -0.006933182 -0.009011212 -0.01260558 0.002149697 -0.0008313939 -0.0008236061 -0.03658994 -0.03758739 -0.006729061 -0.008675182
previous_ws:
0.15 0.05 0.05 0.05 0.15 0.15 0.15 0.05 0.05 0.15 0.00
Now, the solver works if I use for example the following covariance matrix, which does not really make sense to me:
Energy Materials Industrials Consumer Discretionary Consumer Staples Health Care Financials IT Communication Services Utilities Real Estate
Energy 0.0029857628 0.0022884237 0.0017807713 0.0013177959 7.975600e-04 0.0007892713 0.001820128 1.553586e-03 0.0011628653 0.0017787287 0.0006446211
Materials 0.0022884237 0.0045078345 0.0027955769 0.0025927008 1.188452e-03 0.0006518281 0.002672642 2.757089e-03 0.0017992508 0.0011660224 0.0015354476
Industrials 0.0017807713 0.0027955769 0.0028927100 0.0025590297 9.795925e-04 0.0009694254 0.002486007 3.526496e-03 0.0021108035 0.0012218375 0.0011190885
Consumer Discretionary 0.0013177959 0.0025927008 0.0025590297 0.0035662042 9.633868e-04 0.0009565825 0.002743525 4.504239e-03 0.0028407505 0.0006230830 0.0012069240
Consumer Staples 0.0007975600 0.0011884525 0.0009795925 0.0009633868 1.943238e-03 0.0012036624 0.001524322 6.529904e-05 0.0007112165 0.0007460728 0.0003728238
Health Care 0.0007892713 0.0006518281 0.0009694254 0.0009565825 1.203662e-03 0.0023788680 0.001521153 1.093596e-03 0.0014957157 0.0007825620 0.0003020803
Financials 0.0018201278 0.0026726415 0.0024860071 0.0027435247 1.524322e-03 0.0015211531 0.003961717 2.808888e-03 0.0023976164 0.0015009125 0.0012940412
IT 0.0015535858 0.0027570891 0.0035264962 0.0045042387 6.529904e-05 0.0010935961 0.002808888 1.147153e-02 0.0047435038 0.0003170311 0.0016258135
Communication Services 0.0011628653 0.0017992508 0.0021108035 0.0028407505 7.112165e-04 0.0014957157 0.002397616 4.743504e-03 0.0062305636 0.0005553472 0.0009298955
Utilities 0.0017787287 0.0011660224 0.0012218375 0.0006230830 7.460728e-04 0.0007825620 0.001500912 3.170311e-04 0.0005553472 0.0033356235 0.0012661852
Real Estate 0.0006446211 0.0015354476 0.0011190885 0.0012069240 3.728238e-04 0.0003020803 0.001294041 1.625813e-03 0.0009298955 0.0012661852 0.0020295775
And here is the code:
optimize_period <- function(mu, covar_matrix.period, previous_ws) {
n.returns <- 0
for (cell in mu){
if (!is.na(cell)) {
n.returns <- n.returns + 1
}
else {
}
}
print(n.returns)
low_limit <- (1/n.returns)/2
high_limit <- (1/n.returns)*1.5
if (n.returns == 0) {
print("return")
return(0)
}
previous_ws <- as.numeric(previous_ws)
previous_ws.edit <- previous_ws[!is.na(previous_ws)]
mu.edit <- mu[!is.na(mu)]
w <- Variable(n.returns)
risk <- quad_form(w, covar_matrix.period)
N <- 20 #Vaihdetaan 30 myöhemmin
constraints <- list(w >= low_limit, w <= high_limit, sum(w) == 1)
gammas <- 10^seq(-3, 4, length.out = N)
tc <- sum(abs(w - previous_ws.edit)) * transaction_cost
mu
ret <- (t(mu.edit) %*% w)
rets <- rep(N, 0)
stds <- rep(N, 0)
tcs <- rep(N, 0)
sharpes <- rep(N, 0)
w_data <- matrix(0, nrow = N, ncol = n.returns)
for (i in seq_along(gammas)) {
gamma <- gammas[i]
objective <- ret - tc - gamma * risk
prob <- Problem(Maximize(objective), constraints)
result <- solve(prob, solver = "ECOS")
rets[i] <- result$getValue(ret)
stds[i] <- result$getValue(sqrt(risk))
tcs[i] <- result$getValue(tc)
w_data[i,] <- result$getValue(w)
sharpes[i] <- rets[i]/stds[i]
}
max.row <- which.max(sharpes)
opt_w <- (w_data[max.row,])
for (cell in 1:length(mu)){
if(is.na(mu[cell])) {
opt_w[cell] <- 0
}
}
print(opt_w)
return(opt_w)
}
What is also confusing me is the fact that the crash happens when the optimization moves from 10 to 11 sectors to optimize, but this change has not had problems elsewhere. This is why I would guess the issue to be related to the Real Estate-column (as that was the new added sector). I tested this, and just changing the real estate column and real estate row fixes the issue, but I obviously have to use the real figures. My hunch is that for some reason the numbers in the covariance matrix happen to be unfortunate for the solver and make the problem seem non-convex, but I don't really have any clue as to why that would be. Any ideas how this could be worked around? Any help would be greatly appreciated!
Your covariance matrix is not positive definite. Note the negative eigenvalue. Here cc is the covariance matrix.
eigen(cc)$values
## [1] 0.0274352963 0.0121339336 0.0048068270 0.0037462092 0.0023325831
## [6] 0.0012573482 0.0009386141 0.0008094488 0.0006894420 0.0005111973
## [11] -0.0004195685
nearPD in Matrix will find the nearest positive definitive matrix although it still gives an eigenvalue that is close to zero so it may or may not be enough.
library(Matrix)
nearPD(cc)$eigenvalues
## [1] 2.743530e-02 1.213393e-02 4.806827e-03 3.746209e-03 2.332583e-03
## [6] 1.257348e-03 9.386141e-04 8.094488e-04 6.894420e-04 5.111973e-04
## [11] 2.743530e-10
We provide cc in easily reproducible form
cc <-
structure(c(0.0030897173, 0.0023791688, 0.0023784794, 0.0018105179,
0.0009912975, 0.0009773525, 0.0026446238, 0.0015910549, 0.0010843358,
0.0029249959, 0.0019053206, 0.002379169, 0.005259421, 0.003359594,
0.003232786, 0.001609355, 0.000872817, 0.003128297, 0.00284385,
0.001446845, 0.00154839, 0.002754263, 0.0023784794, 0.0033595945,
0.0038704462, 0.0030296529, 0.0009267397, 0.0011744248, 0.0030146396,
0.0042692808, 0.0020026123, 0.0020079747, 0.0020530142, 0.0018105179,
0.0032327858, 0.0030296529, 0.0043492339, 0.0004066661, 0.0006214639,
0.0028768352, 0.0055056428, 0.0029232304, 0.0009575217, 0.0023722226,
0.0009912975, 0.001609355, 0.0009267397, 0.0004066661, 0.002166096,
0.001377174, 0.001026166, -0.001284662, 1.229443e-05, 0.001068868,
0.001100552, 0.0009773525, 0.000872817, 0.001174425, 0.0006214639,
0.001377174, 0.00261383, 0.001074507, -7.264024e-05, 0.001030068,
0.001529622, 0.0008058046, 0.002644624, 0.003128297, 0.00301464,
0.002876835, 0.001026166, 0.001074507, 0.0043202, 0.00286753,
0.002007997, 0.002831038, 0.002155824, 0.001591055, 0.00284385,
0.004269281, 0.005505643, -0.001284662, -7.264024e-05, 0.00286753,
0.0154939, 0.004075777, 0.0002739874, 0.002615808, 0.001084336,
0.001446845, 0.002002612, 0.00292323, 1.229443e-05, 0.001030068,
0.002007997, 0.004075777, 0.005511017, -7.925487e-06, 0.002309896,
0.002924996, 0.00154839, 0.002007975, 0.0009575217, 0.001068868,
0.001529622, 0.002831038, 0.0002739874, -7.925487e-06, 0.005636343,
0.002008177, 0.0019053206, 0.0027542633, 0.0020530142, 0.0023722226,
0.0011005523, 0.0008058046, 0.0021558241, 0.0026158084, 0.0023098958,
0.0020081775, 0.0019311267), dim = c(11L, 11L), dimnames = list(
c("Energy", "Materials", "Industrials", "Consumer_Discretionary",
"Consumer_Staples", "Health_Care", "Financials", "IT", "Communication_Services",
"Utilities", "Real_Estate"), c("Energy", "Materials", "Industrials",
"Consumer_Discretionary", "Consumer_Staples", "Health_Care",
"Financials", "IT", "Communication_Services", "Utilities",
"Real_Estate")))