Search code examples
matlabcombinationscombinatoricslarge-data

"Out of memory" with combvec function for a large number of vectors MATLAB

So I have a set of data which is similar to this:

index |Parameter A| Parameter B

1|1|3

2|1|5

3|1|1

4|2|12

5|2|15

6|2|41

7|3|22

8|3|14

9|3|9

I need to calculate all the possible combinations of parameters B with a different parameter A i.e. (1)3-(2)12-(3)9 and obtain the indices of said combinations, in this case 1-4-9. The way I solved this was defining the index at which parameter A changes (in this case 3 and 6, I should point out it is not periodic) and use the combvec function in a loop:

  • combvec(combvec(1:3,4:6),6:9);

An then obtain a matrix which has all the combination indices in each column, which I can then use to obtain combinations of parameter B. The thing is the list is too big now hence the Out of memory problem.

The objective here is to calculate for each sequence of parameters B obtain a certain result - X - and then choose the combination that minimises said result.

I thought I could divide the list in two but the problem is since X is dependent on all A parameters the minimum of both groups will not necessarily be the global minimum, so I am not so sure.

The other alternative I thought was try obtain each combination in a loop and immediately calculate X and only store X if it was smaller than the previous iteration. This solution would solve my problem of storage but I am not so sure how to do this without a lot of nested loops because I would not be able to use combvec.

I would appreciate if you guys had any ideas on how to solve this or how to come about my suggestions avoiding the problems I mentioned.

My thanks in advance.

Solution

  • I posted this question on reddit and got and answer to it so I am going to copy paste the answer. The reasoning is quite similar to what the user beaker suggested:

    A = 1:4;
B = 5:8;
C = 9:12;
D = 13:16;

AB = combvec(A,B); % 2x16
CD = combvec(C,D); % 2x16
ABCD = combvec(AB,CD); % 4x256
ABCD = combvec(combvec(combvec(A,B),C),D); % 4x256, same as above

    ABCD nested combination gets big REALLY fast. Instead, we can loop.

    looped = [];
for idx=1:length(CD)
   looped = [looped [AB; repmat(CD(:,1),1,length(AB))]];
end

    For each possibility in CD, combine it with all of the AB possibilities. "looped" is the same result as ABCD. Rather than store each combination in the loop, we could test for minima This approach only needed 64 values in memory instead of 1024.

    Credit to /u/BCPull

    Original Post link: https://www.reddit.com/r/matlab/comments/5mo48r/out_of_memory_problem_with_combvec_function_for_a/

    Unfortunately as the user beaker suggested I need to rethink my approach since exchanging storage for processing time uses a lot of computing time and its not a good idea in general.