Any better way to calculate product of multiple lists?

Question

I need to get all possible combinations based on n-number of input lists and do some stuff to them.

current code example:

import itertools

# example inputs
list_small = [1, 2, 3]
list_medium = [444, 666, 242]
list_huge = [1680, 7559, 5573, 43658, 530, 11772, 284, 50078, 783, 37809, 6740, 37765, 74492, 50078, 783, 37809, 6740, 37765, 74492]

# out of the input list, I need to generate all numbers from 0 to the current list element
# e.g. if I have 6, I need to get [0, 1, 2, 3, 4, 5, 6]
# if I get a list [1, 2, 3], the output will be [[0, 1], [0, 1, 2], [0, 1, 2, 3]]
# I achieved this by doing it with xrange: [x for x in xrange(0, current_list_element + 1)]
# after that, I need to generate all possible combinations using the generated lists
# I managed to do this by using itertools.product()

# print this to get all possible combinations
# print list(itertools.product(*[[x for x in xrange(0, current_list_element + 1)] for current_list_element in list_medium]))

cumulative_sum = 0
for current_combination in itertools.product(*[[x for x in xrange(0, current_list_element + 1)] for current_list_element in list_medium]):
    # now I need to do some calculations to the current combination
    # e.g. get sum of all combinations, this is just an example
    cumulative_sum += sum(current_combination)

    # another example
    # get XOR sum of current combination, more at https://en.wikipedia.org/wiki/Exclusive_or
    print reduce(operator.xor, current_combination, 0)

# runs fast for list_small, then takes some time for list_medium and then takes ages for list_huge
print cumulative_sum

This works fine for smaller lists, but takes infinity for larger lists / or throws Runtime Error. Is there any better way to do this? Better way to get all combinations? Or am I using xrange in some wrong way?

I tried this with Python 2.7 and Pypy 2.

EDIT: thanks to @famagusta I got rid of xrange, but the problem still remains

import itertools

# example inputs
list_small = [1, 2, 3]
list_medium = [444, 666, 242]
list_huge = [1680, 7559, 5573, 43658, 530, 11772, 284, 50078, 783, 37809, 6740, 37765, 74492, 50078, 783, 37809, 6740, 37765, 74492]

max_element = max(get_input_stones)
combo_list = range(0, max_element + 1)

cumulative_sum = 0
for current_combination in itertools.product(*combo_list):
    # now I need to do some calculations to the current combination
    # e.g. get sum of all combinations, this is just an example
    cumulative_sum += sum(current_combination)

    # another example
    # get XOR sum of current combination, more at https://en.wikipedia.org/wiki/Exclusive_or
    print reduce(operator.xor, current_combination, 0)

# runs fast for list_small, then takes some time for list_medium and then takes ages for list_huge
print cumulative_sum

Answer 1

Generating such nested lists could get you into trouble with memory limitations. Instead of repeatedly generating sublists, you can use just one super list generated from the largest number in the list. Just store the indices where smaller elements would have stopped.

For e.g., [1, 6, 10] - [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 6, 10]

The second list tells you where to stop in the first list to extract sublists of interest for computation

This should save you some space.

list_small = [1, 2, 3]
list_medium = [444, 666, 242]
list_huge = [1680, 7559, 5573, 43658, 530, 11772, 284, 50078, 783, 37809, 6740, 37765, 74492, 50078, 783, 37809, 6740, 37765, 74492]

max_element = max(list_huge)   # being lazy here - write a max function
combo_list = range(0, max_element + 1)  # xrange does not support slicing

cumulative_sum = 0
for element in list_huge:
    cumulative_sum += sum(combo_list[:element])

print(cumulative_sum)