Python's itertools product memory consumption

Question

The documentation says that the cartesian product function

the actual implementation does not build up intermediate results in memory.

How can that be possible with generators? Can somebody show me an example with a bounded memory consumption for 2 generators?

Answer 1

Looking at the module's source code, itertools.product() actually converts every argument to a tuple:

// product_new() in itertoolsmodule.c
for (i=0; i < nargs ; ++i) {
    PyObject *item = PyTuple_GET_ITEM(args, i);
    PyObject *pool = PySequence_Tuple(item); //<==== Call tuple(arg)
    if (pool == NULL)
        goto error;
    PyTuple_SET_ITEM(pools, i, pool);
    indices[i] = 0;
}

In other words, itertools.product()'s memory consumption appears to be linear in the size of the input arguments.