.Except / Yield return out of memory exception

Question

I'm running out of memory using Linq's Except statement.

Example:

var numbers = Enumerable.Range(1, 10000000);
int i=0;
while (numbers.Any())
{
    numbers = numbers.Except(new List<int> {i++});
}

I decompiled the method, this does the same and also gives an out of memory exception.

var numbers = Enumerable.Range(1, 10000000);
int i=0;
while (numbers.Any())
{
   numbers = CustomExcept(numbers, new List<int>{i++});
}
private IEnumerable<int> CustomExcept(IEnumerable<int> numbers, IEnumerable<int> exceptionList)
{
    HashSet<int> set = new HashSet<int>();
    foreach (var i in exceptionList)
    {
        set.Add(i);
    }
    foreach (var i in numbers)
    {
        if (set.Add(i))
        {
           yield return i;
        }
    }
}

So my question is: why does this throw an out of memory exception?

I would expect the Garbage Collector to clean up the unused HashSets.

Answer 1

When you are at i==10 in numbers.Any(), that "numbers" is not a list of numbers from 10 until 10 million, rather it is:

Enumerable.Range(1, 10000000)
   .Except({0})
   .Except({1})
   .Except({2})
   // (etc)
   .Except({9})

And all those "Excepts" have their own hashset that is very much live. So there is nothing to garbage collect.

You will have to add some .ToList()s to really execute those Excepts and give the garbage collector some chance.