Search code examples
c#multithreadinggarbage-collectionblockingcollection

Garbage collector not able to reclaim memory resources


I am trying to test how the heavy objects are handled inside the .net BlockingCollection.

I am getting the heavy objects using some API, and want to process it in multiple thread. In order to do that, I have one thread that reads the heavy objects and push them into multiple threads where each thread has its own blocking collection and each of those thread takes the object out of the collection and process. I am expecting that when the object is removed from all the collection wherever it is sitting, GC should be able to clean it up. That clean up is not happening and my program is going out of memory.

Calling GC.Collect() in between is helping me to complete the process but it has significant performance hit which I cannot afford to take.

My only question is why Garbage Collector is not able to free up the resources here even if the objects are out of scope.

public class DummyProcessor
{
    List<BlockingCollection<object>> listOfBlockingCollection = null;

    void ProcessCollection(object blockingCollection)
    {
        BlockingCollection<object> collection = (BlockingCollection<object>)blockingCollection;

        while (collection.IsCompleted == false)
        {
            object heavyObject = collection.Take();

            CallExternalProcess(heavyObject);
        }
    }

    private void CallExternalProcess(object heavyObject)
    {
        throw new NotImplementedException();
    }

    public void Analyze(object heavyObject)
    {
        if (listOfBlockingCollection == null)
        {
            listOfBlockingCollection = new List<BlockingCollection<object>>();

            for (int i = 0; i < 25; i++)
            {
                BlockingCollection<object> coll = new BlockingCollection<object>();

                listOfBlockingCollection.Add(coll);

                Thread pt = new Thread(new ParameterizedThreadStart(ProcessCollection));

                pt.Start(coll);
            }
        }

        for (int i = 0; i < 25; i++)
        {
            listOfBlockingCollection[i].Add(heavyObject);
        }
    }
}

Solution

  • After running the application through dotMemory, I found that the heavy objects were being produced too fast that they were getting accumulated into various BlockingCollection in large number beyond the computer memory.
    My explicit call to GC.Collect was working not because it was releasing the used heavy object but it was adding kind of pause to heavy object production giving more time to consumer threads to clear what is already in the BlockingCollection.
    So I had to introduce wait between the heavy object producer and consumer so that I don't overuse available memory. For that I am using the AutoResetEvent. I am calling AutoResetEvent.WaitOne after the collection size reaches some range and as soon as it comes below the range I am calling AutoResetEvent.Set to run the producer again.

    Thank you everyone for your input on this.