Search code examples
pythonlistmemory-managementmemory-leaksdel

Delete/Reset a list or deque?

I am facing a memory leak, it is happening in a thread. My code structure looks like this

while True:
    with self._lock:
        if self.valid_data: 
            # do something

Memory leak

How to delete a list completely.

My Solution 
self.__data.clear()
self.__data = []

Problem

How to clear the memory very efficiently in python for list and deque data structures? Here, 3 consumer threads use the data and one reset thread clears the thread based on some flag. del self.__data or any other way?

It happens very rarely, which makes me hard to reproduce, so I am just thinking the best code will resolve the issue. Please advise.

Other approaches

Solution

  • Clearing, reassigning and deleting are all about the same. They will spend the vast majority of their time removing references and hopefully deleting their contained objects.

    That is, assuming there are no other references to these objects. If there is another reference to the list or deque, assignment and deleting don't do anything besides remove one of the reference counts. You'd have to do del mylist[:] or mydeque.clear() in that case. If the contained objects are referenced somewhere else, they won't really be deleted either.

    Although objects are usually deleted right away, if they have circular references (a.b = b and b.a = a) the garbage collector needs to run to collect them. Its automagical in the background, but calling gc.collect() right after a large delete intended to clear memory is reasonable. You could then also check len(gc.garbage) which holds dead objects that will never be collected. Anything over 0 is kinda bad but mostly harmless until the number gets large. Then you have to figure out why your code creates dead objects (hard).

    But is dumping the list the right thing to do? What does that mean for the validity of the program output? When you detect a problem, it may be better to raise an exception and run.