If a function creates a lot of objects in a loop that are not referenced elsewhere, they will be removed immediately due to Python's reference counting.
If I store the objects in a list this time, the objects will be removed by garbage collection when the function exits. The objects do not have a circular reference. Why are they not removed by ReferenceCounting when the function ends and the list is removed?
As an example I have the following program with two scenarios where I call the run function once with the parameter with_append = True and the other time with with_append = False.
In the first scenario, the garbage collection jumps in and the whole programme takes much longer to finish. In the second scenario, there is no garbage collection active. You can see this by running the program with py-spy and using the native option.
Below is the example programme and the output for both cases.
import time
COUNT = 10000000
class User:
def __init__(self, name):
self.name = name
def run(with_append):
l = []
for i in range(COUNT):
u = User(f"user {i}")
if with_append:
l.append(u)
ts = time.time()
run(with_append=True)
print("function completed - ", time.time() - ts)
Call run with param: with_append = True
Output of: python demo.py
function completed - 10.940133094787598
Output of: py-spy top --native -- python demo.py
Total Samples 1000
GIL: 100.00%, Active: 100.00%, Threads: 1
%Own %Total OwnTime TotalTime Function (filename)
53.00% 53.00% 4.51s 4.51s gc_collect_main (libpython3.10.so.1.0)
38.00% 100.00% 4.41s 10.00s run (demo3.py)
9.00% 10.00% 1.07s 3.13s __init__ (demo3.py)
0.00% 0.00% 0.010s 0.010s 0x7f91a5b2a746 (libc-2.31.so)
0.00% 100.00% 0.000s 10.00s <module> (demo3.py)
0.00% 53.00% 0.000s 4.51s gc_collect_with_callback (libpython3.10.so.1.0)
Half of the time is spend in gc_collect_main.
Call run with param: with_append = False
Output of: python demo.py
function completed - 4.351471424102783
Output of: py-spy top --native -- python demo.py
Total Samples 400
GIL: 100.00%, Active: 100.00%, Threads: 1
%Own %Total OwnTime TotalTime Function (filename)
85.00% 100.00% 3.38s 4.00s run (demo3.py)
13.00% 14.00% 0.560s 0.600s __init__ (demo3.py)
1.00% 1.00% 0.020s 0.020s unicode_dealloc (libpython3.10.so.1.0)
1.00% 1.00% 0.020s 0.020s 0x7fe05fca5742 (libc-2.31.so)
0.00% 0.00% 0.010s 0.010s 0x7fe05fca5746 (libc-2.31.so)
0.00% 0.00% 0.010s 0.010s 0x7fe05fca5724 (libc-2.31.so)
0.00% 100.00% 0.000s 4.00s <module> (demo3.py)
Why is reference counting not used to free memory in both cases? And why did garbage collection take so much more time to remove the objects?
Most likely, in both scenarios, objects are deleted by reference counting.
In Scenario 1, gc_collect_main is automatically called in the middle of the loop to detect collectable objects with circular references. No objects are actually deleted by this in Scenario 1. However, this detection is performed tens of thousands of times, so it seems to be time consuming overall.
This automatic detection is triggered by an increase in the number of objects (I am not absolutely sure about this), so it is not executed during the loop in Scenario 2, where the objects are deleted at the end of each loop by reference counting.
You can put the following code at the top of your script to check when the detection was performed.
import gc
gc.set_debug(gc.DEBUG_STATS)
Alternatively, explicitly disabling detection during the loop should significantly improve performance.
import gc
def run(with_append):
gc.disable()
l = []
for i in range(COUNT):
u = User(f"user {i}")
if with_append:
l.append(u)
gc.enable()
Note that this is mentioned in the official documentation.
Since the collector supplements the reference counting already used in Python, you can disable the collector if you are sure your program does not create reference cycles.
Obviously, Python cannot automatically determine whether a circular reference exists (i.e., whether garbage collection is needed), so if this is a problem, you must explicitly disable it in this way.