memory increase after upgrading setuptools

Question

After upgrading setuptools to 65.3.0 all of sudden there is memory increase for majority of the process in the system. To narrow down the issue, I tried to find out the problematic import and checked, it looks like the distutils import in the script is causing loading of lot of dynamic libraries in the new version and causing more time to load and also increased memory usage.

from distutils.version import StrictVersion
import time

while True:
    time.sleep(1000)

[root@controller(FHGW-83) /home/robot]
# pmap 2440334 | wc -l
196
[root@controller(FHGW-83) /home/robot]
# pmap 2440334 | grep cpython | wc -l
111

# pmap 2432196 | grep cpython
00007f8606e8a000     72K r---- _ssl.cpython-39-x86_64-linux-gnu.so
...
00007f8606f02000     24K r-x-- _json.cpython-39-x86_64-linux-gnu.so
...
00007f8607296000      4K rw--- _csv.cpython-39-x86_64-linux-gnu.so
...
00007f860786f000      4K rw--- _sha512.cpython-39-x86_64-linux-gnu.so
...
00007f86078fe000      8K rw--- pyexpat.cpython-39-x86_64-linux-gnu.so
...
00007f86079f9000      4K rw--- _struct.cpython-39-x86_64-linux-gnu.so
...
and more...


$ top -p 2440334 
    PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND                                                                                                                    
2440334 root      20   0   31.9m  27.2m   0.0   0.2   0:00.68 S python test.py  

In a machine where setuptools is at 57.0.0, this issue is not seen. virtual & RES memory consumption is also less.

# python test.py &
[1] 1562394
]
# pmap 1562394 | wc -l
67
# pmap 1562394 | grep cpyth | wc -l
5


PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND                                                                                                                    
1562394 root      20   0   14.0m   9.3m   0.0   0.1   0:00.05 S python test.py 

Here what is the relation with setuptools? python version (3.9.16) is same in both the machines. why is the script causing python to load lot of libraries?

Answer 1

I was able to solve this in python 3.9 by simply exporting SETUPTOOLS_USE_DISTUTILS=stdlib

This will make sure to use distutils from the standard python distribution. If we export SETUPTOOLS_USE_DISTUTILS=local then distutils is used from setuptools package.

>>> import distutils
>>> distutils.__file__
'/usr/lib/python3.9/distutils/__init__.py'


#after setuptools upgrade
>>> import distutils
>>> distutils.__file__
'/usr/lib/python3.9/site-packages/setuptools/_distutils/__init__.py'