Search code examples
pythonperformancepython-import

Python script stuck at import

I have several Python scripts which run using crontab every minute. After installing python-binance using pip3, only sometimes (seems random to me) all my scripts that use cryptography library get stuck at import.

As you can see from the log below, what was supposed to run every minute and last less than a second, at some point took more than ten minutes.

2019-03-30 08:51:07 INFO     (done)
2019-03-30 08:52:07 INFO     (done)
2019-03-30 08:53:07 INFO     (done)
2019-03-30 09:04:45 INFO     (done)
2019-03-30 09:05:45 INFO     (done)
2019-03-30 09:06:45 INFO     (done)

Trying to debug the problem, I run the script in a loop using python -vv.

During the slow iteration, the Python interpreter was stuck here for several minutes

# trying /home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/scrypt.cpython-36m-x86_64-linux-gnu.so
# trying /home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/scrypt.abi3.so
# trying /home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/scrypt.so
# trying /home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/scrypt.py
# /home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/__pycache__/scrypt.cpython-36.pyc matches /home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/scrypt.py
# code object from '/home/user/.local/lib/python3.6/site-packages/cryptography/hazmat/primitives/kdf/__pycache__/scrypt.cpython-36.pyc'
import 'cryptography.hazmat.primitives.kdf.scrypt' # <_frozen_importlib_external.SourceFileLoader object at 0x7f5aff05d6a0>

Additional info:

  • OS: Ubuntu 18.04

  • Python version: python3 3.6.7-1~18.04

  • Python-binance version: python-binance==0.7.1

  • It's not a resource issue, when the interpreter was stuck the CPU was less then 20% used, plenty of RAM free and no disk bottleneck

  • The issue is occurring on different servers with the same configuration

  • I have tried to run the script using venv with the same result

I have tried to increase the ulimit -n from 1024 to 4096, which unfortunately didn't help.

Solution

  • I figured out the problem was related to wheels.

    I have installed the module in a venv using:

    pip3 install binance --no-binary :all:

    --no-binary :all: is making pip compile dependencies from source code instead of using any pre-compiled wheels potentially available.