Search code examples
python-2.7installationsetuptoolseasy-install

easy_install clobbers my easy-install.pth and drops the python-commands in the beginning and end of the file

I am facing a strange problem with my sys.path; Since I am going to describe the problem clearly this post is going to be a little long one.

my /usr/lib/python2.7/site-packages/easy-install.pth looks like this:

import sys; sys.__plen = len(sys.path)
./setuptools-27.2.0-py2.7.egg
./jmespath-0.9.3-py2.7.egg
./chardet-3.0.4-py2.7.egg
./certifi-2018.01.18-py2.7.egg
./urllib3-1.22-py2.7.egg
./requests-2.18.4-py2.7.egg
./docutils-0.14-py2.7.egg
./python_dateutil-2.7.5-py2.7.egg
./enum34-1.1.6-py2.7.egg
./six-1.10.0-py2.7.egg
./ipaddress-1.0.18-py2.7.egg
./asn1crypto-0.22.0-py2.7.egg
./idna-2.6-py2.7.egg
./pyOpenSSL-17.2.0-py2.7.egg
/usr/lib64/python2.7/site-packages/cryptography-2.0.3-py2.7-linux-x86_64.egg
/usr/lib64/python2.7/site-packages/cffi-1.10.0-py2.7-linux-x86_64.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

With this things work fine for me; specifically, correct version of pyOpenSSL gets included in my sys.path:

[root@sandbox ~]# python -c "import OpenSSL; print OpenSSL.__version__"
17.2.0

when I remove the first and last lines the python commands my pyOpenSSL version changes; which means when my easy-install.pth looks like this:

./setuptools-27.2.0-py2.7.egg
./jmespath-0.9.3-py2.7.egg
./chardet-3.0.4-py2.7.egg
./certifi-2018.01.18-py2.7.egg
./urllib3-1.22-py2.7.egg
./requests-2.18.4-py2.7.egg
./docutils-0.14-py2.7.egg
./python_dateutil-2.7.5-py2.7.egg
./enum34-1.1.6-py2.7.egg
./six-1.10.0-py2.7.egg
./ipaddress-1.0.18-py2.7.egg
./asn1crypto-0.22.0-py2.7.egg
./idna-2.6-py2.7.egg
./pyOpenSSL-17.2.0-py2.7.egg
/usr/lib64/python2.7/site-packages/cryptography-2.0.3-py2.7-linux-x86_64.egg
/usr/lib64/python2.7/site-packages/cffi-1.10.0-py2.7-linux-x86_64.egg

then the pyOpenSSL version is an older version:

[root@sandbox ~]# python -c "import OpenSSL; print OpenSSL.__version__"
0.13.1

I think this is because an older version of pyOpenSSL is present over here /usr/lib64/python2.7/site-packages/pyOpenSSL-0.13.1-py2.7.egg-info

so my first question: What are those commands doing in easy-install.pth and how is easy-install.pth processed to change my sys.path?

Next is when I try to install new python packages using easy_install (from their source code) it clobbers my easy-install.pth -- in the sense that all packages would still be there but without the first and last lines (those python commands).

To be more concrete: I am trying to install idna-2.5 and it does successfully installs but messes up easy-install.pth

[root@sandbox idna-2.5]# easy_install .
Processing .
Writing /opt/proj/sbs-installs/COSpkgs/idna-2.5/setup.cfg
Running setup.py -q bdist_egg --dist-dir /opt/proj/sbs-installs/COSpkgs/idna-2.5/egg-dist-tmp-sLNbiE
warning: no previously-included files matching '*.pyc' found under directory 'tools'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
zip_safe flag not set; analyzing archive contents...
Removing /usr/lib/python2.7/site-packages/idna-2.5-py2.7.egg
Moving idna-2.5-py2.7.egg to /usr/lib/python2.7/site-packages
Removing idna 2.6 from easy-install.pth file
Adding idna 2.5 to easy-install.pth file

Installed /usr/lib/python2.7/site-packages/idna-2.5-py2.7.egg
Processing dependencies for idna==2.5
Finished processing dependencies for idna==2.5

Notice that it uninstalled idnz 2.6. After this my easy-install.pth looks like this:

./setuptools-27.2.0-py2.7.egg
./jmespath-0.9.3-py2.7.egg
./chardet-3.0.4-py2.7.egg
./certifi-2018.01.18-py2.7.egg
./urllib3-1.22-py2.7.egg
./requests-2.18.4-py2.7.egg
./docutils-0.14-py2.7.egg
./python_dateutil-2.7.5-py2.7.egg
./enum34-1.1.6-py2.7.egg
./six-1.10.0-py2.7.egg
./ipaddress-1.0.18-py2.7.egg
./asn1crypto-0.22.0-py2.7.egg
./pyOpenSSL-17.2.0-py2.7.egg
/usr/lib64/python2.7/site-packages/cryptography-2.0.3-py2.7-linux-x86_64.egg
/usr/lib64/python2.7/site-packages/cffi-1.10.0-py2.7-linux-x86_64.egg
./idna-2.5-py2.7.egg

and now when I check the pyOpenSSL version it the old one (as mentioned above):

[root@sandbox ~]# python -c "import OpenSSL; print OpenSSL.__version__"
0.13.1

Thanks for your help in advance.

Solution

  • It took some digging to find the answer. I am posting the answer here for the sake of completeness and searchability :).

    The python command at the beginning of easy-install.pth is called prelude and the one at the last line is called postlude (in setuptools code).

    As is expected easy_install re-writes the PathDistributions, i.e. easy-install.pth, if you installation leads to removal of an old package and installation of your new package.

    Now if you want to include the prelude and postlude the code expects SETUPTOOLS_SYS_PATH_TECHNIQUE environment property to be set to rewrite (default value is raw). To put it more concretely, following commands fixed the problem:

    [root@sandbox idna-2.5]# export SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite
[root@sandbox idna-2.5]# easy_install .
Processing .
Writing /opt/proj/sbs-installs/COSpkgs/idna-2.5/setup.cfg
Running setup.py -q bdist_egg --dist-dir /opt/proj/sbs-installs/COSpkgs/idna-2.5/egg-dist-tmp-gE1V2U
warning: no previously-included files matching '*.pyc' found under directory 'tools'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
zip_safe flag not set; analyzing archive contents...
Removing /usr/lib/python2.7/site-packages/idna-2.5-py2.7.egg
Moving idna-2.5-py2.7.egg to /usr/lib/python2.7/site-packages
Removing idna 2.6 from easy-install.pth file
Adding idna 2.5 to easy-install.pth file

Installed /usr/lib/python2.7/site-packages/idna-2.5-py2.7.egg
Processing dependencies for idna==2.5
Finished processing dependencies for idna==2.5
[root@sandbox idna-2.5]#