I am using an Apple factory-installed distribution of Python 2.7.5 on a Mac laptop running OS X 10.9.4.
This particular distribution comes with "factory-loaded" versions of several popular Python add-on packages, such as numpy, many of which live in the directory /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python.
I find that I'm able to update the version numbers of these pre-loaded packages, but only if I use the Apple factory-installed update utility, /usr/bin/easy_install. If I instead attempt to update using either pip, or directly from .tar.gz source using python setup.py install, then the installation fails in a very particular way, which I'll describe in detail below. I'd like to fix this problem, so that I'm able to update Python packages using whatever method I like.
First, a description of what I believe to be the "correct" behavior: when I update numpy using easy_install, it creates a new subdirectory, numpy-1.9.0-py2.7-macosx-10.9-intel.egg, under /Library/Python/2.7/site-packages, e.g.:
$> ls -l /Library/Python/2.7/site-packages | grep numpy
drwxr-xr-x 4 root wheel 136 Sep 11 10:21 numpy-1.9.0-py2.7-macosx-10.9-intel.egg
and then the easy_install utility also modifies the Python module load path so that the "new" numpy installation will take precedence over the factory default version; e.g., within python, it looks like this:
In [1]: from pprint import pprint
In [2]: import sys
In [3]: pprint(sys.path)
['',
'/usr/local/bin',
'/Library/Python/2.7/site-packages/PyVCF-0.6.7-py2.7-macosx-10.9-intel.egg',
'/Library/Python/2.7/site-packages/distribute-0.7.3-py2.7.egg',
'/Library/Python/2.7/site-packages/setuptools-3.3-py2.7.egg',
'/Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.9-intel.egg',
'/Library/Python/2.7/site-packages/wheel-0.24.0-py2.7.egg',
'/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg',
'/Library/Python/2.7/site-packages/numpy-1.9.0-py2.7-macosx-10.9-intel.egg',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
'/Library/Python/2.7/site-packages',
'/Library/Python/2.7/site-packages/IPython/extensions',
'/Users/stachyra/.ipython']
In [4]: import numpy
In [5]: numpy.__file__
Out[5]: '/Library/Python/2.7/site-packages/numpy-1.9.0-py2.7-macosx-10.9-intel.egg/numpy/__init__.pyc'
In [6]: numpy.__version__
Out[6]: '1.9.0'
If I uninstall numpy, for example using sudo pip uninstall numpy (I couldn't actually figure out how to do an uninstall with easy_install), it results in numpy being deleted from /Library/Python/2.7/site-packages, while the factory-installed "default" version in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python still remains.
To observe the "incorrect" installation behavior, I can re-upgrade the factory-loaded version of numpy, this time using sudo pip install -U numpy. This leads to a new pair of directories being created in /Library/Python/2.7/site-packages, e.g.:
$> ls -l /Library/Python/2.7/site-packages | grep numpy
drwxr-xr-x 35 root wheel 1190 Sep 11 10:24 numpy
drwxr-xr-x 8 root wheel 272 Sep 11 10:24 numpy-1.9.0.dist-info
However, when I look at the load path in python, and then load numpy and look at the version number and file for numpy, I get the following result:
In [1]: from pprint import pprint
In [2]: import sys
In [3]: pprint(sys.path)
['',
'/usr/local/bin',
'/Library/Python/2.7/site-packages/PyVCF-0.6.7-py2.7-macosx-10.9-intel.egg',
'/Library/Python/2.7/site-packages/distribute-0.7.3-py2.7.egg',
'/Library/Python/2.7/site-packages/setuptools-3.3-py2.7.egg',
'/Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.9-intel.egg',
'/Library/Python/2.7/site-packages/wheel-0.24.0-py2.7.egg',
'/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
'/Library/Python/2.7/site-packages',
'/Library/Python/2.7/site-packages/IPython/extensions',
'/Users/stachyra/.ipython']
In [4]: import numpy
In [5]: numpy.__file__
Out[5]: '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/__init__.pyc'
In [6]: numpy.__version__
Out[6]: '1.6.2'
As you can see, based upon the above, it appears as if pip is failing to update the sys.path variable with the newly updated numpy version, and instead of the latest one (numpy 1.9.0) being loaded, python instead loads the older, factory default version, 1.6.2.
Besides the above two methods, I can also install numpy directly from sourceforge .tar.gz files, using sudo python setup.py install, and in that case I get a very similar-looking result as when I use pip.
So my question is, first of all, what exactly is broken here? Is it the Apple factory default version of Python, for having installed pre-loaded Python packages into some other non-standard directory area (/System/Library/Frameworks/Python.framework/Versions/2.7) besides the usual default (/Library/Python/2.7/site-packages)? Or is it pip and the .tar.gz source files, for failing to override the Apple factory default installation properly?
Also, the second part of my question: how do I fix this problem, so that I may use pip and .tar.gz in the future if I want to? And FWIW, if the answer turns out to include uninstalling Apple factory defaults, then I would prefer not to have to go into the /System/Library/Frameworks directory and just start manually deleting stuff, as that seems like a potential recipe for breaking things.
One more thing: I'm aware that I could probably sidestep this entire problem by simply using some other alternative distribution such as Anaconda or Enthought Canopy, however it seems much cleaner and more elegant to me to fix the issue "properly" if possible.
easy_install uses a method which munges the sys.path to inject the installed items to the front of the sys.path while on the other hand pip simply places them in the directory that Python looks in by default (site-packages). This pretty much always works, unless the source of your Python has modified Python like Apple has to inject their own pre-installed projects before the site-packages directory.
There is an open issue against pip to come up with some solution to this problem but fundamentally the problem is because Apple broke the default sys.path order.