Search code examples
python-3.xpippythonpathsite-packageslinuxbrew

Migrate site-packages/modules installed with pip from older to newer/latest version of python

At first, I moved the python modules installed in the /../python3.7/site-packages to the new /../python3.8/site-packages directory manually.

To generalize the following question, I'm introducing the following nomenclature: The initial (lower) python-version will be represented by 3.x, the newer one by 3.y. In my case, x=7 and y=8.

After the manual moving of the 3.x site-packages to the 3.y site-packages directory, I found that upon executing python scripts using the new python version 3.y certain errors occurred when calling specific modules like numpy, scipy, matplotlib etc. The quite unelegant workaround which did it for me at last was simply python -m pip uninstall packagename followed by python -m pip install packagename, i.e. simply uninstall, then reinstall it. This way, the errors vanished. Fortunately, I didn't have to do this with all of my more than 100 packages, it was rather necessary for just around 10.

Nevertheless, for the future I'd like to know how to migrate the packages more elegantly from the former to the current python version and update those automatically where it's needed.

System specifics:

  • OS: Linux Ubuntu 18.04 LTS
  • Python: Python 3.8.3 (default, Jul 9 2020, 19:37:58) [GCC 5.4.0 20160609] on linux (maintained via linuxbrew / homebrew)
  • Editor: VS Code
Solution

  • EDIT: The most elegant solution can be found here:

    I found a solution as a combination of this answer and changing the PYTHONPATH in two of my config-files:

    /home/user/.bashrc 
/home/user/.bash_profile

    Nomenclature: The initial (lower) python-version will be represented by 3.x, the newer one by 3.y.

    NOTE on the necessity of previously modifying the PYTHONPATH:

    In my case, as there were many python-versions installed in various ways and locations, it was needed to prepend the target sitepackage directory to every other one present in the entire PYTHONPATH, since only this way the extracted python-modules will be installed in the correct directory via python -m pip install --user -r packages_python_3_x.txt.

    If on your system, the PYTHONPATH points directly to the site-packages of desire, i.e. they are in the first position of your entire PYTHONPATH, you wouldn't need to modify it. Moreover, if you're working with virtual-environments (e.g. pyenv) or other environment managers, you will be able to separate each site-packages associated to every virtual-environment automatically. Then, there should also be no need to modify the PYTHONPATH. By contrast, I didn't work with virtual, local and global python environments and that's why it was necessary for me to prepend the site-package-directory of interest to the PYTHONPATH. Now, I'm working with virtual-environments for different projects, which I recommend to everyone.

    1.) Preparing the PYTHONPATH (if needed)

    Continuing with said config-files, therein, insert the following line:

    PYTHONPATH="/home/linuxbrew/.linuxbrew/lib/python3.x/site-packages:$PYTHONPATH"

    In case there was a similar line pointing to the previous python-version directory, replace that with the most recent one in accordance with your situation (e.g. 3.7 with 3.8 in the directory path).

    Moreover, if the system python3-path is prepended to the PYTHONPATH, comment that comment out in order to give your python distribution (be it homebrew, anaconda, or whatever you like) precedence.

    # NOTE on the following commando: it is commented out on purpose (you could also delete it from the config-files, if you don't consider to use these site-packages again, but I kept it for playing safe)
#export PYTHONPATH="/usr/lib/python3/dist-packages:$PYTHONPATH"

    In order to make changes take effect immediately, carry out the following commandos in your UNIX-terminal:

    source /home/user/.bash_profile 
source /home/user/.bashrc

    2.) Extract currently installed modules and reinstall them with new version 3.y

    Finally, extract all python-packages installed to a textfile. In my case, this had been done with pip, so this would be:

    python -m pip freeze --path /home/linuxbrew/.linuxbrew/lib/python3.x/site-packages > packages_python_3_x.txt

    The textfile should contain lines like so:

    absl-py==0.11.0
access==1.1.1
affine==2.3.0
aiohttp==3.7.2
aiohttp-cors==0.7.0
...

    Finally, (re)install all these packages into the latest site-packages folder (here done using pip):

    python -m pip install --user -r packages_python_3_x.txt

    The package manager pip should choose the correct folder since you have updated the PYTHONPATH previously. The new site-packages directory in this example should be /home/linuxbrew/.linuxbrew/lib/python3.y/site-packages.