Search code examples
pythonpython-3.xpipsubprocess

Import runtime installed module using pip in python 3

I want to install and import Python 3 modules at runtime.

I'm using the following function to install modules at runtime using pip:

def installModules(modules):
    for module in modules:
        print("Installing module {}...".format(module))

        subprocess.call([sys.executable, "-m", "pip", "install", "--user", module])

The module is installed successfully, but I'm not able to import it at runtime, after the installation finishes. So if I do:

modules = [ "wget", "zipfile2" ]
installModules(module)
import wget

I get a ModuleNotFoundError. If, after that, I start another Python 3 session, I am able to use the modules e.g. wget, which means that the modules have been installed, but they are not available for this current Python 3 session.

Is it possible in Python 3 to install and then import the installed modules in the same Python 3 session i.e. right after installation?

Thank you!

EDIT:

On a fresh Ubuntu 19.04 install inside VirtualBox, after a sudo apt-get install python3-pip, running the following script:

import os, sys
import subprocess


def installModules(modules):
    for module in modules:
        print("Installing module {}...".format(module))

        subprocess.call([sys.executable, "-m", "pip", "install", "--user", module])

def process():
    modulesToInstall = [ "wget", "zipfile2" ]
    installModules(modulesToInstall)

process()

import wget

def main():
    wget.download("http://192.168.2.234/test/configure.py")

if __name__ == "__main__":
    main()

I get:

user@user-VirtualBox:~$ python3 script.py
Installing module wget...
Collecting wget
Installing collected packages: wget
Successfully installed wget-3.2
Installing module zipfile2...
Collecting zipfile2
  Using cached https://files.pythonhosted.org/packages/60/ad/d6bc08f235b66c11bbb76df41b973ce93544a907cc0e23c726ea374eee79/zipfile2-0.0.12-py2.py3-none-any.whl
Installing collected packages: zipfile2
Successfully installed zipfile2-0.0.12
Traceback (most recent call last):
  File "script.py", line 17, in <module>
    import wget
ModuleNotFoundError: No module named 'wget'

The Python 3 version is:

user@user-VirtualBox:~$ python3 --version
Python 3.7.3

The pip3 version is:

user@user-VirtualBox:~$ pip3 --version
pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)

Other info:

user@user-VirtualBox:~$ whereis python3
python3: /usr/bin/python3.7m /usr/bin/python3.7-config /usr/bin/python3.7 /usr/bin/python3 /usr/bin/python3.7m-config /usr/lib/python3.7 /usr/lib/python3.8 /usr/lib/python3 /etc/python3.7 /etc/python3 /usr/local/lib/python3.7 /usr/include/python3.7m /usr/include/python3.7 /usr/share/python3 /usr/share/man/man1/python3.1.gz

Any ideas?

Solution

  • By default, at startup Python adds the user site-packages dir (I'm going to refer to it as USPD) in the module search paths. But this only happens if the directory exists on the file system (disk). I didn't find any official documentation to support this statement 1, so I spent some time debugging and wondering why things seem to be so weird.

    The above behavior has a major impact on this particular scenario (pip install --user). Considering the state (at startup) of the Python process that will install modules:

    1. USPD exists:

      • Things are straightforward, everything works OK

    2. USPD doesn't exist:

      • Module installation will create it

      • But, since it's not in the module search paths, all the modules installed there won't be available for (simple) import statements

    When another Python process is started, it will fall under #1.

    To fix things, USPD should be manually added to module search paths. Here's how the (beginning of the) script should look like:

    import os
import site
import subprocess
import sys

user_site = site.getusersitepackages()
if user_site not in sys.path:
    sys.path.append(user_site)

# ...

    Update #0

    1 I just came across [Python]: PEP 370 - Per user site-packages directory - Implementation (emphasis is mine):

    The site module gets a new method adduserpackage() which adds the appropriate directory to the search path. The directory is not added if it doesn't exist when Python is started.