"unable to import requests" in web2py even though requests is accessible directly through python

Question

I'm attempting to integrate MSAL, which requires the requests module.

I'm running Python 3.7 on Linux and using pipenv to manage the environment. I'm also using web2py 2.24.1 from source (as in I download the web2py framework via the source button on the web2py website).

When I am in the pipenv shell and go into the python shell, I can access the requests module; however, when I try to access it from web2py (running in the same shell), I get a "module not found" error.

When I check the site-packages folder, the requests package is present. I have checked PYTHONPATH and seen that the path from the virtual environment is present.

When attempting to load the web2py python shell, it gives the same error.

I'm probably missing something, but it sometimes appears as if web2py does some code compilation and then uses the compiled stuff and ignores code changes after a certain point. Asking as I have commented out all the code involving the requests module in an effort to get the web2py shell working, but still get the error.

Not sure what to try next. Any ideas are appreciated.

Answer 1

This is due to a buggy interaction between web2py's custom importer and the urllib3 module (which is imported by requests).

1. web2py's custom importer raises an ImportError if a module is not found (code).
   (Arguably, it should instead raise ModuleNotFoundError, which is the subclass of ImportError specifically for this situation.)

2. urllib3 uses except ModuleNotFoundError: to ignore errors when importing urllib3_secure_extra (code; only used when distributed via PyPI).
   (Arguably, it should instead catch the more general ImportError.)

The result is that when urllib3_secure_extra is missing, the loading of urllib3 fails instead of continuing gracefully.

---

Until this is fixed in web2py and/or urllib3, here are some workarounds:

1. Patch web2py (change the line with the "-" to the line with the "+")

--- web2py/gluon/custom_import.py.orig
+++ web2py/gluon/custom_import.py
@@ -77,7 +77,7 @@
                     try:
                         result = sys.modules[modules_prefix]
                     except KeyError:
-                        raise ImportError("No module named %s" % modules_prefix)
+                        raise ModuleNotFoundError("No module named %s" % modules_prefix)
             return result
         else:
             # "from x import a, b, ..."

2. Patch urllib3

--- <virtualenv>/lib/python3.9/site-packages/urllib3/__init__.py.orig
+++ <virtualenv>/lib/python3.9/site-packages/urllib3/__init__.py
@@ -48,7 +48,7 @@
 # See: https://github.com/urllib3/urllib3/issues/2680
 try:
     import urllib3_secure_extra  # type: ignore # noqa: F401
-except ModuleNotFoundError:
+except ImportError:
     pass
 else:
     warnings.warn(

3. Dynamically patch web2py's importer (does not require modifying the web2py/urllib3 files)

Insert the following in your application's initialization code (e.g. web2py/applications/<yourapp>/__init__.py):

import builtins
_real_import = builtins.__import__
def _import_with_modulenotfounderror(*args, **kwargs):
    try:
        return _real_import(*args, **kwargs)
    except ImportError as e:
        if e.__class__ is ImportError and str(e).startswith("No module named "):
            raise ModuleNotFoundError(str(e)) from e
        else:
            raise
builtins.__import__ = _import_with_modulenotfounderror

---

P.S. I'm not sure why you keep getting the error after commenting out the code. Some things to try:

• In the admin interface, click "Manage" next to your app then click "Remove compiled" (this button will only be shown if the application is compiled).
• Stop web2py then start it again.
• Look in the stack trace for lines from your source files, and double check that the lines at those line numbers are commented out.
• A similar issue was described here (search for the word "recompile"); the workaround mentioned there is to recompile programmatically.