Search code examples
pythonjinja2pex

How does python load a module from a path which is not in sys.path

I was debugging a Python import issue where I can't load jinja2 properly while running my code using a pex file.

Setup a breakpoint in my code, and found myself in a very interesting situation

(Pdb) import jinja2
(Pdb) import jinja2.utils
(Pdb) jinja2.utils
*** AttributeError: 'module' object has no attribute 'utils'
(Pdb) sys.modules.get("jinja2")
<module 'jinja2' from '/home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/__init__.py'>
(Pdb) sys.modules.get("jinja2.utils")
<module 'jinja2.utils' from '/usr/local/lib/python2.7/dist-packages/jinja2/utils.pyc'>
(Pdb) [i for i in sys.path if "usr/local/lib/python2.7" in i]
[]

Before the breakpoint, jinja2 was already partially loaded because I set up the breakpoint inside jinja2's __init__.py, here is the stacktrace:

-> exec code in run_globals
  /home/xxx/.pex/code/e7fe7e585c258eb7224ec5030b9d53b74cdfb06f/application/app.py(2)<module>()
-> from flask import Flask, abort, request
  /home/xxx/.pex/install/Flask-1.0-py2.py3-none-any.whl.abe8dd5f49600e54576b3360b7622e1cc97e86fc/Flask-1.0-py2.py3-none-any.whl/flask/__init__.py(19)<module>()
-> from jinja2 import Markup, escape
> /home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/__init__.py(58)<module>()
-> print("aaaaaaaaa")

Questions are:

  1. How does python even load the jinja2.utils from /usr/local/lib/python2.7/dist-packages when it is not even not part of the sys.path?
  2. Why python found jinja2's __init__.py path correctly but failed to find the utils.py which is under the same directory?
~$ ls /home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/
asyncfilters.py  bccache.py  compiler.py   debug.py     environment.py  ext.py      _identifier.py  __init__.py  loaders.py  nativetypes.py  optimizer.py  runtime.py  tests.py  visitor.py
asyncsupport.py  _compat.py  constants.py  defaults.py  exceptions.py   filters.py  idtracking.py   lexer.py     meta.py     nodes.py        parser.py     sandbox.py  utils.py
Solution

  • Confirmed it is a pex issue.

    Pex loaded jinja2 in the beginning (which also included jinja2.utils), then it removed the jinaj2 module and the path but failed to remove jinja2.utils.

    The issue had been fixed with latest version of pex:https://github.com/pantsbuild/pex/commit/0f9bf622e1565dcdd27c3b79b788571c9608c272