Python module in path: ImportError raised in mako template

There is an ImportError that has the potential to drive me mad. The situation looks like this:

tests/
    testWebsite.py
website/
    __init__.py
    __main__.py
    _webtools/
        __init__.py
        templatedefs.py
        ...
    _templates/
        base.mako
        article.mako
        ...

The code (sans the tests directory, which I hesitate to commit, before the problem is solved) is online here: https://github.com/Boldewyn/website/.

When I call python -m website.__main__ build, the main routine creates from some input static HTML files using the templates under website/_templates. This works just fine in any given directory.

However, in the tests/testWebsite.py I have a unit test, that should run the same thing, too. But there the Mako templates raise import errors for files, that are imported fine in the other case.

$ head -n 5 website/_templates/article.mako
# -*- coding: utf-8 -*-
<%!
from website._webtools.templatedefs import strip_tags
%>
<%inherit file="base.mako" />

Running the test then yields:

$ python -m unittest tests.testWebsite
...
ERROR: test_initial_build (tests.testWebsite.BuildTestCase)
Check, if building directly after bootstrap works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/testWebsite.py", line 99, in test_initial_build
  File "website/_webtools/build.py", line 89, in build
    article.save(articles=articles)
  File "website/_webtools/articles.py", line 514, in save
    template_engine.render_article(self, **ctx)
  File "website/_webtools/templates.py", line 52, in render_article
    r.render_article(article, **ctx)
  File "website/_webtools/templates.py", line 277, in render_article
    tpl = self.lookup.get_template(filename)
  File "/usr/lib/python2.7/dist-packages/mako/lookup.py", line 217, in get_template
    return self._load(srcfile, uri)
  File "/usr/lib/python2.7/dist-packages/mako/lookup.py", line 277, in _load
    **self.template_args)
  File "/usr/lib/python2.7/dist-packages/mako/template.py", line 205, in __init__
    module = self._compile_from_file(path, filename)
  File "/usr/lib/python2.7/dist-packages/mako/template.py", line 249, in _compile_from_file
    filename)
  File "/usr/lib/python2.7/dist-packages/mako/template.py", line 470, in _compile_text
    exec code in module.__dict__, module.__dict__
  File "_templates_article_mako", line 16, in <module>
ImportError: No module named templatedefs

Now, the funny part is, that I can print sys.path directly from the template:

<%!
import sys
print sys.path
from website._webtools.templatedefs import strip_tags
%>

And I can confirm there, that website is in the path. Also, the import does work well in every other deployment scenario.

Importing website or website._webtools also works well. Only the website._webtools.templatedefs part goes wrong.

Has anyone an idea, where I could look to find signs of what might go wrong?

The test code is quite straight-forward:

class BuildTestCase(unittest.TestCase):

    def setUp(self):
        self.tmpdir = tempfile.mkdtemp()
        self.cwd = os.getcwd()
        os.chdir(self.tmpdir)
        bootstrap(self.tmpdir, { # this initiates a new project
          "URL": "localhost",
          "TITLE": "Example",
          "DEFAULTS": {
              "AUTHOR": "John Doe",
          }
        })

    def test_initial_build(self):
        """Check, if building directly after bootstrap works"""
        build()

    def tearDown(self):
        os.chdir(self.cwd)
        shutil.rmtree(self.tmpdir)

Edit: One more diagnostic: I let mako compile the template and executed the resulting Python file stand-alone. Works like a charm. I also reduced templatedefs.py to the bare minimum (only defs returning empty strings), so that I can exclude ImportErrors (or other weirdness) in that file as well.

System info: Ubuntu 11.04, Python 2.7, Mako 0.3.6.

Solution

This indeed drives one mad. However here are some things:

./nosetests: this works and all 9 tests pass
'templatedefs' is the only key missing in '_webtools.__dict__' when you add an 'from website import _webtools' to your mako template and compare 'nosetests' to 'python -m unittest tests.testWebsite': the other parts were already imported earlier
sys.path contains '' (a relative path) in the 'python -m unittest tests.testWebsite' case, but not in the 'nosetests' case, where sys.path contains only absolute paths. This results in different values for 'website._webtools.__file__': one is relative ['website/_webtools'], the other is absolute ['/home/username/tmp/website/_webtools']. Since you make a os.chdir, the relative paths don't work any more.

SO: If you want to use pure unittest, you can add 'import website._webtools.templatedefs' at the beginning of your test file. This ensures that you have templatedefs imported when you run os.chdir. And I would suggest to use nose. Hope that helps.