Unable To Run IPython nbconvert From Python2.7 Virtual Environment

Question

I have a virtual environment of Python 2.7 with ipython installed (Ubuntu 16.04.2 (Xenial) LTS.)

When I’m working in the virtual environment (after running source venv/bin/activate in bash shell while being in the parent directory of the virtual environment) I have no problem executing conversion of my jupiter’s notebook from bash shell like so:

ipython nbconvert --to html --execute my_notes.ipynb --stdout > /tmp/report.html 

But when I’m trying to call that command from fabric’s task using subprocess:

command = ['ipython', 'nbconvert', '--to', 'html', '--execute', notebook_path, '--stdout']
output = subprocess.check_output(command,
                                 cwd=os.environ['PYTHONPATH'],
                                 env=os.environ.copy())

It always fails with this exception I cannot find a reason for it:

Traceback (most recent call last):
  File "/opt/backend/venv/bin/ipython", line 7, in <module>
    from IPython import start_ipython
  File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
    from .core.application import Application
  File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
    from IPython.core import release, crashhandler
  File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
    from IPython.core import ultratb
  File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/ultratb.py", line 119, in <module>
    from IPython.core import debugger
  File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/debugger.py", line 46, in <module>
    from pdb import Pdb as OldPdb
  File "/usr/lib/python2.7/pdb.py", line 59, in <module>
    class Pdb(bdb.Bdb, cmd.Cmd):
AttributeError: 'module' object has no attribute 'Cmd'

More info to save your time.

I’ve tried:

• Using same paths for PYTHONPATH as I got from PyCharm run/debug configuration.
• Using nbconvert as python library from this documentation.
• Tried os.system("ipython nbconvert…").
• Wrapped working command (ipython nbconvert…) with a shell script and used it in subprocess.check_output and os.system.
• Get drunk and bang my head on a brick wall.

And always end-up with that cursed exception.

Answer 1

Reposting as an answer for completeness:

There was a file called cmd.py somewhere where Python was finding it as an importable module. This was shadowing the cmd module in the standard library, which is used by pdb, which IPython imports. When pdb tried to subclass a class from cmd, that class wasn't there. Moving cmd.py out of the way lets it find the cmd module it needs.

This is an unfortunate annoyance with Python - lots of short words are already used as module names, and using them yourself produces crashes, with a wide range of different errors.