Search code examples
pythonpython-importgit-submodulesconventionspylint

Avoiding pylint complaints when importing Python packages from submodules


Background

I have a Python application dependent upon another package which is provided as a git submodule, yielding a directory structure similar to the following:

foo/
    bar/
        bar/
            __init__.py
            eggs.py
        test/
        setup.py
    foo/
        __init__.py
        ham.py
    main.py

Accessing the foo package is simple enough, as main.py is executed from the toplevel foo/ directory; but the bar package is nested within another bar directory and is not directly importable.

This is solvable readily enough, by modifying sys.path at the beginning of main.py:

import sys

# Or sys.path.append()
sys.path.insert(0, './bar')

from bar.eggs import Eggs
from foo.ham import Ham

(Note: this code example assumes that main.py will always be invoked from foo/; in cases where this may not be the case, '.bar' could be replaced with os.path.join(os.path.dirname(__file__), 'bar') though this is clearly more unwieldy.)

The Problem

Unfortunately, pylint doesn't like this solution. While the code works, the linter considers the sys.path modifications to be a block of code ending the "top of the module" and gives an undesirable wrong-import-position warning:

C: 6, 0: Import "from bar.eggs import Eggs" should be placed at the top of the module (wrong-import-position)
C: 7, 0: Import "from foo.ham import Ham" should be placed at the top of the module (wrong-import-position)

Similar questions

Adding a path to sys.path in python and pylint

This questioner has an issue with pylint failing to correctly parse the imports altogether. The lone answer to the this question suggests adding to pylint's internal path; this does nothing to avoid complaints about an interleaved sys.path modification.


Solution

  • Configure pylint

    Disabling the wrong-import-position checker in .pylintrc is the simplest solution, but throws away valid warnings.

    A better solution is to tell pylint to ignore the wrong-import-position for these imports, inline. The false-positive imports can be nested in an enable-disable block without losing any coverage elsewhere:

    import sys
    
    sys.path.insert(0, './bar')
    
    #pylint: disable=wrong-import-position
    
    from bar.eggs import Eggs
    from foo.ham import Ham
    
    #pylint: enable=wrong-import-position
    
    Ham()
    
    # Still caught
    import something_else
    

    However, this does have the slight downside of funkiness if wrong-import-order is ever disabled in .pylintrc.


    Avoid modifying sys.path

    Sometimes unwanted linting warnings stem from going about a problem incorrectly to start with. I've come up with a number of ways to avoid modifying sys.path in the first place, though they are not applicable to my own situation.

    Perhaps the most straightforward method is to modify PYTHONPATH to include the submodule directory. This, however, must then either be specified each time the application is invoked, or modified on a system/user level, potentially harming other processes. The variable could be set in a wrapping shell or batch script, but this requires either further environmental assumptions or limits changes to the invocation of Python.

    A more modern and less trouble-fraught analog is to install the application in a virtual environment and simply add the submodule path to the virtual environment.

    Reaching further afield, if the submodule includes a setuptools setup.py, it may simply be installed, avoiding path customization altogether. This may be accomplished by maintaining a publication to repositories such as pypi (a non-starter for proprietary packages) or by utilizing/abusing pip install -e to install either the submodule package directly or from its repository. Once again, virtual environments make this solution simpler by avoiding potential cross-application conflicts and permissions issues.

    If the target OS set can be limited to those with strong symlink support (in practice this excludes all Windows through at least 10), the submodules may be linked into to bypass the wrapping directory and put the target package directly in the working directory:

    foo/
        bar/ --> bar_src/bar
        bar_src/
            bar/
                __init__.py
                eggs.py
            test/
            setup.py
        foo/
            __init__.py
            ham.py
        main.py
    

    This has the downside of limiting the potential users of the application and filling the foo directory with confusing clutter, but may be an acceptable solution in some cases.