Search code examples
pythonpippackaging

Python 3.8 - Package release broken, but works when installed locally


Okay, so I'm developing this python package that we release on Pypi and Anaconda. In it, we have a number of python files containing classes. Historically, I've used the following structure, without issue.

repo/
    conda/
        conda_build_config.yaml
        meta.yaml
    setup.py
    src/
        A
            ClassA
        B
            ClassB
        ...
        __init__.py
            from .A import ClassA
            from .B import ClassB
            ...

We do GitHub releases, and GitHub actions bundles everything up for us and publishes to PyPi and Condas. Pretty simple. But the number of classes has grown to the point where a flat hierarchy doesn't really make sense, and we want to start grouping them into subpackages nested in individual groups, like so:

repo/
    conda/
        conda_build_config.yaml
        meta.yaml
    setup.py
    package/
        Group1/
            A
                ClassA
            B
                ClassB
            __init__.py
                from .A import ClassA
                from .B import ClassB
        Group2/
            C
                ClassC
            D
                ClassD
            __init__.py
                from .C import ClassC
                from .D import ClassD
        ...
        __init__.py
            from .Group1 import ClassA
            from .Group1 import ClassB
            from .Group2 import ClassA
            from .Group2 import ClassB
            ...

Here's the weird thing: if I run pip install -e . to test it locally, this new setup works exactly how I want it to. But, if I release it to PyPi, pip install package, and then run import package, I get the following error:

ModuleNotFoundError: No module named package.A

I've tried several different organizational structures, using init.py in various ways, and all of the suggestions and structures I've found from other related questions, but it seems invariably that, what works as a local install doesn't work when pushed to PyPi. What am I missing here?

---Edit---

In addition to the solution described in the accepted answer, I had to move my package/ into a src/ and add the following to my setup.py:

package_dir={"": "src"},
packages=find_packages(where="src"),

This made sure that all the subpackages got bundled into the package.


Solution

  • We can try to add the __all__ variable in your __init__.py files.
    For example in the __init__.py from Group1, we can add :

    from .A import ClassA
    from .B import ClassB
    
    __all__ = [
        'ClassA',
        'ClassB'
    ]
    

    And apply the same trick to the other __init__.py from the other folders.
    This should do the job.