Search code examples
setuptoolssetup.pypython-packagingpython-wheelsdist

Wheel File Not Including templates Directory and Root Files in Python Package


Background:

I am new to using the standard Python packaging tools and have typically relied on custom tooling. However, I’ve decided to get a better understanding of the packaging process, but I’m running into an issue.

I’m trying to create artifacts for distributing my application, and I’ve successfully generated both an sdist file and a wheel file. While I could probably set up a process to install from the sdist file, I’m concerned that I might be missing something. The wheel file, however, is being generated with an incomplete inventory.

Project Structure:

.
├── bin
│   ├── conn_mgr.sh
│   └── ora_tapi.sh
├── config
│   ├── OraTAPI.ini
│   └── OraTAPI.ini.sample
├── controller
│   ├── conn_mgr.py
│   ├── __init__.py
│   └── ora_tapi.py
├── lib
│   ├── config_manager.py
│   └── __init__.py
├── LICENSE
├── MANIFEST.in
├── model
│   ├── api_generator.py
│   ├── db_objects.py
│   ├── framework_errors.py
│   ├── __init__.py
│   ├── session_manager.py
│   └── user_security.py
├── pyproject.toml
├── README.md
├── requirements.txt
├── setup.py
├── setup.sh
├── templates
│   ├── column_expressions
│   │   ├── inserts
│   │   │   └── updated_on.tpt
│   │   └── updates
│   │       └── updated_on.tpt
│   ├── misc
│   │   ├── trigger
│   │   │   └── table_name_biu.tpt
│   │   └── view
│   │       └── view.tpt
│   └── packages
│       ├── body
│       │   ├── package_footer.tpt
│       │   └── package_header.tpt
│       ├── procedures
│       │   ├── delete.tpt
│       │   ├── insert.tpt
│       │   ├── select.tpt
│       │   └── update.tpt
│       └── spec
│           ├── package_footer.tpt
│           └── package_header.tpt
└── view
    ├── console_display.py
    ├── __init__.py
    ├── interactions.py
    └── ora_tapi_csv.py

setup.py File:

from setuptools import setup, find_packages

setup(
    name="OraTAPI",
    version="1.0.6",
    description="Oracle TAPI Application",
    author="Your Name",
    author_email="[email protected]",
    packages=find_packages(),
    include_package_data=True,  # Include files from the MANIFEST.in
    package_data={  # Include non-Python files in specific packages
        "templates": ["**/*.tpt", "**/*.tpt.sample"],
    },
    data_files=[  # Include root-level files and other extras
        (".", ["setup.py", "LICENSE", "setup.sh", "requirements.txt", "README.md"]),
    ],
    entry_points={
        "console_scripts": [
            "conn_mgr=controller.conn_mgr:main",
            "ora_tapi=controller.ora_tapi:main",
        ]
    },
)

MANIFEST.in File:

include *.sh
recursive-include bin *.sh

recursive-include templates *.tpt *.tpt.sample

include config/*.ini
include config/*.ini.sample

include LICENSE
include README.md

The Issue:

I’m creating the package by running the following:

source venv/bin/activate
python3 setup.py sdist bdist_wheel

Both the sdist and wheel files are created, but I’m finding that the wheel file doesn’t include my templates directory or any of its sub-directories and their contents. I notice that files from the root folder (such as README.md, LICENSE, etc.) are displaced to an OraTAPI-1.0.6.data/datadirectory.

Whether I use the MANIFEST.in file or not, I seem to get the same results.

What I’ve Tried:

  • Ensured include_package_data=True is set in setup.py.
  • Explicitly defined package_data in setup.py.
  • Double-checked that the templates directory is correctly included in MANIFEST.in.
  • Ran the python3 setup.py sdist bdist_wheel command multiple times after making changes.

Questions:

  1. What do I need to do to ensure that the wheel file includes both the templates directory and root-level files like LICENSE, README.md, etc.?
  2. Am I missing any steps in the packaging process to make sure the wheel is complete?

Update:

I forgot to mention that the modules discovered during deployment are deployed under the global site packages (something which I will avoid using a venv), but only Python modules are deployed - the templates are not:

clive@ryzen-mint:~/.local/lib/python3.10/site-packages$ ls -1
bump2version-1.0.1.dist-info
bumpversion
controller
lib
model
OraTAPI-1.0.6.dist-info
view


Solution

  • After looking at serveral resourced (and a link included by sinoroc in the comments - thanks) and Youtube videos, I ended up reorganising my layout:

    ├── LICENSE
    ├── pyproject.toml
    ├── README.md
    └── src
        ├── controller
        │   ├── conn_mgr.py
        │   └── ora_tapi.py
        ├── __init__.py
        ├── lib
        │   ├── config_manager.py
        │   └── __init__.py
        ├── model
        │   ├── api_generator.py
        │   ├── db_objects.py
        │   ├── framework_errors.py
        │   ├── __init__.py
        │   ├── session_manager.py
        │   └── user_security.py
        ├── OraTAPI.csv
        ├── resources
        │   ├── config
        │   │   ├── OraTAPI.ini
        │   │   └── OraTAPI.ini.sample
        │   └── templates
        │       ├── column_expressions
        │       │   ├── inserts
        │       │   │   ├── created_by.tpt
        │       │   │   ├── created_by.tpt.sample
        │       │   │   ├── updated_on.tpt
        │       │   │   └── updated_on.tpt.sample
        │       │   └── updates
        │       │       ├── created_by.tpt
        │       │       ├── created_by.tpt.sample
        │       │       ├── updated_on.tpt
        │       │       └── updated_on.tpt.sample
        │       ├── misc
        │       │   ├── trigger
        │       │   │   ├── table_name_biu.tpt
        │       │   │   └── table_name_biu.tpt.sample
        │       │   └── view
        │       │       ├── view.tpt
        │       │       ├── view.tpt.lbase_sample
        │       │       └── view.tpt.sample
        │       └── packages
        │           ├── body
        │           │   ├── package_footer.tpt
        │           │   ├── package_footer.tpt.sample
        │           │   ├── package_header.tpt
        │           │   └── package_header.tpt.sample
        │           ├── procedures
        │           │   ├── delete.tpt
        │           │   ├── delete.tpt.sample
        │           │   ├── upsert.tpt
        │           │   └── upsert.tpt.sample
        │           └── spec
        │               ├── package_footer.tpt
        │               ├── package_footer.tpt.sample
        │               ├── package_header.tpt
        │               └── package_header.tpt.sample
        ├── setup.sh
        └── view
            ├── console_display.py
            ├── __init__.py
            ├── interactions.py
            └── ora_tapi_csv.py
    

    I removed the setup.py and MANIFEST.in and went with the following pyproject.toml file:

    [build-system]
    requires = ["setuptools", "setuptools-scm"]
    build-backend = "setuptools.build_meta"
    
    [project]
    name = "OraTAPI"
    version = "1.0.6"
    description = "Oracle Table API Generator Application"
    authors = [
        { name = "Clive" }
    ]
    
    # Useful if publishing through PyPI
    keywords = [
        "python",
        "oracle",
        "database",
        "plsql",
        "table api",
        "stored procedures",
        "views",
        "database triggers",
        "code generator",
        "automation"
    ]
    
    # Metadata
    classifiers = [
        "Programming Language :: Python :: 3",
        "Development Status :: 4 - Beta",
        "Intended Audience :: Developers",
        "Topic :: Database",
        "Topic :: Software Development :: Code Generators",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent"
    ]
    
    # Specify the package directory
    [tool.setuptools.packages.find]
    where = ["src"]
    
    [project.urls]
    "Repository" = "https://github.com/avalon60/OraTAPI"
    
    # Include additional resources
    [tool.setuptools.package-data]
    "*" = ["*.ini", "*.ini.sample", "*.tpt", "*.tpt.sample"]
    
    # Declare scripts as dynamic
    dynamic = ["scripts"]
    
    # Scripts defined under `[project.scripts]`
    [project.scripts]
    conn_mgr = "controller.conn_mgr:main"
    ora_tapi = "controller.ora_tapi:main"
    

    NOTE: I didn't include the dependencies in the above.

    It was this section which was critical:

    *# Include additional resources
    [tool.setuptools.package-data]
    "*" = ["*.ini", "*.ini.sample", "*.tpt", "*.tpt.sample"]*
    

    This caused the wheel file to start including the resources folder. However, it didn't include the resources folders into the dist file. To get these to be included, I had to use this command:

    git add src/**/*.ini src/**/*.tpt src/**/*.tpt.sample
    

    Note that you need to use setuptools-scm for the above solution.

    I was hoping that I would be able to get the entrypoint scripts working:

    *# Scripts defined under `[project.scripts]`
    [project.scripts]
    conn_mgr = "controller.conn_mgr:main"
    ora_tapi = "controller.ora_tapi:main"*
    

    But this just didn't seem to work - nothing got placed in the venv/bin directory. Anyway, I came to the conclusion that deploying an application, which includes config files and templates that the end-user needs to modify, just wasn't that practical - they get hidden deep in the site-packages directory, e.g. venv/lib/python3.10/site-packages. If I had gotten the entrypoints working I may have considered having the program clone the config and templates to a more suitable location, but the juice didn't apear to be worth the squeeze.

    At least I got to the point where I'd be able to develop a distributable package the newer way (using pyproject.toml).

    Anyone interested may find this Youtube video useful, and educational. Especially the techniques used to handle your own packages: https://www.youtube.com/watch?v=v6tALyc4C10&t=1613s