I am new to using the standard Python packaging tools and have typically relied on custom tooling. However, I’ve decided to get a better understanding of the packaging process, but I’m running into an issue.
I’m trying to create artifacts for distributing my application, and I’ve successfully generated both an sdist file and a wheel file. While I could probably set up a process to install from the sdist file, I’m concerned that I might be missing something. The wheel file, however, is being generated with an incomplete inventory.
.
├── bin
│ ├── conn_mgr.sh
│ └── ora_tapi.sh
├── config
│ ├── OraTAPI.ini
│ └── OraTAPI.ini.sample
├── controller
│ ├── conn_mgr.py
│ ├── __init__.py
│ └── ora_tapi.py
├── lib
│ ├── config_manager.py
│ └── __init__.py
├── LICENSE
├── MANIFEST.in
├── model
│ ├── api_generator.py
│ ├── db_objects.py
│ ├── framework_errors.py
│ ├── __init__.py
│ ├── session_manager.py
│ └── user_security.py
├── pyproject.toml
├── README.md
├── requirements.txt
├── setup.py
├── setup.sh
├── templates
│ ├── column_expressions
│ │ ├── inserts
│ │ │ └── updated_on.tpt
│ │ └── updates
│ │ └── updated_on.tpt
│ ├── misc
│ │ ├── trigger
│ │ │ └── table_name_biu.tpt
│ │ └── view
│ │ └── view.tpt
│ └── packages
│ ├── body
│ │ ├── package_footer.tpt
│ │ └── package_header.tpt
│ ├── procedures
│ │ ├── delete.tpt
│ │ ├── insert.tpt
│ │ ├── select.tpt
│ │ └── update.tpt
│ └── spec
│ ├── package_footer.tpt
│ └── package_header.tpt
└── view
├── console_display.py
├── __init__.py
├── interactions.py
└── ora_tapi_csv.py
setup.py File:from setuptools import setup, find_packages
setup(
name="OraTAPI",
version="1.0.6",
description="Oracle TAPI Application",
author="Your Name",
author_email="[email protected]",
packages=find_packages(),
include_package_data=True, # Include files from the MANIFEST.in
package_data={ # Include non-Python files in specific packages
"templates": ["**/*.tpt", "**/*.tpt.sample"],
},
data_files=[ # Include root-level files and other extras
(".", ["setup.py", "LICENSE", "setup.sh", "requirements.txt", "README.md"]),
],
entry_points={
"console_scripts": [
"conn_mgr=controller.conn_mgr:main",
"ora_tapi=controller.ora_tapi:main",
]
},
)
MANIFEST.in File:include *.sh
recursive-include bin *.sh
recursive-include templates *.tpt *.tpt.sample
include config/*.ini
include config/*.ini.sample
include LICENSE
include README.md
I’m creating the package by running the following:
source venv/bin/activate
python3 setup.py sdist bdist_wheel
Both the sdist and wheel files are created, but I’m finding that the wheel file doesn’t include my templates directory or any of its sub-directories and their contents. I notice that files from the root folder (such as README.md, LICENSE, etc.) are displaced to an OraTAPI-1.0.6.data/datadirectory.
Whether I use the MANIFEST.in file or not, I seem to get the same results.
include_package_data=True is set in setup.py.package_data in setup.py.templates directory is correctly included in MANIFEST.in.python3 setup.py sdist bdist_wheel command multiple times after making changes.templates directory and root-level files like LICENSE, README.md, etc.?I forgot to mention that the modules discovered during deployment are deployed under the global site packages (something which I will avoid using a venv), but only Python modules are deployed - the templates are not:
clive@ryzen-mint:~/.local/lib/python3.10/site-packages$ ls -1
bump2version-1.0.1.dist-info
bumpversion
controller
lib
model
OraTAPI-1.0.6.dist-info
view
After looking at serveral resourced (and a link included by sinoroc in the comments - thanks) and Youtube videos, I ended up reorganising my layout:
├── LICENSE
├── pyproject.toml
├── README.md
└── src
├── controller
│ ├── conn_mgr.py
│ └── ora_tapi.py
├── __init__.py
├── lib
│ ├── config_manager.py
│ └── __init__.py
├── model
│ ├── api_generator.py
│ ├── db_objects.py
│ ├── framework_errors.py
│ ├── __init__.py
│ ├── session_manager.py
│ └── user_security.py
├── OraTAPI.csv
├── resources
│ ├── config
│ │ ├── OraTAPI.ini
│ │ └── OraTAPI.ini.sample
│ └── templates
│ ├── column_expressions
│ │ ├── inserts
│ │ │ ├── created_by.tpt
│ │ │ ├── created_by.tpt.sample
│ │ │ ├── updated_on.tpt
│ │ │ └── updated_on.tpt.sample
│ │ └── updates
│ │ ├── created_by.tpt
│ │ ├── created_by.tpt.sample
│ │ ├── updated_on.tpt
│ │ └── updated_on.tpt.sample
│ ├── misc
│ │ ├── trigger
│ │ │ ├── table_name_biu.tpt
│ │ │ └── table_name_biu.tpt.sample
│ │ └── view
│ │ ├── view.tpt
│ │ ├── view.tpt.lbase_sample
│ │ └── view.tpt.sample
│ └── packages
│ ├── body
│ │ ├── package_footer.tpt
│ │ ├── package_footer.tpt.sample
│ │ ├── package_header.tpt
│ │ └── package_header.tpt.sample
│ ├── procedures
│ │ ├── delete.tpt
│ │ ├── delete.tpt.sample
│ │ ├── upsert.tpt
│ │ └── upsert.tpt.sample
│ └── spec
│ ├── package_footer.tpt
│ ├── package_footer.tpt.sample
│ ├── package_header.tpt
│ └── package_header.tpt.sample
├── setup.sh
└── view
├── console_display.py
├── __init__.py
├── interactions.py
└── ora_tapi_csv.py
I removed the setup.py and MANIFEST.in and went with the following pyproject.toml file:
[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"
[project]
name = "OraTAPI"
version = "1.0.6"
description = "Oracle Table API Generator Application"
authors = [
{ name = "Clive" }
]
# Useful if publishing through PyPI
keywords = [
"python",
"oracle",
"database",
"plsql",
"table api",
"stored procedures",
"views",
"database triggers",
"code generator",
"automation"
]
# Metadata
classifiers = [
"Programming Language :: Python :: 3",
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Topic :: Database",
"Topic :: Software Development :: Code Generators",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent"
]
# Specify the package directory
[tool.setuptools.packages.find]
where = ["src"]
[project.urls]
"Repository" = "https://github.com/avalon60/OraTAPI"
# Include additional resources
[tool.setuptools.package-data]
"*" = ["*.ini", "*.ini.sample", "*.tpt", "*.tpt.sample"]
# Declare scripts as dynamic
dynamic = ["scripts"]
# Scripts defined under `[project.scripts]`
[project.scripts]
conn_mgr = "controller.conn_mgr:main"
ora_tapi = "controller.ora_tapi:main"
NOTE: I didn't include the dependencies in the above.
It was this section which was critical:
*# Include additional resources
[tool.setuptools.package-data]
"*" = ["*.ini", "*.ini.sample", "*.tpt", "*.tpt.sample"]*
This caused the wheel file to start including the resources folder. However, it didn't include the resources folders into the dist file. To get these to be included, I had to use this command:
git add src/**/*.ini src/**/*.tpt src/**/*.tpt.sample
Note that you need to use setuptools-scm for the above solution.
I was hoping that I would be able to get the entrypoint scripts working:
*# Scripts defined under `[project.scripts]`
[project.scripts]
conn_mgr = "controller.conn_mgr:main"
ora_tapi = "controller.ora_tapi:main"*
But this just didn't seem to work - nothing got placed in the venv/bin directory. Anyway, I came to the conclusion that deploying an application, which includes config files and templates that the end-user needs to modify, just wasn't that practical - they get hidden deep in the site-packages directory, e.g. venv/lib/python3.10/site-packages. If I had gotten the entrypoints working I may have considered having the program clone the config and templates to a more suitable location, but the juice didn't apear to be worth the squeeze.
At least I got to the point where I'd be able to develop a distributable package the newer way (using pyproject.toml).
Anyone interested may find this Youtube video useful, and educational. Especially the techniques used to handle your own packages: https://www.youtube.com/watch?v=v6tALyc4C10&t=1613s