Search code examples
pythonpublishpypi

Python packaging: What to do with auxiliary modules that I use in most of my projects


I have a number of auxiliary modules that I use throughout my different projects. For example, these auxiliary modules contain some code for logging, files, decorators, etc.

For my personal use, I can simply bundle and install all auxiliary modules.

If I want to publish my projects on PyPI, however, what is the most appropriate way to go? Is it appropriate to also publish the auxiliary modules as a package -- even though they hardly make sense to bundle and are hardly useful for anyone to install except when they use my other projects -- and then require the installation of that auxiliary package for each of my other projects?


Solution

  • It is not uncommon to split libraries like these into separate "project"/package, for example krux/tcp-stream-kafka-producer is based on krux/java-stdlib (random example - just because I was browsing the repos yesterday...)

    So, I would suggest you name the utilities as <something>-util (or "various", or "stuff" :)) and release them separately as a module on pipy. This is the cleaner way to do this since it allows different projects to use different versions of the underlying lib without having to maintain multiple copies of that library and with clear versioning/dependencies.

    Your other options:

    1. Include the library in the most popular project: This leads to project dependencies that do not really make sense. For example, if that project changes major version, are the dependent projects affected? (you cannot know)

    2. Copy in every project you use: This might be OK for a short time and a few projects, however, soon you will loose track of changes and the common files will get out of sync. I have done that before and it is very annoying spending a few hours to debug something that you have already patched in another project!

    The only downside I see with releasing it as a separate package is that you have to be careful with conflicts: ie A project needs v0.1.1 and not v0.1.0 while B project is the other way around. Thankfully, python venv can help with such cases.

    Hope it helps