How to separate development and production requirements.txt for Machine Learning Project?

Question

I'm looking for a better AI/ML project code structure. I know that cookiecutter is there and I really like it.

Here is the problem: I want my Jupyter Notebook added to the project structure like cookiecutter. But when I want to deploy the model and I pip install requirements.txt, all of the package (including Jupyter Notebook requirements) will be installed. I didn't like it.

Is there any project structure, that include notebook inside but separate requirements.txt for analysis and deployment?

Is it good idea to create two folder: one for analysis on notebook with requirements.txt and one for model deployment with its own requirements.txt?

Answer 1

The best solution that comes to my mind is Poetry. It automatically creates the folder structure like a python package.

Folder structure

Poerty creates a project.toml file when project is initialized. This can serve as requirement.txt file for production.You can add production and development package separately in this file using command line or editing the file.

Project.toml

It also creates separate environment for the project which helps in minimizing the conflict with other project.