How to create diff environments for different Python packages in production

Question

I am new to Python as well as Azure, so I might be missing some crucial information with my design. We have couple of libraries which are created to be used by user while working on their Azure ML workspace. Now, this questions remain same if we are creating libraries for user to be used in a simple Jupyter notebook.

Now, both of these libraries have diff packages which could be different then what a user is using. Eg : User might be using an environment which is using numpy x.1 but package A might be using x.2 and package B needs x.3. This is a possibility since all these packages are developed by different teams.

Now, what could be the best way to handle this problem in real world. So far, I am able to come up with below approaches :

1. Install these files in different docker container where the needed packages are installed. And get the desired output done in separate environments.
2. Use Custom Environment options provided by Azure itself. And run the incompatibles ones in different environment.
3. not sure, if we can create different virtual environments and run the packages in different virtual environments but something similar to this if possible.

So, I wanted to know if there is any right way of doing this in real world. I see that we should create a different environment for each project but what about the case when we have different packages which needs different versions of common dependencies. How to handle such case?

Answer 1

You are right, You can create Docker Containers to create separate environment for different libraries for users to manage them separately. You can also make use of azure ML Custom environment and create isolated environment for different packages.

For virtual environment,One option is to directly pip install the packages and second option is to create a requirements.txt file add all the packages in that file, and pip install the requirements.txt in venv. And third option is to use setup.py > Add all your packages in setup.py and run the setup.py code to install those packages and then import them according to the users requirement.

Virtual environment:-

Created one requirements.txt file like below:-

numpy==1.19.5
pandas==1.3.0

In the Azure ML Notebook, Create Virtual environment and install the requirements.txt packages like below:-

!python3 -m venv myenv

!source myenv/bin/activate

!pip install -r requirements.txt

You can refer my SO thread answer here where I created a setup.py python file with packages in my github repository and installed them in my notebook.

You can directly create different python files with packages and run it in your Notebook like below:-

setup.py from my answer above:-

from setuptools import setup, find_packages

setup(

name='pycode',

version='0.1',

packages=find_packages(),

install_requires=[

'numpy',

'pandas',

'scikit-learn'

],

entry_points={

'console_scripts': [

'pycode=pycode.cli:main'

]

}

)

Output:-