Search code examples
pythonazureazure-machine-learning-serviceazureml-python-sdk

Install local python libary in azure ml studio environment

I am trying to run a Machine Learning training script I wrote in azure by using:

env = Environment.from_conda_specification("experiment_env", "my_environment.yml")
script_config = ScriptRunConfig(source_directory=experiment_folder,
                                script='path_to_file/classifier_train.py',
                                arguments=arguments,
                                environment=env,
                                docker_runtime_config=DockerConfiguration(use_docker=True)

my problem is, that in order to run the code of classifier_train.py I need to call

pip install -e .

since I need code from a local package I wrote (the setup.py for this package is in the experiment folder).

Can anyone tell me how I can run pip install -e . after azure installed all other package when building the environment and before running the classifier_train.py script?

Solution

  • To install local libraries on azure ML environment, we need to create the Data Science VM on the local machine and connect it with the workspace subscription details.

    Create ML studio workspace and download the JSON file which consists of the details of the workspace need to be connected to the VM.

    1. Create a python virtual environment on the local machine
    2. Either virtualenv or conda is fine
    3. Activate the virtual environment
    4. Install the Azure Machine Learning Python SDK
    5. Configure the Azure ML Studio with local machine

    enter image description here

    1. Open the Jupyter Notebook installed and configured. Create a virtual environment.
    2. conda install notebook ipykernel – enable all the ipykernal things
    3. ipython kernel install --user --name <myenv> --display-name "Python (myenv)" – creating a kernel
    4. Launch the jupyter notebook server

    Now we need to get the ARM template by following the procedure taken from MS Docs

    Using the following code block create Windows DSVM

    az vm create --resource-group YOUR-RESOURCE-GROUP-NAME --name YOUR-VM-NAME --image microsoft-dsvm:dsvm-windows:server-2016:latest --admin-username YOUR-USERNAME --admin-password YOUR-PASSWORD --authentication-type password

    Using the following code block create Ubuntu DSVM

    az vm create --resource-group YOUR-RESOURCE-GROUP-NAME --name YOUR-VM-NAME --image microsoft-dsvm:linux-data-science-vm-ubuntu:linuxdsvmubuntu:latest --admin-username YOUR-USERNAME --admin-password YOUR-PASSWORD --generate-ssh-keys --authentication-type password

    Create conda environment:

    conda create -n py310 python=310

    Activate the environment and install the libraries locally which are directly impacted in azure ml platform

    conda activate py310
pip install azure-ai-ml