ModuleNotFound Error - Azure ML with prebuilt docker image

Question

I have developed a module which works perfectly when executed locally.

I have created an environment on azure using a prebuilt docker image found here: "azureml/minimal-ubuntu18.04-py37-cpu-inference" https://mcr.microsoft.com/v2/_catalog . Also, Using pythonScriptStep, to run a pipeline. Here is how the step looks

StepPreprocessing = PythonScriptStep(
    name="Preprocessing",
    script_name=e.preprocess_script_path,
    arguments=[
        "--config_path", e.preprocess_config_path,
        "--task", e.preprocess_task,
    ],
    inputs=None,
    compute_target=aml_compute,
    runconfig=run_config,
    source_directory=e.sources_directory,
    allow_reuse=False
)
print("Step Preprocessing created")

This results in error:

Traceback (most recent call last):
[stderr]  File "Pipeline/custom_pipeline.py", line 4, in <module>
[stderr]    from Preprocess.logger import logger
[stderr]ModuleNotFoundError: No module named 'Preprocess'

in the 1st line of entry script (custom_pipeline.py):

import sys
sys.path.append(".") 
from Preprocess.logger import logger

The folder structure is as:

-Preprocess
  -__init__.py
  - Module1
    -__init__.py
    -somefile.py
  - Module2
    -__init__.py
    -someOtherfile.py
  - Pipeline
    -__init__.py
    -custom_pipeline.py
  - logger
    -__init__.py
    -logger.py

Answer 1

I found out that the python script step copies everything inside the source_dir and therefore in my case it was copying the modules and not the root folder. So I had to put the dir Preprocess inside another dir and mention the new dir as source_dir.