How do I make my relative paths working when running scripts from my Azure Pipeline?

Question

I have a problem with my Azure Pipeline. The pipeline itself is quite simple and its purpose is in the example to run gen.py:

# azure-pipelines.yml
trigger:
- master

pool:
  vmImage: 'windows-latest'

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.x'
    addToPath: true

- script: |
    python.exe -m pip install --upgrade pip
    python.exe -m pip install setuptools
    pip install -r $(Build.Repository.LocalPath)\requirements312.txt
  displayName: 'Install dependencies'

- script: |
    set PYTHONPATH=$(Build.Repository.LocalPath)\cfg\00-config-tools-ext
    set CONFIG_PATH=$(Build.Repository.LocalPath)\cfg\config.yml
    python $(Build.Repository.LocalPath)\cfg\gen.py
  displayName: 'Run gen.py'

The program cfg/gen.py relies on other programs (.yml/.py) from the same directory (cfg), for example cfg/00-config-tools-ext/sanity_check.py.

gen.py calls sanity_check.py like this:

from sanity_check import check_sanity_config

if __name__ == '__main__':
   check_sanity_config()

In sanity_check.py is now following function now wants to open cfg/config.yml (an excerpt from the original file):

import yaml
def check_sanity_config():

    with open('config.yml', "r", encoding="utf-8") as stream:
            config_yml = yaml.safe_load(stream)

if __name__ == '__main__':
check_sanity_config()

When I run the pipeline like it is as shown above I get this error:

PYTHONPATH: D:\a\1\s\cfg\00-config-tools-ext 
  File "D:\a\1\s\cfg\gen.py", line 53, in <module>
Using config file at: D:\a\1\s\cfg\config.yml
    check_sanity_config()
  File "D:\a\1\s\cfg\00-config-tools-ext\sanity_check.py", line 132, in check_sanity_config
    with open('config.yml', "r", encoding="utf-8") as stream:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'

The only workaround right now for me is to use the absolute path from Azure like this:

def check_sanity_config():

    config_path = 'D:\\a\\1\\s\\cfg\\config.yml'

    with open(config_path, "r", encoding="utf-8") as stream:
            config_yml = yaml.safe_load(stream)

Yet I am not happy with this solution as it requires to change hundreds of relative paths throughout the project and local execution will no longer be possible. On my local Windows machine everything works perfectly fine. The issue seems to revolve around the "with open [file]" somehow Azure is not able to locate it Could you suggest a better way? I already tried many different approaches using relative paths with $(Pipeline.Workspace) for example that do not work. How do I make those relative paths work or do I have to rewrite all "with open [file]" statements?

Thank you

Answer 1

The error you’re seeing is because the sanity_check.py is looking for config.yml in the same directory it is located (cfg/00-config-tools-ext/), but the config.yml file is actually in the cfg/ directory.

You can provide the full path to the config.yml file in your sanity_check.py. Refer to the below scripts.

import os
import yaml  
 
def check_sanity_config():
    script_dir = os.path.dirname(os.path.abspath(__file__))
    config_path = os.path.abspath(os.path.join(script_dir, '..', 'config.yml'))
 
    with open(config_path, "r", encoding="utf-8") as stream:
        config_yml = yaml.safe_load(stream)