Search code examples
pythonazureazure-synapse

Use Config.INI file in Azure Synapse Notebook


I have a on prem code/ python project which I need to move to azure cloud. Alot of functionality runs around config.ini file which contains pre defined variables and their values.

I am not able to use the same config file in Azure synapse, I have kept it in adls gen-2 folder.

import configparser
read_config =configparser.ConfigParser()
config_path = r'https://prodstaticblob.blob.core.windows.net/data/config.ini'
read_config.read(config_path)
list_sections = read_config.sections()
print(list_sections)

My output to section list is empty list '[]'

I have tried this path aswell. Am I working with wrong path ?
config_path = r'abfss://data@prodstaticblob.dfs.core.windows.net/config.ini'

Connection to that storage location is authorized, I am able to read a sample csv file from same directory. SS below enter image description here


Solution

  • You can use abfss protocol with spark context only and the path you need to pass to configparser should be from root filesystem.

    So, copy the file from adls to local filesystem and read from there.

    Below are the content of config.ini

    enter image description here

    Copy file using mssparkutils

    mssparkutils.fs.cp("abfss://data@jadls2.dfs.core.windows.net/path_to_config.ini","file:/tmp/conf.ini")
    

    This will copy the file under tmp directory.

    import configparser
    read_config =configparser.ConfigParser()
    config_path = "/tmp/conf.ini"
    read_config.read(config_path)
    list_sections = read_config.sections()
    print(list_sections)
    print(read_config.get("Database","db_host"))
    

    Output:

    enter image description here