Search code examples
pythondatabricksconfiguration-files

In DataBricks how can I assign a variable a string value from external file?

In DataBricks I am trying to use an external file for my config variables. I created a CSV file and uploaded it to DBFS. Now I can only access it in the form of tables (DataFrames). How can I assign it to a variable as a string value? What are the other practices to get configuration variables in python?

Solution

  • You can read config file using pandas and initialize the variables accordingly with the columns present inside your csv config file

    conf_df = pd.read_csv('/dbfs/mnt/<your path>')
date_conf = conf_df['date']

    Since the config file is not gonna be significant enough to actually use it in conjuction with Spark , you can resort to pandas