Search code examples
pythonpython-3.xapache-sparkpysparkconfigparser

Pyspark - Python3 get variable from file using configparser


I am trying to get a variable from a file using Configparser but it always returns a string instead of a variable. Please assist

config.ini

[db]
connection_sting =sqlContext.read.format(driver).load(table_nm)

config_conn = ConfigParser() 
conn_string = config_conn.get('db', 'connection_sting')

Current result:

conn_string = 'sqlContext.read.format(driver).load(table_nm)'

Expected:

conn_string = sqlContext.read.format(driver).load(table_nm)

Solution

  • Just don't try. Configuration files are used to provide configuration options, not executable code.

    Instead

    config.ini

    [db]
    driver = some_format
    table_nm = some_table
    

    and

    config = configparser.ConfigParser()
    config.read("config.ini")
    
    connection_sting = (sqlContext.read
        .format(config.get("db", "driver")
        .load(config.get("db", "table_nm")))
    

    And in case when you need executable code use proper modules, not configuration.