Search code examples
pythonapache-sparkpyspark

How can the options set on a Spark DataFrameReader be viewed?


The Spark DataFrameReader has an options() method: but it is a setter: ie it is used to impose options on the Reader

        datasource_options = {'driver': 'com.microsoft.sqlserver.jdbc.SQLServerDriver', 
                              'timestampFormat': 'yyyy-MM-dd HH:mm:ss.SSSSSS'}
        df_reader = (
            self.spark.read.option("kustoCluster", self.data_source.cluster)
            .option("kustoDatabase", self.data_source.database)
            .option("kustoQuery", resource.query)
            .options(**datasource_options)
        )

I would like to confirm the options that have been set into the Reader but have not found a method to do so. Does it exist?


Solution

  • It doesn't exist as part of the public api, it's a private field in a private class.

    The content is additionally interpreted differently for different datasources, it's not something you should attempt to workaround either (although doable) as it's possible to change between versions.