The Spark
DataFrameReader
has an options()
method: but it is a setter: ie it is used to impose options on the Reader
datasource_options = {'driver': 'com.microsoft.sqlserver.jdbc.SQLServerDriver',
'timestampFormat': 'yyyy-MM-dd HH:mm:ss.SSSSSS'}
df_reader = (
self.spark.read.option("kustoCluster", self.data_source.cluster)
.option("kustoDatabase", self.data_source.database)
.option("kustoQuery", resource.query)
.options(**datasource_options)
)
I would like to confirm the options that have been set into the Reader
but have not found a method to do so. Does it exist?
It doesn't exist as part of the public api, it's a private field in a private class.
The content is additionally interpreted differently for different datasources, it's not something you should attempt to workaround either (although doable) as it's possible to change between versions.