I am working on connecting to data in Hadoop that allows dynamic data type connections.
I need to be able to connect to Hive Thrift Server A, pull in some data, and then connect to Hive Thrift Server B and pull in more data.
To my understanding enableHiveSupport needs to be set up on the initial SparkSession.builder. Is there a way to add/change a thrift connection after the fact?
The only possible solution I have come across is using newSession():SparkSession however I am not sure if this is the correct solution that I am looking for.
I am using Spark 2.1, Hadoop 2.7
According to the book Spark The Definitive Guide "After you create [SparkConf], the SparkConf is immutable for that specific Spark Application" Chambers and Zaharia (2018, p.276)