The method spark.catalog.tableExists("newDb.newTable") throws NoSuchDatabaseException instead of returning false ("newDb" does not exist)

Question

I have a block of code that checks if a given hive "database.table" exists in the current spark session.

val tableExists = spark.catalog.tableExists("newDb.newTable")

I want to store the result in a boolean, so, if the table or the database don't exist, i want to get false.

I run the exact same code on a different environment and it worked just fine, but now I am running it with Scalatest and getting NoSuchDatabaseException thrown when the database (in this case "newDb") doesn't exist.

Any ideas why this is happening?

Answer 1

The issue was that hive support was not enabled in the default SparkSession provided by the DataframeSuiteBase class (from Holdenkarau's spark-testing-base package) that I was extending.

To solve it, override the DataframeSuiteBase method beforeAll() (runs before every test) by adding the enableHiveSupport() method to the SparkSession build chain:

override def beforeAll(): Unit = {
  SparkSessionProvider._sparkSession = SparkSession.builder()
    .master("local") // add whatever other configurations...
    .enableHiveSupport()
    .getOrCreate()
}