Search code examples
apache-sparkhadoophivehive-query

How to fix 'org.apache.hadoop.hive.ql.metadata.HiveException' in Hive-Spark, when using simple query?


I am trying to perform a simple query on Hive - Spark engine:

SELECT count(*) FROM classification_output GROUP BY model_id;

I keep receiving the below error:

Error: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. (state=42000,code=40000)

Only ONT time it worked after 12 minutes

The number of records on the table are around ~7K, I suspect a configuration issue, but I am not sure which parameter.

I tried to:

  • Restart HIVE server
  • Increased the timeout for Hive server

Note:

Using:

SELECT count(*) FROM classification_output;

With no group by.... works just fine and return the below:

+-------+
|  _c0  |
+-------+
| 7164  |
+-------+
1 row selected (0.092 seconds)

Solution

  • After spending some time around this issue, I managed to find the solution. It has nothing to do with the query. There was another SparkClient process running, once I stopped it and executed the query it works fine.