Datastax Spark Sql Thriftserver with Spark Application

Question

I have an analytics node running, with Spark Sql Thriftserver running on it. Now I can't run another Spark Application with spark-submit.

It says it doesn't have resources. How to configure the dse node, to be able to run both?

Answer 1

The SparkSqlThriftServer is a Spark application like any other. This means it requests and reserves all resources in the cluster by default.

There are two options if you want to run multiple applications at the same time:

1. Allocate only part of your resources to each application.

This is done by setting spark.cores.max to a smaller value than the max resources in your cluster.

See Spark Docs

2. Dynamic Allocation

Which allows applications to change the amount of resources they use depending on how much work they are trying to do.

See Spark Docs