How to setup yarn client in code?

Question

I want to run my spark application on my hortonworks data platform. As in this setup I don't have a spark master standalone I want to run as a yarn client.

I am trying to create the SparkSession like this:

SparkSession
    .builder()
    .master("yarn-client")
    .appName("my-app")
    .getOrCreate())

I know I am missing some properties to let spark client where my yarn server is running but I can't seem to find those properties.

Currently the app just hangs init with no error or exception.

Any ideas what I am missing?

Answer 1

It looks like you're trying to run your app locally while your Hortonworks HDP is somewhere else.

Unlike Spark standalone and Mesos modes, in which the master’s address is specified in the --master parameter, in YARN mode the ResourceManager’s address is picked up from the Hadoop configuration.

So your app should be run from Hortonworks itself, which has all the Hadoop configuration in place.