can't add alluxio.security.login.username to spark-submit

I have a spark driver program which I'm trying to set the alluxio user for.

I read this post: How to pass -D parameter or environment variable to Spark job? and although helpful, none of the methods in there seem to do the trick.

My environment:

- Spark-2.2
- Alluxio-1.4
- packaged jar passed to spark-submit

The spark-submit job is being run as root (under supervisor), and alluxio only recognizes this user.

Here's where I've tried adding "-Dalluxio.security.login.username=alluxio":

spark.driver.extraJavaOptions in spark-defaults.conf
on the command line for spark-submit (using --conf)
within the sparkservices conf file of my jar application
within a new file called "alluxio-site.properties" in my jar application

None of these work set the user for alluxio, though I'm easily able to set this property in a different (non-spark) client application that is also writing to alluxio.

Anyone able to make this setting apply in spark-submit jobs?

Solution

If spark-submit is in client mode, you should use --driver-java-options instead of --conf spark.driver.extraJavaOptions=... in order for the driver JVM to be started with the desired options. Therefore your command would look something like:

./bin/spark-submit ... --driver-java-options "-Dalluxio.security.login.username=alluxio" ...

This should start the driver with the desired Java options.

If the Spark executors also need the option, you can set that with:

--conf "spark.executor.extraJavaOptions=-Dalluxio.security.login.username=alluxio"