Search code examples
scalaakkaspark-submitalpakka

Unable to find akka Configurations with Spark Submit

I built a fat jar and I am trying to run it with spark-submit on an EMR or locally. here is the command: 

spark-submit \
--deploy-mode client  \
--class com.stash.data.omni.source.Runner myJar.jar  \
<arguments>

I keep getting an error related to akka configurations:

Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.version'

It seems like the jar cannot find the reference.confs for akka at all. Has anyone dealt with this? I am able to run it without spark-submit on my local machine.

Solution

  • it was my merge strategy. i had a catch all case _ => MergeStrategy.first. i changed it to case x => MergeStrategy.defaultMergeStrategy(x) and it worked.