Spark how to prefer class from packaged jar

Question

I am using sbt assembly plugin to create a fat jar. I need some jars which are part of default hadoop/spark but with newer versions.

I want spark worker jvm to prefer the version that is packaged with my fat jar file and not the default hadoop/spark distribution. How can I do this?

Answer 1

The solution to this is to set spark.{driver,executor}.userClassPathFirst in configuration(--conf option) while submitting the spark application. This will first include jars from uber jar and then from spark classpath.

Other solution is to use shading in sbt assembly. And shade the jars in our uber jar whose previous version are included with spark.