Use local .jar as a dependency for spark application written in scala

Question

I am building a spark application to ingest data from Aerospike key value store using aerospike-spark client. I have followed the instructions here https://github.com/aerospike/aerospark to build a .jar file.
Now, I am plannig to use this .jar in another spark application by providing it in spark-submit as:

bin/spark-submit --jars my_jar.jar

Now, what should I include in my build.sbt file in order to accomplish this?

Answer 1

In this case, you are looking to use aerospark as a library in your own spark app. Inside the aerospark project, you can run command: sbt publish-local, which will publish the jar to ivy2 local repository.

Then treat aerospike-spark as a standard library by adding to libraryDependencies in your build.sbt file.