Search code examples
apache-sparkapache-zeppelinscalapb

How to created a jar file from the com.trueaccord.scalapb generated sources

I am using ScalapB to generated source code for the protobuf schema. Works great. How do I generate a jar consisting of only the generated source code. I would like to create a jar file and add it as a dependency to Zeppelin Spark Interpreter.

Solution

  • I assume you mean you wanted the compiled generated code. If you are using SBT, just type package and the jar you want will be generated.

    However, your may run into problems related to the fact that (1) Spark ships with a different version of google-protobuf than the one ScalaPB needs and (2) you need to provide to Spark not just your compiled code, but also all its dependencies (like ScalaPB itself). In that case, you can use the sbt-assembly package. Instructions are here: https://scalapb.github.io/sparksql.html