Search code examples
javascalaapache-sparksbtsbt-assembly

Including unmanaged jars on classpath, when building scala project using sbt


I am trying to build an old project.

The project depends on a java package found in sparrow-1.0.jar

I copied the jar to the lib_managed/jars and lib directory, however when compiling I still get a missing class path error.

project/SparkBuild.sbt contains a referenec to lib directory

unmanagedJars in Compile <<= baseDirectory map { base => (base / "lib" ** "*.jar").classpath },

and the contents of ./lib

root@26eefef538b1:/sparrow/spark-sparrow# tree -D lib
lib
|-- jars
|   `-- sparrow-1.0-SNAPSHOT.jar
|-- sparrow-1.0-SNAPSHOT.jar
`-- sparrow.jar

However when I compile I still get the same errors:

root@26eefef538b1:/sparrow/spark-sparrow# sbt/sbt package assembly
[info] Loading project definition from /sparrow/spark-sparrow/project/project
[info] Loading project definition from /sparrow/spark-sparrow/project
[info] Set current project to root (in build file:/sparrow/spark-sparrow/)
[info] Compiling 260 Scala sources and 16 Java sources to /sparrow/spark-sparrow/core/target/scala-2.9.3/classes...
[error] /sparrow/spark-sparrow/core/src/main/scala/spark/scheduler/sparrow/SparrowScheduler.scala:28: not found: object edu
[error] import edu.berkeley.sparrow.thrift.FrontendService
[error]        ^

Is there a command line option or additional configuration I am missing for importing add classpath from the jars to the project?

The project is compiled against scala 2.9.3 using sbt 0.12.3

I noticed I can run scala with the jar on the classpath and import the objects.

root@26eefef538b1:/sparrow/spark-sparrow# scala -cp $SPARROW_JAR                                                                                                                                                                                                                                                             
Welcome to Scala version 2.9.3 (OpenJDK 64-Bit Server VM, Java 1.7.0_201).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import edu.berkeley.sparrow.thrift.FrontendService
import edu.berkeley.sparrow.thrift.FrontendService

scala>

Solution

  • Adding the lib directory and jar to a sub directory. e.g. ./core/lib/sparrow.jar seems to have solved the problem