Search code examples
javascalasbtsbt-assemblyinformation-extraction

OpenIE 4.2 - Can't generate standalone .jar using sbt

I'm trying to generate a standalone .jar of the OpenIE tool from Washington's Knowitall project, as described in this Github page, using the command:

sbt clean compile assembly

but it stops after a while, without giving any error. I can execute sbt clean compile, but sbt assembly does not work. The output is as follows when running sbt assembly:

$ sbt assembly
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=384m; support was removed in 8.0
[info] Loading project definition from /Users/john/Desktop/Tese/OPENIE/OpenIE 4/openie-master/project
[info] Set current project to openie (in build file:/Users/raposo/Desktop/Tese/OPENIE/OpenIE%204/openie-master/)
[info] Compiling 1 Scala source to /Users/john/Desktop/Tese/OPENIE/OpenIE 4/openie-master/target/scala-2.10/test-classes...
[info] Including: jetty-servlet-7.6.9.v20130131.jar
[info] Including: srlie_2.10-1.0.3.jar
[info] Including: jetty-security-7.6.9.v20130131.jar
[info] Including: breeze-math_2.10-0.2.jar
[info] Including: jetty-server-7.6.9.v20130131.jar
[info] Including: scala-library-2.10.3.jar
[info] Including: javax.servlet-2.5.0.v201103041518.jar
[info] Including: jetty-continuation-7.6.9.v20130131.jar
[info] Including: jetty-http-7.6.9.v20130131.jar
[info] Including: jetty-io-7.6.9.v20130131.jar
[info] Including: common-scala_2.10-1.1.2.jar
[info] Including: scopt_2.10-2.1.0.jar
[info] Including: breeze-core_2.10-0.2.jar
[info] Including: clearnlp-2.0.2.jar
[info] Including: paranamer-2.2.jar
[info] Including: netlib-java-0.9.3.jar
[info] Including: args4j-2.0.23.jar
[info] Including: arpack_combined_all-0.1.jar
[info] Including: log4j-1.2.17.jar
[info] Including: jregex-1.2_01.jar
[info] Including: hppc-0.5.2.jar
[info] Including: jblas-1.2.1.jar
[info] Including: breeze-learn_2.10-0.2.jar
[info] Including: clearnlp-dictionary-1.0.jar
[info] Including: clearnlp-general-en-pos-1.1.jar
[info] Including: nlptools-srl-clear_2.10-2.4.5.jar
[info] Including: nlptools-parse-clear_2.10-2.4.5.jar
[info] Including: nlptools-postag-clear_2.10-2.4.5.jar
[info] Including: nlptools-tokenize-clear_2.10-2.4.5.jar
[info] Including: nlptools-core_2.10-2.4.5.jar
[info] Including: liblinear-1.8.jar
[info] Including: dispatch-core_2.10-0.11.0.jar
[info] Including: scalaz-core_2.10-7.0.3.jar
[info] Including: async-http-client-1.7.16.jar
[info] Including: netty-3.6.3.Final.jar
[info] Including: slf4j-api-1.7.5.jar
[info] Including: unfiltered-filter_2.10-0.7.0.jar
[info] Including: unfiltered_2.10-0.7.0.jar
[info] Including: commons-cli-1.2.jar
[info] Including: mallet-2.0.7.jar
[info] Including: unfiltered-util_2.10-0.7.0.jar
[info] Including: commons-codec-1.4.jar
[info] Including: unfiltered-jetty_2.10-0.7.0.jar
[info] Including: jetty-webapp-7.6.9.v20130131.jar
[info] Including: jetty-xml-7.6.9.v20130131.jar
[info] Including: jetty-util-7.6.9.v20130131.jar
[info] Including: openregex-scala_2.10-1.1.2.jar
[info] Including: bsh-2.0b4.jar
[info] Including: openregex-1.1.1.jar
[info] Including: jsr305-2.0.1.jar
[info] Including: trove4j-2.0.2.jar
[info] Including: guava-15.0.jar
[info] Including: mtj-0.9.14.jar
[info] Including: opennlp-tools-1.5.3.jar
[info] Including: opennlp-maxent-3.0.3.jar
[info] Including: junit-4.11.jar
[info] Including: hamcrest-core-1.3.jar
[info] Including: opennlp-postag-models-1.5.jar
[info] Including: reverb-core-1.4.3.jar
[info] OpenIESpecTest:
[info] Including: opennlp-chunk-models-1.5.jar
[info] Including: opennlp-tokenize-models-1.5.jar
[info] OpenIE
[info] Including: opennlp-sent-models-1.5.jar
[info] Including: nlptools-chunk-opennlp_2.10-2.4.5.jar
[info] Including: nlptools-postag-opennlp_2.10-2.4.5.jar
[info] Including: nlptools-tokenize-opennlp_2.10-2.4.5.jar
[info] Including: reverb-models-1.4.0.jar
[info] Including: nlptools-stem-morpha_2.10-2.4.5.jar
Loading feature templates.
Loading models.
[info] Including: morpha-stemmer-1.0.5.jar
[info] Including: scala-arm_2.10-1.3.jar
Loading lexica.
[info] Including: nlptools-sentence-opennlp_2.10-2.4.5.jar
[info] Including: common-java-2.0.2.jar
[info] Including: commons-lang-2.6.jar
[info] Including: logback-core-1.0.13.jar
[info] Including: logback-classic-1.0.13.jar
[info] Including: chunkedextractor_2.10-2.2.1.jar
Loading configuration.
Loading feature templates.
Loading models.
Loading feature templates.
Loading models.
Loading lexica.
Loading feature templates.
Loading models.
Loading feature templates.
Loading models.
Loading lexica.
Loading feature templates.
Loading models.
Loading lexica.
[info] - should instantiate and extract correctly
[info] Run completed in 3 minutes, 47 seconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

It looks like it did everything fine, but then it just stays there, without finishing, and I can't find the .jar anywhere.

My java version is:

java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

And the SBT version this project is using is [info] 0.13.0. I'm doing this in a Macbook pro running the OS X 10.10.5 Yosemite.

Thanks for any help!

Solution

  • Finally I was able to solve this. Will post answer in case anyone else runs into this problem.

    First, I chose a newer version of sbt-assembly (0.14.3), by changing the file openie-master/project/plugins.sbt to:

    addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.6.2")
addSbtPlugin("com.github.gseitz" % "sbt-release" % "0.8")
addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "0.8.1")

    Then, I tweaked the build options a bit, mostly by adding javaOptions to increase memory usage, and changing some imports that have been deprecated. The final version of openie-master/build.sbt was like this:

    import NativePackagerKeys._
import sbt._
import Keys._

name := "openie"
organization := "edu.washington.cs.knowitall.openie"
crossScalaVersions := Seq("2.10.2")
scalaVersion <<= crossScalaVersions { (vs: Seq[String]) => vs.head }

resolvers += "Sonatype SNAPSHOTS" at "https://oss.sonatype.org/content/repositories/snapshots/"

libraryDependencies ++= Seq(
  // extractor components
  "edu.washington.cs.knowitall.srlie" %% "srlie" % "1.0.3",
  "edu.washington.cs.knowitall.chunkedextractor" %% "chunkedextractor" % "2.2.1",
  // for splitting sentences
  "edu.washington.cs.knowitall.nlptools" %% "nlptools-sentence-opennlp" % "2.4.5",
  // for remote components
  "net.databinder.dispatch" %% "dispatch-core" % "0.11.0",
  // resource management
  "com.jsuereth" %% "scala-arm" % "1.3",
  // logging
  "org.slf4j" % "slf4j-api" % "1.7.5",
  "ch.qos.logback" % "logback-core" % "1.0.13",
  "ch.qos.logback" % "logback-classic" % "1.0.13",
  "org.scalatest" % "scalatest_2.10" % "2.0.RC1" % "test",
  "commons-lang" % "commons-lang" % "2.6")

mainClass in assembly := Some("edu.knowitall.openie.OpenIECli")

scalacOptions ++= Seq("-optimize", "-unchecked", "-deprecation")

assemblyOption in assembly := (assemblyOption in assembly).value.copy(cacheUnzip = true)
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = true, includeDependency = true)
assemblyOption in assembly := (assemblyOption in assembly).value.copy(cacheOutput = false)

// custom options for high memory usage

javaOptions += "-J-XX:MaxMetaspaceSize=2G"
javaOptions += "-J-XX:+CMSClassUnloadingEnabled"
javaOptions += "-J-Xmx3G"
javaOptions += "-XX:+UseConcMarkSweepGC"

traceLevel in run := 0
fork in run := true

//test in assembly := {}
//fork in Test := true

connectInput in run := true // forward stdin/out to fork

licenses := Seq("Open IE Software License Agreement" -> url("https://raw.github.com/knowitall/openie/master/LICENSE"))
homepage := Some(url("https://github.com/knowitall/openie"))

publishMavenStyle := true
publishTo <<= version { (v: String) =>
  val nexus = "https://oss.sonatype.org/"
  if (v.trim.endsWith("SNAPSHOT"))
    Some("snapshots" at nexus + "content/repositories/snapshots")
  else
    Some("releases"  at nexus + "service/local/staging/deploy/maven2")
}

pomExtra := (  <scm>
    <url>https://github.com/knowitall/openie</url>
    <connection>scm:git://github.com/knowitall/openie.git</connection>
<developerConnection>scm:git:git@github.com:knowitall/openie.git</developerConnection>
    <tag>HEAD</tag>
  </scm>
  <developers>
   <developer>
      <name>Michael Schmitz</name>
    </developer>
    <developer>
      <name>Bhadra Mani</name>
    </developer>
  </developers>)

packagerSettings
packageArchetype.java_application

mappings in Universal ++= Seq(
  file("README.md") -> "README.md",
  file("LICENSE") -> "LICENSE"
)

    The assemblyOption values includeScala and includeDependency are true by default. They are there because I was trying to take some stuff out of the JAR that ends up being 800Mb, but I couldn't run it if I took any.